Generation method and device of mapping data, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a mapping data generation method and device, electronic equipment and a storage medium, and relates to the technical field of robot mapping, and the method comprises the steps: obtaining to-be-utilized images collected by image collection equipment and to-be-utilized poses corresponding to the to-be-utilized images when a robot drives in a to-be-mapped scene; calculating a three-dimensional coordinate of a real position represented by each effective feature point; calculating to obtain a loopback observation relative pose between the to-be-utilized images with the loopback relationship; performing data optimization by taking a change relationship between poses to be utilized corresponding to the images to be utilized, an association relationship between each effective feature point and a represented real position, and a loopback observation relative pose between the images to be utilized with a loopback relationship as constraint conditions; and the final real pose of the robot and the final real coordinate of the real position represented by each effective feature point when each image to be utilized is collected are obtained and are used as current mapping data.

Description

Technical Field

[0001] The present application relates to the field of robot mapping technology, and in particular to a method, device, electronic device and storage medium for generating mapping data. Background Art

[0002] SLAM (Simultaneous Localization and Mapping) technology is a technology that allows robots to autonomously locate and navigate in unknown environments. In order for the robot to accurately locate, accurate mapping data must first be generated.

[0003] However, in order to generate accurate mapping data, the robot needs to be installed with a variety of acquisition devices to collect information about the scene to be mapped. For example, a stereo camera, inertial measurement unit, and lidar need to be installed in the robot. In addition, the cost of these acquisition devices is relatively high, which makes the cost of the robot that generates mapping data relatively high.

[0004] Therefore, how to reduce the cost of robots while generating accurate mapping data is an urgent problem to be solved. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and storage medium for generating mapping data, so as to reduce the cost of the robot while generating accurate mapping data. The specific technical solution is as follows:

[0006] The present application embodiment first provides a method for generating mapping data, the method comprising:

[0007] Acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the pose to be used corresponding to each image to be used; wherein the pose to be used corresponding to each image to be used is: the pose of the robot captured by the odometer of the robot when the image to be used is captured;

[0008] Based on the to-be-used postures corresponding to the to-be-used images and the pixel coordinates of the effective feature points in the to-be-used images, the three-dimensional coordinates of the real positions represented by the effective feature points are calculated; wherein any effective feature point has other feature points with feature matching, and the to-be-used images to which the other feature points with feature matching belong are adjacent to the to-be-used images to which the effective feature points belong;

[0009] Based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, the posture change of the robot when the two images to be used are collected is calculated to obtain the loop observation relative posture between the two images to be used;

[0010] Taking the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the represented real positions, and the relative postures of the loop observations between the images to be used with a loop relationship as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, so as to obtain the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points as the current mapping data.

[0011] Optionally, the method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with a loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including:

[0012] Construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used;

[0013] Taking the minimization of the sum of the first constraint function, the second constraint function and the third constraint function as the optimization goal, data optimization is performed on the three-dimensional coordinates of the real position represented by each valid feature point and the posture to be used corresponding to each image to be used, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each valid feature point.

[0014] Optionally, the method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with a loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including:

[0015] Taking the change relationship between the postures to be used corresponding to the images to be used and the correlation relationship between the effective feature points and the represented real positions as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used to obtain the intermediate real posture of the robot when each image to be used is collected;

[0016] Taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as the constraint conditions, the intermediate real postures corresponding to the images to be used are optimized to obtain the final real posture of the robot when collecting the images to be used;

[0017] Based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used, the final real coordinates of the real position are calculated.

[0018] Optionally, the data optimization of the posture to be used corresponding to each image to be used is performed based on the change relationship between the postures to be used corresponding to each image to be used and the association relationship between each valid feature point and the represented real position as constraint conditions to obtain the intermediate real posture of the robot when each image to be used is collected, including:

[0019] Selecting a first specified number of images to be used from the currently unselected images to be used;

[0020] Taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, data optimization is performed on the postures to be used corresponding to the selected images to be used to obtain the intermediate real posture of the robot when each selected image to be used is collected;

[0021] Return to the step of selecting a first specified number of images to be used from the currently unselected images to be used, until the intermediate true poses corresponding to the images to be used are obtained.

[0022] Optionally, taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, data optimization is performed on the postures to be used corresponding to the selected images to be used to obtain the intermediate real posture of the robot when each selected image to be used is collected, including:

[0023] Constructing a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting each two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs;

[0024] Taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization goal, data optimization is performed on the posture to be used corresponding to the selected image to be used, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected;

[0025] The method uses the change relationship between the intermediate real postures corresponding to the images to be used and the loop-closed observation relative postures between the images to be used with a loop-closed relationship as constraint conditions, performs data optimization on the intermediate real postures corresponding to the images to be used, and obtains the final real posture of the robot when collecting the images to be used, including:

[0026] Constructing a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used that have a loop relationship, and the difference between the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used;

[0027] Taking the minimization of the sum of the third constraint function and the sixth constraint function as the optimization goal, data optimization is performed on the intermediate real posture corresponding to each image to be used to obtain the final real posture of the robot when each image to be used is collected.

[0028] Optionally, before calculating the change in the posture of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each two images to be used that have a loop relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtaining the relative posture of the loop observation between the two images to be used, the method further includes:

[0029] For each image to be used, an image corresponding to the image to be used and having a posture to be used that satisfies the position loop is determined from other images to be used before the image to be used at a collection time, as a rough screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold;

[0030] From the coarse-screened images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value is determined as an image having a loop relationship with the image to be used.

[0031] Optionally, the step of determining, from the coarse-screened image corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value as an image having a loop relationship with the image to be used, includes:

[0032] From the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold is determined as a similar image corresponding to the image to be used;

[0033] From the similar images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset number threshold is determined as an image having a loop relationship with the image to be used.

[0034] Optionally, the posture to be used corresponding to each image to be used is obtained by the following steps:

[0035] In the case where the posture of the robot changes at a specified angle, a motion estimation is performed based on the image captured by the image acquisition device, and a timestamp of the image captured when the robot changes at the specified angle is determined as a first timestamp; and based on the posture captured by the odometer, a timestamp of the posture captured by the odometer when the robot changes at the specified angle is determined as a second timestamp;

[0036] Based on the time difference between the first timestamp and the second timestamp, time aligning the timestamp of each image to be used and the timestamp of the pose collected by the odometer;

[0037] According to the timestamps of the aligned images to be used and the timestamps of the aligned poses, the collected poses are interpolated so that the timestamps of the interpolated poses are the same as the timestamps of the aligned images to be used, and the pose of the robot collected by the odometer of the robot when collecting each image to be used is obtained as the pose to be used corresponding to the image to be used.

[0038] Optionally, each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to every two adjacent images to be used is greater than a preset posture difference.

[0039] Optionally, the method further comprises:

[0040] For each valid feature point, calculate the average of the feature similarities between the valid feature point and other valid feature points matched with the feature, as the average similarity of the valid feature point;

[0041] For each real position represented by the effective feature point, the effective feature point with the smallest average similarity among the effective feature points representing the real position is used as the stable feature point of the real position, and the characteristics of each determined stable feature point and the association relationship between each stable feature point and the image to be used are recorded in the current mapping data; wherein, the image and stable feature point with an associated relationship indicate that: the image contains the associated stable feature point, or contains feature points that match the characteristics of the associated stable feature point.

[0042] Optionally, the current mapping data further includes global image features of the image to be used. After obtaining the current mapping data, the method further includes:

[0043] Extracting the global image features and the contained feature points of the image to be located;

[0044] Determine an image to be used whose global image features match the global image features of the image to be located as a matching image of the image to be located;

[0045] Determining, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned;

[0046] Based on the final real coordinates of the actual position represented by the determined stable feature points, the position and posture of the robot when the image to be positioned is collected is calculated as the positioning result of the image to be positioned.

[0047] Optionally, the current mapping data also includes global image features of the image to be used, and the method further includes:

[0048] Acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and association relationships between historically collected images and stable feature points;

[0049] Perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images;

[0050] For the matched image to be used and the historically collected image, feature matching is performed on the stable feature points associated with the image to be used and the stable feature points associated with the historically collected image to obtain matched stable feature points;

[0051] The matched stable feature points are used as stable feature points representing the same real position, and the current mapping data and the historical mapping data are merged.

[0052] Optionally, the method further comprises:

[0053] Generate and display a mapping map based on the current mapping data;

[0054] When receiving an editing instruction for the generated mapping map, performing an editing operation on the generated mapping map according to the editing instruction;

[0055] The editing operation includes at least one of the following:

[0056] The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole;

[0057] Delete at least one pose contained in the current map;

[0058] Delete the coordinates of at least one real location contained in the current mapping map.

[0059] Optionally, the features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

[0060] The present application also provides a device for generating mapping data, the device comprising:

[0061] An information acquisition module is used to acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the posture to be used corresponding to each image to be used; wherein the posture to be used corresponding to each image to be used is: the posture of the robot captured by the odometer of the robot when the image to be used is captured;

[0062] A three-dimensional coordinate calculation module is used to calculate the three-dimensional coordinates of the real position represented by each effective feature point based on the posture to be used corresponding to each image to be used and the pixel coordinates of the effective feature points in each image to be used; wherein any effective feature point has other feature points with feature matching, and the image to be used to which the other feature points with feature matching belong is adjacent to the image to be used to which the effective feature point belongs;

[0063] A loop closure observation relative posture determination module is used to calculate the posture change of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each two images to be used that have a loop closure relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, so as to obtain the loop closure observation relative posture between the two images to be used;

[0064] The data optimization module is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point based on the change relationship between the posture to be used corresponding to each image to be used, the association relationship between each effective feature point and the real position represented, and the relative posture of the loop observation between the images to be used with a loop relationship as constraints, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point as the current mapping data.

[0065] Optionally, the data optimization module includes:

[0066] The first function construction submodule is used to construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used;

[0067] A first data optimization submodule is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point, taking the minimum sum of the first constraint function, the second constraint function and the third constraint function as the optimization target, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point;

[0068] and / or,

[0069] The data optimization module comprises:

[0070] The intermediate real posture determination submodule is used to optimize the data of the posture to be used corresponding to each image to be used, taking the change relationship between the postures to be used corresponding to each image to be used and the association relationship between each valid feature point and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each image to be used is collected;

[0071] The final real posture determination submodule is used to optimize the data of the intermediate real postures corresponding to the images to be used, taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as constraints, so as to obtain the final real posture of the robot when collecting the images to be used;

[0072] A final real coordinate determination submodule, used to calculate the final real coordinates of the real position based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used;

[0073] and / or,

[0074] The intermediate real pose determination submodule includes:

[0075] An image selection unit, used for selecting a first specified number of images to be used from the images to be used that are not currently selected;

[0076] The intermediate real posture determination unit is used to optimize the data of the posture to be used corresponding to the selected images to be used, taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; trigger the image selection unit to execute the step of selecting a first specified number of images to be used from the images to be used that are not currently selected, until the intermediate real posture corresponding to each image to be used is obtained;

[0077] and / or,

[0078] The intermediate real pose determination unit includes:

[0079] A function construction subunit, used to construct a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs;

[0080] The intermediate real posture determination subunit is used to optimize the posture to be used corresponding to the selected image to be used by taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization target, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected;

[0081] The final real pose determination submodule includes:

[0082] A function construction unit, used to construct a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop pose differences; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used;

[0083] A final real posture determination unit is used to optimize the data of the intermediate real posture corresponding to each image to be used by taking the minimum sum of the third constraint function and the sixth constraint function as the optimization target, so as to obtain the final real posture of the robot when each image to be used is collected;

[0084] And / or, the device further comprises:

[0085] A coarse screening module is used to calculate the posture change of the robot when the two images to be used are collected based on the pixel coordinates of the feature points of feature matching in each two images to be used with a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtain the relative posture of the loop closure observation between the two images to be used. For each image to be used, from other images to be used before the image to be used at the time of collection, an image to be used corresponding to the image to be used that satisfies the position loop is determined as the coarse screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold;

[0086] A loop determination module is used to determine, from the coarse screened images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used;

[0087] and / or,

[0088] The loop determination module includes:

[0089] A similar image determination submodule is used to determine, from the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold, as a similar image corresponding to the image to be used;

[0090] A loop determination submodule, used to determine, from similar images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used;

[0091] and / or,

[0092] The information acquisition module comprises:

[0093] A timestamp determination submodule is used for, when the posture of the robot changes by a specified angle, performing motion estimation based on the images captured by the image acquisition device, determining the timestamp of the images captured when the robot changes by the specified angle as a first timestamp; and determining the timestamp of the posture captured by the odometer when the robot changes by the specified angle as a second timestamp based on the posture captured by the odometer;

[0094] A time alignment submodule, configured to perform time alignment on the time stamp of each image to be used and the time stamp of the pose collected by the odometer based on the time difference between the first time stamp and the second time stamp;

[0095] an interpolation submodule, for interpolating the acquired posture according to the timestamps of the aligned images to be used and the timestamps of the aligned postures, so that the timestamps of the interpolated postures are the same as the timestamps of the aligned images to be used, and obtaining the posture of the robot acquired by the odometer of the robot when acquiring each image to be used as the posture to be used corresponding to the image to be used;

[0096] and / or,

[0097] Each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference;

[0098] and / or,

[0099] The device also includes:

[0100] The average similarity calculation module is used to calculate, for each valid feature point, the average value of the feature similarities between the valid feature point and other valid feature points matched with the feature point, as the average similarity of the valid feature point;

[0101] The stable point determination module is used to, for each real position represented by the effective feature point, use the effective feature point with the smallest average similarity among the effective feature points representing the real position as the stable feature point of the real position, and record the features of each determined stable feature point and the association relationship between each stable feature point and the image to be used in the current mapping data; wherein, the image and stable feature point with an association relationship means: the image contains the associated stable feature point, or contains feature points that match the features of the associated stable feature point;

[0102] and / or,

[0103] The current mapping data also includes global image features of the image to be used, and the device also includes:

[0104] The image processing module to be located is used to extract the global image features and the feature points contained in the image to be located after generating the current mapping data;

[0105] A matching image determination module, used to determine an image to be used whose global image features match the global image features of the image to be located, as a matching image of the image to be located;

[0106] A matching feature point determination module, used to determine, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned;

[0107] A posture calculation module, used to calculate the posture of the robot when collecting the image to be positioned based on the final real coordinates of the actual position represented by the determined stable feature points, as the positioning result of the image to be positioned;

[0108] and / or,

[0109] The current mapping data also includes global image features of the image to be used, and the device further includes:

[0110] A historical mapping data acquisition module, used to acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and associations between historically collected images and stable feature points;

[0111] A global image feature matching module is used to perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images;

[0112] A stable feature point matching module is used to perform feature matching on the stable feature points associated with the image to be used and the historically collected image to obtain matched stable feature points.

[0113] A merging module is used to use the matched stable feature points as stable feature points representing the same real position, and merge the current mapping data with the historical mapping data;

[0114] and / or,

[0115] The device also includes:

[0116] A mapping generation module is used to generate and display a mapping map based on the current mapping data;

[0117] The editing module is configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following:

[0118] The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole;

[0119] Delete at least one pose contained in the current map;

[0120] Deleting the coordinates of at least one real location contained in the current mapping map;

[0121] and / or,

[0122] The features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

[0123] The present application also provides an electronic device, including:

[0124] Memory, used to store computer programs;

[0125] The processor is used to implement any of the above-mentioned methods for generating mapping data when executing the program stored in the memory.

[0126] An embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any of the above-mentioned methods for generating mapping data is implemented.

[0127] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned methods for generating mapping data.

[0128] Beneficial effects of the embodiments of the present application:

[0129] The method for determining the mapping data provided in the embodiment of the present application obtains the images to be used and the postures to be used determined based on the odometer when the robot is driving in the scene to be mapped, and then optimizes the data with the change relationship between the postures to be used corresponding to each image to be used, the pixel coordinates of each effective feature point in the image to be used, and the relative postures of the loop observation between the images to be used with the loop relationship as constraints, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point. It can be seen that this scheme adopts the method of data optimization to optimize the posture corresponding to the image to be used and the real position represented by the effective feature points contained in the image to be used, which can reduce the error of the real posture of the robot finally determined and the real position represented by each effective feature point, improve the accuracy of the mapping data, and the robot of this scheme only needs to set up two sensors, the image acquisition device and the odometer, to generate accurate mapping data. Therefore, this scheme can reduce the cost of the robot while generating accurate mapping data.

[0130] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0131] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0132] Figure 1 A schematic diagram of a first flow chart of a method for generating mapping data provided in an embodiment of the present application;

[0133] Figure 2 A schematic diagram of map optimization in the method for generating map data provided in an embodiment of the present application;

[0134] Figure 3 A second flow chart of the method for generating mapping data provided in an embodiment of the present application;

[0135] Figure 4 A third flow chart of the method for generating mapping data provided in an embodiment of the present application;

[0136] Figure 5 A fourth flow chart of the method for generating mapping data provided in an embodiment of the present application;

[0137] Figure 6 A fifth flow chart of the method for generating mapping data provided in an embodiment of the present application;

[0138] Figure 7 A sixth flow chart of the method for generating mapping data provided in an embodiment of the present application;

[0139] Figure 8 A schematic diagram of the structure of a device for generating mapping data provided in an embodiment of the present application;

[0140] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0141] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0142] In order to reduce the cost of the robot while generating accurate mapping data, the embodiments of the present application provide a method, device, electronic device and storage medium for generating mapping data. Among them, the method for generating mapping data can be applied to electronic devices, such as computers, servers, etc. In specific applications, the method can also be applied to a robot with data processing capabilities, and the robot can be provided with an image acquisition device and an odometer. The method can include the following steps:

[0143] Acquire each image to be used that is captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the pose to be used corresponding to each image to be used; wherein the pose to be used corresponding to each image to be used is: the pose of the robot captured by the robot's odometer when the image to be used is captured;

[0144] Based on the to-be-used postures corresponding to the to-be-used images and the pixel coordinates of the effective feature points in the to-be-used images, the three-dimensional coordinates of the real positions represented by the effective feature points are calculated; wherein any effective feature point has other feature points with feature matching, and the to-be-used images to which the other feature points with feature matching belong are adjacent to the to-be-used images to which the effective feature points belong;

[0145] Based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, the posture change of the robot when the two images to be used are collected is calculated to obtain the loop observation relative posture between the two images to be used;

[0146] Taking the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop observations between the images to be used with a loop relationship as constraints, data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and the final real postures of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points are obtained as the current mapping data.

[0147] In this embodiment, the data optimization method is used to optimize the corresponding posture of the image to be used and the actual position represented by the effective feature points contained in the image to be used, which can reduce the error of the actual posture of the robot finally determined and the actual position represented by each effective feature point, improve the accuracy of the mapping data, and the robot of this solution only needs to be equipped with two sensors, an image acquisition device and an odometer, to generate accurate mapping data. Therefore, this solution can reduce the cost of the robot while generating accurate mapping data.

[0148] The following is an exemplary introduction to the method for generating mapping data provided in the embodiment of the present application in conjunction with the accompanying drawings. Figure 1 As shown, the method for generating the mapping data may include the following steps:

[0149] S101, acquiring each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the pose to be used corresponding to each image to be used; wherein the pose to be used corresponding to each image to be used is: the pose of the robot captured by the odometer of the robot when the image to be used is captured;

[0150] Among them, the posture of the robot may include the position of the robot, such as the coordinates of the robot in a preset world coordinate system, and the orientation of the robot. The odometer may be a wheel odometer, an inertial odometer (IMU, Inertial Measurement Unit), etc. In order to save costs, the odometer may be a wheel odometer. By setting an odometer for the robot, the posture of the robot can be collected during the robot's driving process, and then for each image to be used collected by the robot's image acquisition device, the posture collected by the odometer can be used to determine the same posture as the collection time of the image to be used as the posture to be used corresponding to the image to be used. The image acquisition device of the robot can be a monocular camera, but of course it is not limited to this.

[0151] The collected images to be used may be all the images collected when the robot is driving in the scene to be mapped, or may be obtained by extracting frames from the images collected when the robot is driving in the scene to be mapped, and the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference. The preset posture difference may include a preset distance difference threshold, and / or a preset angle difference threshold, for example, the preset distance difference threshold may be 0.5m, and the preset angle difference threshold may be 10°. If the difference between the robot postures when collecting two images is greater than the preset posture difference, it indicates that there is a certain difference between the two images. By extracting frames in this way, data redundancy can be reduced, the amount of data of subsequent images to be processed can be reduced, and the processing speed can be improved.

[0152] S102, based on the to-be-used postures corresponding to the to-be-used images and the pixel coordinates of the effective feature points in the to-be-used images, calculating the three-dimensional coordinates of the real positions represented by the effective feature points; wherein any effective feature point has other feature points with feature matching, and the to-be-used images to which the other feature points with feature matching belong are adjacent to the to-be-used images to which the effective feature points belong;

[0153] Feature points in an image can be obtained by processing the image using a feature point extraction algorithm. The feature point extraction algorithm can be a SIFT (Scale-Invariant Feature Transform) algorithm, a SURF (Speeded-Up Robust Feature) algorithm, etc. The feature point extraction algorithm can determine the pixel coordinates of the feature points in the image, as well as the features of the feature points, also known as feature descriptors. In order to improve the robustness of feature points to displacement, rotation, deformation, and even illumination changes, so as to be suitable for complex scenes, a deep learning network model can be used to extract feature points and features of feature points in the image. The deep learning network model can be a SuperPoint model, a DISK (learning local features with policy gradients) model, etc. In addition, the RANSAC (random sampling consensus) algorithm can be used to match feature points in every two images.

[0154] In this embodiment, feature matching feature points in two adjacent images to be used, as well as the to-be-used postures corresponding to the two images to be used and the pixel coordinates of feature matching feature points in the two images to be used can be used to perform triangulation calculations to determine the three-dimensional coordinates of the actual position represented by the feature points. Therefore, for each feature point in each image to be used, if there is no matching relationship between the feature point and its adjacent image to be used, it indicates that the feature point cannot be triangulated and can be regarded as an invalid feature point. In this way, valid feature points can be determined from each feature point.

[0155] In order to avoid mismatching, in this embodiment, feature points that are successfully matched continuously can also be used as valid feature points. For example, if there is a feature point a1 in an image to be used, if there are feature points matching the feature point a1 in a specified number of other images to be used adjacent to the image to be used, then the feature point a1 can be determined as a valid feature point. For example, the specified number can be 5, and the acquisition time of the other images to be used adjacent to the image to be used can be before the image to be used, or after the image to be used.

[0156] After determining the effective feature points, the triangulation algorithm can be used to calculate the three-dimensional coordinates of the actual position represented by each effective feature point based on the posture to be used corresponding to each image to be used and the pixel coordinates of the effective feature points in each image to be used. In this way, the initial mapping data can be obtained. The three-dimensional coordinates obtained at this time may have a large error, so subsequent data optimization can be performed on them. The specific process will be described below. Furthermore, if the calculated three-dimensional coordinates of the actual position represented by a certain feature point exceed the image field of view of the image to be used to which the effective feature point belongs, it indicates that the feature point may be a mismatched point, and the feature point can be removed from the effective feature points. In addition, if the distance between the pixel positions of the feature points of feature matching in two adjacent images to be used is small, that is, less than the preset pixel distance threshold, since the error is large when calculating the three-dimensional coordinates of the actual position based on the two feature points, it can also be removed from the effective feature points.

[0157] Furthermore, after this step, it is also possible to establish an association relationship between each valid feature point and the actual position represented, and determine the change relationship between the postures to be used corresponding to each image to be used, so as to facilitate the subsequent data optimization of the postures to be used corresponding to each image to be used and the three-dimensional coordinates of the actual position represented by each valid feature point, wherein the change relationship between the postures to be used corresponding to each image to be used can be the posture difference between the postures to be used corresponding to every two images to be used, and in order to reduce the amount of calculation, it is also possible to only determine the posture difference between every two adjacent images to be used at the acquisition time. The specific data optimization process will be introduced below.

[0158] S103, based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, calculate the posture change of the robot when the two images to be used are collected, and obtain the loop observation relative posture between the two images to be used;

[0159] A loop refers to the robot returning to a place it has been to before during its movement. The images to be used with a loop relationship refer to the images to be used that are collected when the robot passes the same place twice.

[0160] In one implementation, in order to determine the image to be used that has a loop relationship, for each image to be used, feature point matching can be performed on the image to be used and other images to be used before the acquisition time, and the image whose number of feature points matching the feature of the image to be used is greater than the preset number threshold is determined as the image that has a loop relationship with the image to be used. If there is no image in the other images to be used whose number of feature points matching the feature of the image to be used is greater than the preset number threshold, it indicates that the other images to be used do not have a loop relationship with the image to be used. The preset number threshold can be set according to the number of feature points that can be extracted from the image. For example, if the number of feature points that can be extracted from the image is about 500, the preset number threshold can be set to a value between 200-300, and can be set specifically based on experience.

[0161] In one implementation, before step S103, the following method may be used to determine the image to be used that has a loop relationship:

[0162] Step A1, for each image to be used, determine an image whose posture to be used corresponding to the image to be used satisfies the position loop from other images to be used before the image to be used at the time of acquisition, as a rough screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance of the robot movement collected by the odometer between the acquisition times of the two images satisfying the position loop is greater than a second distance threshold;

[0163] The above-mentioned total distance refers to: the cumulative distance moved by the robot collected by the odometer during the interval between the collection of two images that satisfy the position loop. Since the robot may have turned during this interval, the total distance is not necessarily the relative distance between the corresponding positions of the two images. It can be understood that during the interval between the collection of two images to be used, the cumulative distance moved by the robot should be long enough, and the relative distance of the robot when collecting the two images to be used should be close enough. Such two images to be used are more likely to be images with a loop relationship. In this embodiment, for each image to be used, the image to be used whose posture satisfies the position loop can be first determined to preliminarily screen other images to be used to obtain a coarse screen image corresponding to the image to be used.

[0164] Step A2, from the coarse-screened images corresponding to the image to be used, determine an image whose number of feature points matching the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used.

[0165] Since the amount of computation required for feature point matching is relatively large, this embodiment may first perform a preliminary screening of the images to reduce the number of images to be matched with feature points, thereby reducing the amount of computation required for determining the images to be used that have loop relationships and improving the processing speed.

[0166] In one implementation, feature points of the image to be used may be directly matched with the corresponding coarse-screened image to determine an image that has a loop relationship with the image to be used.

[0167] In another implementation, the above-mentioned determining, from the coarse-screened image corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold as an image having a loop relationship with the image to be used, may include the following steps:

[0168] Step A21, determining, from the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold, as a similar image corresponding to the image to be used;

[0169] The above-mentioned global image features refer to features that can represent the entire image, such as the overall features such as the color and shape of the image or target, and are also called global descriptors. The global image features can be HOG (Histogram of Oriented Gradient) features. The global image features can also be obtained by processing the image using a deep learning network model, such as a CNN (Convolutional Neural Networks) model.

[0170] It is understandable that when the robot returns to a place it has been to, there will be a certain similarity between the two captured images. Therefore, in this step, on the basis of determining the rough screening image corresponding to the image to be used, the similarity between each rough screening image and the global image feature of the image to be used can be calculated. For example, the cosine distance between the rough screening image and the global image feature of the image to be used can be calculated. Then, the image with a global image feature similarity greater than a preset similarity threshold is determined from the rough screening image as the similar image corresponding to the image to be used, so as to achieve further screening of the rough screening image.

[0171] Step A22, determining, from similar images corresponding to the image to be used, an image whose number of feature points matching the image to be used is greater than a preset threshold value, as an image having a loop relationship with the image to be used.

[0172] In this embodiment, by utilizing the similarity of global image features, the coarse-screened images corresponding to the images to be used are further screened, thereby further reducing the number of images to be matched with feature points, thereby further reducing the amount of calculation and improving the processing speed.

[0173] It can be understood that although two images to be used that have a loop relationship are determined, the posture of the robot may not be exactly the same when the two images to be used are collected. Therefore, in this step, the pixel coordinates of the feature points of the feature matching in the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the matched feature points can be used to calculate the change in the posture of the robot when the two images to be used are collected. The change in the posture of the robot may refer to the change in position and angle of the robot. Specifically, for each of the two images to be used, a PnP (Perspective-n-Point) algorithm can be used to calculate the posture of the robot when the image to be used is collected based on the pixel coordinates of the effective feature points in the image to be used and the three-dimensional coordinates of the real position represented by the effective feature points. Then, the posture change of the robot when the two images to be used are collected can be calculated as the relative posture of the loop observation.

[0174] S104, taking the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the represented real positions, and the relative postures of the loop observations between the images to be used with a loop relationship as constraints, data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, so as to obtain the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points as the current mapping data.

[0175] In this embodiment, the real position of the robot when collecting each image to be used and the real coordinates of the real position represented by the effective feature points can be used as optimization variables, and a constraint function is constructed according to each constraint condition to perform data optimization. Since the matched effective feature points are actually feature points representing the same real position, the real coordinates of the real position represented by the matched multiple effective feature points can be used as an optimization variable.

[0176] like Figure 2As shown in , the data optimization process can be regarded as a graph optimization problem. The real posture of the robot when collecting each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point can be used as nodes in the graph respectively; the constraint function determined based on the change relationship between the postures to be used corresponding to each two images to be used is used as the edge between the nodes corresponding to the two images to be used; the constraint function determined based on the pixel coordinates of each effective feature point in the image to be used is used as the edge between the effective feature point and the node corresponding to the image to be used; the constraint function determined based on the pixel coordinates of the two images to be used (corresponding to the pixels of the two images to be used) with a loop relationship is used as the edge between the effective feature point and the node corresponding to the image to be used; Figure 2 The constraint function determined by the relative pose of the loop closure observation between the two images to be used is used as the edge between the nodes corresponding to the two images to be used.

[0177] In one implementation, the final real position of the robot when collecting each image to be used and the final real coordinates of the actual position represented by each valid feature point can be determined by overall optimization, which may include:

[0178] Step B1, constructing a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used, and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used, and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop pose difference; wherein the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points of feature matching in the two images to be used;

[0179] The first constraint function can be expressed as: ;in, It represents the difference between the actual posture of the robot when collecting the image i to be used and the image j to be used, that is, ; represents the actual position of the robot when collecting the image i to be used; represents the real position of the robot when collecting the image j to be used; , is the optimization variable.

[0180] Indicates the posture difference between the postures to be used corresponding to the image i to be used and the image j to be used. In order to reduce the amount of calculation, the first constraint function can be constructed by only using the images to be used that are adjacent at the acquisition time and the images to be used that have a loop relationship. That is to say, the image i to be used and the image j to be used are the images to be used that are adjacent at the acquisition time or the images to be used that have a loop relationship.

[0181] The second constraint function can be expressed as: ; Where K is the camera intrinsic parameter of the image acquisition device, and is the optimization variable, represents the actual position of the robot when collecting the image i to be used, represents the real coordinates of the actual position to be represented by the effective feature point k in the image i; it can be understood that The pixel coordinates representing the real coordinates of the actual position represented by the effective feature point k projected in the image i to be used; Represents the pixel coordinates of the effective feature point k in the image i to be used.

[0182] The third constraint function can be expressed as: ;in, Represents the relative pose of the loop observation between the image i to be used and the image j to be used that has a loop relationship.

[0183] Step B2, taking the minimization of the sum of the first constraint function, the second constraint function and the third constraint function as the optimization goal, performs data optimization on the three-dimensional coordinates of the real position represented by each valid feature point and the posture to be used corresponding to each image to be used, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each valid feature point.

[0184] That is, this step can construct an objective function based on the sum of the first constraint function, the second constraint function and the third constraint function. For example, the objective function can be expressed as: .

[0185] Afterwards, the optimal solution of the objective function can be calculated by using the LM (Levenberg-Marquardt method) algorithm or the gradient descent method, with the pose to be used corresponding to each image to be used and the three-dimensional coordinates of the actual position represented by each effective feature point as initial values. By obtaining the value of the optimization variable when the objective function reaches the optimal solution, the final real pose of the robot when collecting each image to be used and the final real coordinates of the actual position represented by each effective feature point can be obtained.

[0186] In this embodiment, the data optimization method is used to optimize the corresponding posture of the image to be used and the actual position represented by the effective feature points contained in the image to be used, which can reduce the error of the actual posture of the robot finally determined and the actual position represented by each effective feature point, improve the accuracy of the mapping data, and the robot of this solution only needs to be equipped with two sensors, an image acquisition device and an odometer, to generate accurate mapping data. Therefore, this solution can reduce the cost of the robot while generating accurate mapping data.

[0187] Since the above overall optimization process has a large amount of calculation and occupies a lot of computing resources, especially a large amount of memory, as the map scale increases, the overall optimization may be difficult to perform. Therefore, in an embodiment of the present application, a step-by-step optimization method can also be used. Figure 3 As shown, the above-mentioned constraint conditions are the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the represented real positions, and the relative postures of the loop-closed observations between the images to be used with the loop-closed relationship, and the data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points to obtain the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, which may include the following steps:

[0188] S1041, taking the change relationship between the postures to be used corresponding to the images to be used and the association relationship between the effective feature points and the represented real positions as constraint conditions, optimizing the postures to be used corresponding to the images to be used, and obtaining the intermediate real posture of the robot when each image to be used is collected;

[0189] That is, in this step, two constraints are first used to perform data optimization to obtain an intermediate result of data optimization, namely, the intermediate real posture of the robot. In one implementation, the process may include the following steps:

[0190] Step C1, constructing a first constraint function and a second constraint function; wherein the first constraint function represents: the difference between the posture difference between the real posture of the robot when collecting each two images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used, and the pixel coordinates of the effective feature point in the image to be used;

[0191] The first constraint function and the second constraint function in this step may be in the same form as the first constraint function and the second constraint function in the above embodiment, and will not be described in detail here.

[0192] Step C2, taking the minimum sum of the first constraint function and the second constraint function as the optimization goal, optimizing the data of the posture to be used corresponding to each image to be used, and obtaining the intermediate real posture of the robot when each image to be used is collected;

[0193] In this step, an objective function may be constructed based on the first constraint function and the second constraint function, and the objective function may be expressed as: .

[0194] By solving the optimal solution of the objective function and determining the value of the optimization variable when the objective function reaches the optimal solution, the intermediate real posture of the robot when collecting each image to be used can be obtained.

[0195] In one implementation, in order to further reduce the amount of optimization calculation, the above-mentioned process of optimizing the three-dimensional coordinates of the to-be-used posture corresponding to each to-be-used image and the actual position represented by each effective feature point with the change relationship between the to-be-used postures corresponding to each to-be-used image and the association relationship between each effective feature point and the actual position represented is used as a constraint condition, and obtaining the intermediate real posture of the robot when collecting each to-be-used image can also be a segmented optimization method, which can specifically include the following steps:

[0196] Step E1, selecting a first specified number of images to be used from the images to be used that are not currently selected;

[0197] The first specified number of images to be used selected each time may be images to be used that are adjacent in acquisition time. For example, the first specified number of images to be used with the earliest acquisition time may be selected from the currently unselected images to be used in the order of acquisition time.

[0198] Step E2, taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraints, optimize the data of the postures to be used corresponding to the selected images to be used, and obtain the intermediate real posture of the robot when each selected image to be used is collected; return to execute step E1 until the intermediate real posture corresponding to each image to be used is obtained.

[0199] In this step, some images to be used can be selected first, and the real posture of the robot when each selected image to be used is collected, and the real coordinates of the actual position represented by the effective feature points in the selected image to be used are used as optimization variables for optimization. This can reduce the amount of data to be processed each time the optimization is performed, thereby further reducing the occupation of computing resources.

[0200] Specifically, the process may include:

[0201] Step E21, constructing a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs;

[0202] Step E22, taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization goal, performing data optimization on the posture to be used corresponding to the selected image to be used, and obtaining the intermediate real posture of the robot when each selected image to be used is collected.

[0203] The optimization process of this step can be similar to the above steps C1-C2, except that in this step, only the real pose corresponding to a part of the images to be used and the real coordinates of the real position represented by the effective feature points in the images to be used are optimized each time, rather than optimizing the real pose corresponding to all the images to be used and the real coordinates of the real position represented by the effective feature points in all the images to be used at the same time. Specifically, the fourth constraint function can be expressed as: ;in, It represents the difference between the actual posture of the robot when collecting the image i1 to be used and the image j1 to be used, that is, ; Indicates the actual position and posture of the robot when collecting the currently selected image i1 to be used; Indicates the actual position of the robot when collecting the currently selected image j to be used; , is the optimization variable. It represents the posture difference between the posture to be used corresponding to the image i1 to be used and the posture to be used corresponding to the image j1 to be used.

[0204] The fifth constraint function can be expressed as: ; Where K is the camera intrinsic parameter of the image acquisition device, and is the optimization variable, It indicates the real position of the robot when collecting the currently selected image i1 to be used. represents the coordinates of the actual position represented by the valid feature point k1 in the currently selected image i1 to be used; The pixel coordinates representing the real coordinates of the actual position represented by the effective feature point k1 projected in the image i1 to be used; Represents the pixel coordinates of the effective feature point k1 in the image i1 to be used.

[0205] The objective function constructed based on the fourth constraint function and the fifth constraint function can be expressed as: .

[0206] S1042, taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures observed by loop closure between the images to be used with a loop closure relationship as constraint conditions, optimizing the data of the intermediate real postures corresponding to the images to be used, and obtaining the final real posture of the robot when collecting the images to be used;

[0207] After obtaining the intermediate real posture corresponding to each image to be used in the previous step, this step can use the real posture of the robot when collecting each image to be used as the optimization variable, and use the change relationship between the intermediate real postures corresponding to each image to be used to construct a constraint function, and then, based on the constraint function constructed based on the change relationship between the intermediate real postures corresponding to each image to be used, and the constraint function constructed based on the relative posture of the loop observation between the images to be used with a loop relationship, the intermediate real posture corresponding to each image to be used is optimized to obtain the final real posture of the robot when collecting each image to be used. Specifically, the process can include the following steps:

[0208] Step D1, constructing a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop posture difference; wherein the loop posture difference represents: the difference between the posture of the robot when collecting the two images to be used, determined based on the feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two images to be used, and the difference between the posture difference between the intermediate real posture of the robot when collecting the two images to be used;

[0209] The third function in this step may be in the same form as the third function in the above embodiment, and will not be described again here.

[0210] The sixth constraint function can be expressed as: ;in, It represents the difference between the actual posture of the robot when collecting the image i to be used and the image j to be used, that is, ; represents the actual position of the robot when collecting the image i to be used; represents the real position of the robot when collecting the image j to be used; , is the optimization variable; It represents the difference between the intermediate real poses of the robot when collecting the image i to be used and the image j to be used, that is, .

[0211] Step D2, taking the minimum sum of the third constraint function and the sixth constraint function as the optimization goal, performs data optimization on the intermediate real posture corresponding to each image to be used, and obtains the final real posture of the robot when collecting each image to be used.

[0212] In this step, an objective function may be constructed based on the sum of the third constraint function and the sixth constraint function. For example, the objective function may be expressed as: By calculating the optimal solution of the objective function and obtaining the value of the optimization variable when the objective function reaches the optimal solution, the final real posture of the robot when collecting each image to be used can be obtained.

[0213] S1043, based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used, calculate the final real coordinates of the real position.

[0214] Since the real coordinates of the actual position represented by the effective feature points in the image to be used are not optimized in the above step S1042, after obtaining the final real posture of the robot when each image to be used is collected, the final real coordinates of the actual position can be calculated based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used. The calculation method can be a triangulation algorithm.

[0215] In this embodiment, accurate mapping data can be generated while reducing the cost of the robot. Furthermore, this embodiment uses a step-by-step optimization method to optimize data, which can reduce the amount of calculation in an optimization process and reduce the occupation of computing resources.

[0216] In one embodiment of the present application, the posture to be used corresponding to each image to be used can be obtained by the following steps:

[0217] Step F1, when the posture of the robot changes at a specified angle, a motion estimation is performed based on the image captured by the image acquisition device, and a timestamp of the image captured when the robot changes at the specified angle is determined as a first timestamp; and based on the posture captured by the odometer, a timestamp of the posture captured by the odometer when the robot changes at the specified angle is determined as a second timestamp;

[0218] Since there may be a certain delay between the clock of the image acquisition device and the clock of the odometer, in this embodiment, a delay calibration may be performed first to align the clock of the image acquisition device and the clock of the odometer. Figure 4As shown, before obtaining the posture to be used corresponding to each image to be used, the robot can be controlled to rotate a certain angle so that the posture of the robot changes by a specified angle. During this period, the image acquisition device of the robot can continuously acquire images, and the robot's odometer can continuously acquire the posture of the robot. At the same time, the robot is estimated to move based on every two adjacent images acquired, for example, the optical flow method can be used for motion estimation. When an angle change is determined based on the image, that is, it is detected through motion estimation that the robot's posture has changed by an angle in the process of acquiring two adjacent images, a first timestamp can be determined based on the timestamp of at least one of the two adjacent images, for example, the timestamp of the image acquired later in the two adjacent images can be used as the first timestamp.

[0219] In addition, when an angle change is determined based on the odometer, that is, an angle change occurs between the postures of the robot collected by the odometer at two adjacent moments, the second timestamp can be determined based on the timestamps of the two postures collected by the odometer.

[0220] Step F2, based on the time difference between the first timestamp and the second timestamp, time aligning the timestamp of each image to be used and the timestamp of the pose collected by the odometer;

[0221] By calculating the time difference between the first timestamp and the second timestamp, the time delay calibration between the image acquisition device and the odometer can be obtained, that is, the time delay of the image acquisition device relative to the odometer. Afterwards, the time difference can be used to time align the timestamps of each image to be used and the timestamps of the postures collected by the odometer. For example, the timestamp of each image to be used can be subtracted from the time difference to obtain the timestamp of the image to be used after alignment. In this case, the timestamp of the aligned posture is still the original timestamp; or, the timestamp of each posture collected by the odometer can be added to the time difference to obtain the timestamp of the aligned posture. In this case, the timestamps of each image to be used after alignment are still the original timestamps.

[0222] Step F3, interpolate the collected posture according to the timestamps of the aligned images to be used and the timestamps of the aligned postures, so that the timestamps of the interpolated postures are the same as the timestamps of the aligned images to be used, and obtain the posture of the robot collected by the robot's odometer when collecting each image to be used as the posture to be used corresponding to the image to be used.

[0223] Since the acquisition frequencies of the image acquisition device and the odometer may be different, for example, the image acquisition device acquires a frame of image every 33 milliseconds, while the odometer acquires a pose every 10 milliseconds. In order to determine the pose to be used corresponding to the image to be used, the acquired pose can be interpolated according to the timestamps of the aligned images to be used and the timestamps of the aligned poses. For example, the timestamp of an aligned image to be used is 33 milliseconds, and the poses acquired by the odometer include pose 1 and pose 2. The timestamp of the aligned pose 1 is 30 milliseconds, and the timestamp of the aligned pose 2 is 40 milliseconds. Then, interpolation can be performed between pose 1 and pose 2 to obtain the pose when the timestamp is 33 milliseconds, which is used as the pose to be used corresponding to the image to be used, that is, the pose of the aligned image to be used.

[0224] In this embodiment, the cost of the robot can be reduced while generating accurate mapping data. Furthermore, when the robot's posture changes at a specified angle, the timestamp of the specified angle change is determined based on the image and the odometer to determine the time delay between the image acquisition device and the odometer, and then the timestamps of each image to be used and the timestamps of the posture collected by the odometer are time-aligned, and the collected postures are interpolated so that the timestamps of the interpolated postures are the same as the timestamps of each image to be used after alignment, which can ultimately make the posture to be used corresponding to the image to be used more accurate, and can also further improve the accuracy of the final mapping data.

[0225] In an embodiment of the present application, the method for generating mapping data may further include the following steps:

[0226] Step G1, for each valid feature point, calculating the average value of feature similarities between the valid feature point and other feature-matched valid feature points, as the average similarity of the valid feature point;

[0227] The mapping data in this embodiment may also include the features of each valid feature point, and since each valid feature point may have multiple feature matching valid feature points, the matching feature points actually represent the same real position, therefore, for multiple matching stable points, only the features of one stable feature point need to be retained. Specifically, for each valid feature point, the feature similarity between the valid feature point and other feature matching valid feature points may be calculated, for example, the Euclidean distance or the cosine distance may be calculated, and then for each valid feature point, the average value of the obtained feature similarities is calculated as the average similarity of the valid feature point.

[0228] Step G2, for each real position represented by the effective feature point, the effective feature point with the smallest average similarity among the effective feature points representing the real position is used as the stable feature point of the real position, and the characteristics of each determined stable feature point and the association relationship between each stable feature point and the image to be used are recorded in the current mapping data; wherein, the image and stable feature point with an associated relationship indicate that: the image contains the associated stable feature point, or contains feature points that match the characteristics of the associated stable feature point.

[0229] It can be understood that the image and the stable feature point that are associated actually indicate that the image field of view of the image contains the real position represented by the stable feature point. Therefore, the image may contain associated stable feature points, and may also contain feature points that match the associated stable feature points.

[0230] In this embodiment, by recording the characteristics of each determined stable feature point and the association between each stable feature point and the image to be used in the current mapping data, a basis for the subsequent positioning and map merging process can be provided. The specific process will be introduced below.

[0231] In one embodiment of the present application, the current mapping data also includes the global image features of the image to be used. After obtaining the current mapping data, Figure 5 As shown, the method may also include the following steps:

[0232] S501, extracting global image features and feature points contained in the image to be located;

[0233] The image to be positioned may be an image collected when the robot is traveling in the scene to be mapped, and the present embodiment may determine the position and posture of the robot based on the image to be positioned. At this time, the global image features and the feature points contained in the image to be positioned may be firstly extracted.

[0234] S502, determining an image to be used whose global image features match the global image features of the image to be located as a matching image of the image to be located;

[0235] In this step, the feature similarity between the global image features of the image to be located and the global image features contained in the current mapping data can be calculated, and the image to be used whose feature similarity is greater than a preset similarity threshold is used as the matching image of the image to be located.

[0236] S503, determining stable feature points that match the feature points contained in the image to be positioned from the stable feature points associated with the matching image of the image to be positioned;

[0237] After the matching image of the image to be located is determined in the above steps, feature matching of feature points can be performed on the image to be located and the matching image to obtain stable feature points that match the feature points in the image to be located. Since the current mapping data contains the association between each stable feature point and the image to be used, the above step of determining the matching image actually uses the global image features to screen each stable feature point contained in the current mapping data, which can reduce the amount of calculation of subsequent matching feature points.

[0238] S504, based on the final real coordinates of the actual position represented by the determined stable feature points, the position and posture of the robot when the image to be positioned is collected is calculated as the positioning result of the image to be positioned.

[0239] Since the matching feature points represent the same real position, the final real coordinates of the real position represented by the determined stable feature points can be used to calculate the posture of the robot when collecting the image to be positioned, for example, the above-mentioned PnP algorithm can be used for calculation.

[0240] The features of each feature point and the global image features of each image in this embodiment can be extracted using a deep learning network model. In this way, the features of the feature points and the global image features of each image can be highly robust to displacement, rotation, deformation or illumination changes, can be applied to complex scenes, and are conducive to improving the accuracy of positioning. In addition, the amount of data of the features of the feature points and the global image features to be saved in the mapping data is relatively small, and the size of the generated mapping map does not exceed 15MB / km. In addition, the features of the feature points based on deep learning are highly robust, and can achieve a positioning accuracy of less than 15 mm when used for positioning.

[0241] In this embodiment, the cost of the robot can be reduced while generating accurate mapping data. Furthermore, the image to be used whose global image features match the global image features of the image to be located is determined as the matching image of the image to be located; stable feature points that match the feature points contained in the image to be located are determined from the stable feature points associated with the matching image of the image to be located; based on the final real coordinates of the actual position represented by the determined stable feature points, the robot's posture when the image to be located is collected is calculated, so that the robot's posture can be accurately located.

[0242] In one embodiment of the present application, the current mapping data also includes the global image features of the image to be used. After obtaining the current mapping data, Figure 6 As shown, the method may also include the following steps:

[0243] S601, obtaining historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and association relationships between historically collected images and stable feature points;

[0244] The above historical mapping data may also be generated for historical mapping scenes, for example, by the mapping data generation method provided in the above embodiments of the present application. The scene to be mapped may overlap with the historical mapping scene, so that the historical mapping data can be merged with the current mapping data.

[0245] S602, performing feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images;

[0246] S603, for the matched image to be used and the historically collected image, feature matching is performed on the stable feature points associated with the image to be used and the stable feature points associated with the historically collected image to obtain matched stable feature points;

[0247] Similarly, in this embodiment, global image feature matching may be performed first for preliminary screening, and then feature point matching may be performed to determine stable feature points that match the historical mapping data and the current mapping data.

[0248] S604: Use the matched stable feature points as stable feature points representing the same real position, and merge the current mapping data with the historical mapping data.

[0249] Since the matched stable feature points represent the same real position, after the matched stable feature points are determined, the current mapping data and the historical mapping data can be merged into the mapping data of the same map. For example, the final real pose of the robot when collecting each image to be used and the final real coordinates of the real position represented by each stable feature point in the current mapping data can be adjusted according to the matched stable feature points, so that the final real coordinates of the real position represented by the adjusted effective feature points are consistent with the final real coordinates of the real position represented by the matched stable feature points in the historical mapping data, and for the features of the matched stable feature points in the historical mapping data and the current mapping data, the features of one of the historical mapping data and the current mapping data can be retained.

[0250] In this embodiment, the cost of the robot can be reduced while generating accurate mapping data. Furthermore, by matching the global image features of each image to be used with the global image features of the historically collected images, matching images to be used and historically collected images are obtained; for the matching images to be used and historically collected images, the stable feature points associated with the image to be used and the stable feature points associated with the historically collected images are matched to obtain matching stable feature points; the matching stable feature points are used as stable feature points representing the same real position to realize the merging of the current mapping data and the historical mapping data, which can support operations such as splicing and supplementing of multiple maps, and is conducive to later maintenance and updating.

[0251] In an embodiment of the present application, after obtaining the current mapping data, the method further includes the following steps:

[0252] Step H1, generating and displaying a mapping map based on current mapping data;

[0253] The mapping map can be displayed on a display screen of an electronic device. The displayed mapping map can include the pose and actual position of each image to be used in the current mapping data. For example, the pose and actual position of each image to be used can be displayed in the form of points.

[0254] Step H2, when an editing instruction for the generated mapping map is received, an editing operation is performed on the generated mapping map according to the editing instruction;

[0255] The above editing operation may include at least one of the following:

[0256] The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole;

[0257] Delete at least one pose contained in the current map;

[0258] Delete the coordinates of at least one real location contained in the current mapping map.

[0259] For example, after displaying the construction map, the user can issue an editing command through any human-computer interaction method to perform overall translation, rotation, etc. on the coordinates of each pose and each real position contained in the current construction map, so that the processed construction map achieves a horizontal and vertical effect, that is, it is more in line with human observation habits. In addition, the coordinates of the poses or real positions contained in the current construction map can be deleted according to actual needs, and only the coordinates of the required poses or real positions can be retained.

[0260] In this embodiment, accurate mapping data can be generated while reducing the cost of the robot. Furthermore, the mapping map generated by this embodiment supports editing functions, and the user can flexibly delete, translate, and rotate the mapping map.

[0261] In one embodiment of the present application, Figure 7 As shown, the method may include the following steps:

[0262] S701, data preprocessing: collect image and odometer data, calibrate the time delay and external parameters of the two, extract the deep learning feature descriptor of the image, and obtain aligned image and odometer data.

[0263] That is, when the robot is driving in the scene to be mapped, the images to be used collected by the robot's image acquisition device and the posture to be used corresponding to each image to be used are obtained, and the global image features and feature point features of the images to be used are extracted.

[0264] S702, data association: select image key frames for feature matching, and construct 2D-3D observation constraints of the image and inter-frame odometer pose constraints.

[0265] That is, determine the effective feature points for feature matching in each image to be used, establish the association between the effective feature points and the actual position represented, determine the change relationship between the postures to be used corresponding to each image to be used, and thus construct a constraint function.

[0266] S703, loop detection: detect the same area image (loop) of the map key frame and add loop constraints.

[0267] That is, based on the pixel coordinates of the feature points of feature matching in each two images to be used that have a loop closure relationship, the posture change of the robot when the two images to be used are collected is calculated to obtain the loop closure observation relative posture between the two images to be used; and a constraint function is constructed based on the loop closure observation relative posture between the images to be used that have a loop closure relationship.

[0268] S704, map optimization: segmentally optimize the map according to the map key frame image reprojection constraints, inter-frame odometer constraints, and inter-frame pose constraints of loop closure to obtain a globally consistent map.

[0269] That is, the above-mentioned segmented optimization process is adopted to obtain the final real posture of the robot when each image to be used is collected; based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used, the final real coordinates of the real position are calculated.

[0270] S705, map editing: performing translation, rotation, and partial deletion on a single map, and performing editing operations such as splicing and supplementation on multiple maps.

[0271] That is, according to the editing instructions, the generated mapping map is translated and rotated as a whole; the current mapping data is merged with the historical mapping data; at least one pose contained in the current mapping map is deleted; or the coordinates of at least one real position contained in the current mapping map are deleted. This step is optional and is mainly used for later editing of the map.

[0272] S706, map saving: according to different requirements of mapping and positioning, the map data structure is streamlined and sorted to obtain a mapping map for later editing and a positioning map for positioning.

[0273] Simplifying and organizing the map data structure means that when merging map data, for the features of stable feature points that match the historical mapping data and the current mapping data, the features of one of the historical mapping data and the current mapping data are retained.

[0274] In this embodiment, the data optimization method is used to optimize the corresponding posture of the image to be used and the actual position represented by the effective feature points contained in the image to be used, which can reduce the error of the actual posture of the robot finally determined and the actual position represented by each effective feature point, improve the accuracy of the mapping data, and the robot of this solution only needs to be equipped with two sensors, an image acquisition device and an odometer, to generate accurate mapping data. Therefore, this solution can reduce the cost of the robot while generating accurate mapping data.

[0275] The present application also provides a device for generating mapping data, such as Figure 8 As shown, the device comprises:

[0276] The information acquisition module 801 is used to acquire each image to be used acquired by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the posture to be used corresponding to each image to be used; wherein the posture to be used corresponding to each image to be used is: the posture of the robot acquired by the odometer of the robot when acquiring the image to be used;

[0277] The three-dimensional coordinate calculation module 802 is used to calculate the three-dimensional coordinates of the real position represented by each effective feature point based on the posture to be used corresponding to each image to be used and the pixel coordinates of the effective feature points in each image to be used; wherein any effective feature point has other feature points with feature matching, and the image to be used to which the other feature points with feature matching belong is adjacent to the image to be used to which the effective feature point belongs;

[0278] The loop closure observation relative posture determination module 803 is used to calculate the posture change of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each two images to be used that have a loop closure relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, so as to obtain the loop closure observation relative posture between the two images to be used;

[0279] The data optimization module 804 is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point based on the change relationship between the posture to be used corresponding to each image to be used, the association relationship between each effective feature point and the represented real position, and the relative posture of the loop observation between the images to be used with a loop relationship as constraints, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point as the current mapping data.

[0280] Optionally, the data optimization module 804 includes:

[0281] The first function construction submodule is used to construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used;

[0282] The first data optimization submodule is used to optimize the data of the three-dimensional coordinates of the real position represented by each valid feature point and the posture to be used corresponding to each image to be used, with the minimization of the sum of the first constraint function, the second constraint function and the third constraint function as the optimization target, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each valid feature point.

[0283] Optionally, the data optimization module 804 includes:

[0284] The intermediate real posture determination submodule is used to optimize the data of the posture to be used corresponding to each image to be used, taking the change relationship between the postures to be used corresponding to each image to be used and the association relationship between each valid feature point and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each image to be used is collected;

[0285] The final real posture determination submodule is used to optimize the data of the intermediate real postures corresponding to the images to be used, taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as constraints, so as to obtain the final real posture of the robot when collecting the images to be used;

[0286] The final real coordinate determination submodule is used to calculate the final real coordinates of the real position based on the final real posture of the robot when collecting each image to be used, and the pixel coordinates of the effective feature points representing the same real position in each image to be used.

[0287] Optionally, the intermediate real pose determination submodule includes:

[0288] An image selection unit, used for selecting a first specified number of images to be used from the images to be used that are not currently selected;

[0289] The intermediate real posture determination unit is used to optimize the data of the posture to be used corresponding to the selected images to be used based on the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraints, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; and trigger the image selection unit to execute the step of selecting a first specified number of images to be used from the images to be used that are not currently selected, until the intermediate real posture corresponding to each image to be used is obtained.

[0290] Optionally, the intermediate real pose determination unit comprises:

[0291] A function construction subunit, used to construct a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs;

[0292] The intermediate real posture determination subunit is used to optimize the posture to be used corresponding to the selected image to be used by taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization target, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected;

[0293] The final real pose determination submodule includes:

[0294] A function construction unit, used to construct a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop pose differences; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used;

[0295] The final real posture determination unit is used to optimize the data of the intermediate real posture corresponding to each image to be used with the minimum sum of the third constraint function and the sixth constraint function as the optimization goal, so as to obtain the final real posture of the robot when collecting each image to be used.

[0296] Optionally, the device further comprises:

[0297] A coarse screening module is used to calculate the posture change of the robot when the two images to be used are collected based on the pixel coordinates of the feature points of feature matching in each two images to be used with a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtain the relative posture of the loop closure observation between the two images to be used. For each image to be used, from other images to be used before the image to be used at the time of collection, an image to be used corresponding to the image to be used that satisfies the position loop is determined as the coarse screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold;

[0298] The loop determination module is used to determine, from the coarse-screened image corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used.

[0299] Optionally, the loop determination module includes:

[0300] A similar image determination submodule is used to determine, from the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold, as a similar image corresponding to the image to be used;

[0301] The loop determination submodule is used to determine, from similar images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used.

[0302] Optionally, the information acquisition module 801 includes:

[0303] A timestamp determination submodule is used for, when the posture of the robot changes by a specified angle, performing motion estimation based on the images captured by the image acquisition device, determining the timestamp of the images captured when the robot changes by the specified angle as a first timestamp; and determining the timestamp of the posture captured by the odometer when the robot changes by the specified angle as a second timestamp based on the posture captured by the odometer;

[0304] A time alignment submodule, configured to perform time alignment on the time stamp of each image to be used and the time stamp of the pose collected by the odometer based on the time difference between the first time stamp and the second time stamp;

[0305] The interpolation submodule is used to interpolate the collected posture according to the timestamps of the aligned images to be used and the timestamps of the aligned postures, so that the timestamps of the interpolated postures are the same as the timestamps of the aligned images to be used, and the posture of the robot collected by the odometer of the robot when collecting each image to be used is obtained as the posture to be used corresponding to the image to be used.

[0306] Optionally, each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to every two adjacent images to be used is greater than a preset posture difference.

[0307] Optionally, the device further comprises:

[0308] The average similarity calculation module is used to calculate, for each valid feature point, the average value of the feature similarities between the valid feature point and other valid feature points matched with the feature point, as the average similarity of the valid feature point;

[0309] The stable point determination module is used to, for each real position represented by the effective feature point, use the effective feature point with the smallest average similarity among the effective feature points representing the real position as the stable feature point of the real position, and record the characteristics of each determined stable feature point and the association relationship between each stable feature point and the image to be used in the current mapping data; wherein, the image and stable feature point with an associated relationship represent: the image contains the associated stable feature point, or contains feature points that match the characteristics of the associated stable feature point.

[0310] Optionally, the current mapping data also includes global image features of the image to be used, and the device further includes:

[0311] The image processing module to be located is used to extract the global image features and the feature points contained in the image to be located after generating the current mapping data;

[0312] A matching image determination module, used to determine an image to be used whose global image features match the global image features of the image to be located, as a matching image of the image to be located;

[0313] A matching feature point determination module, used to determine, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned;

[0314] The posture calculation module is used to calculate the posture of the robot when collecting the image to be positioned based on the final real coordinates of the actual position represented by the determined stable feature points, as the positioning result of the image to be positioned.

[0315] Optionally, the current mapping data also includes global image features of the image to be used, and the device further includes:

[0316] A historical mapping data acquisition module, used to acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and associations between historically collected images and stable feature points;

[0317] A global image feature matching module is used to perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images;

[0318] A stable feature point matching module is used to perform feature matching on the stable feature points associated with the image to be used and the historically collected image to obtain matched stable feature points.

[0319] The merging module is used to use the matched stable feature points as stable feature points representing the same real position to merge the current mapping data with the historical mapping data.

[0320] Optionally, the device further comprises:

[0321] A mapping generation module is used to generate and display a mapping map based on the current mapping data;

[0322] The editing module is configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following:

[0323] The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole;

[0324] Delete at least one pose contained in the current map;

[0325] Delete the coordinates of at least one real location contained in the current mapping map.

[0326] Optionally, the features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

[0327] The present application also provides an electronic device, such as Fig. 9 As shown, including:

[0328] Memory 901, used for storing computer programs;

[0329] The processor 902 is used to implement any of the above-mentioned methods for generating mapping data when executing the program stored in the memory 901.

[0330] In addition, the electronic device may further include a communication bus and / or a communication interface, and the processor 902, the communication interface, and the memory 901 communicate with each other through the communication bus. The communication bus mentioned in the electronic device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The communication bus may be divided into an address bus, a data bus, a control bus, and the like. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0331] The communication interface is used for communication between the above electronic device and other devices.

[0332] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0333] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0334] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned methods for generating mapping data are implemented.

[0335] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the method for generating mapping data described in any one of the above embodiments.

[0336] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0337] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0338] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, and computer-readable storage medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0339] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for generating mapping data, characterized in that: The method comprises: Acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the pose to be used corresponding to each image to be used; wherein the pose to be used corresponding to each image to be used is: the pose of the robot captured by the odometer of the robot when the image to be used is captured; Based on the to-be-used postures corresponding to the to-be-used images and the pixel coordinates of the effective feature points in the to-be-used images, the three-dimensional coordinates of the real positions represented by the effective feature points are calculated; wherein any effective feature point has other feature points with feature matching, and the to-be-used images to which the other feature points with feature matching belong are adjacent to the to-be-used images to which the effective feature points belong; Based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, the posture change of the robot when the two images to be used are collected is calculated to obtain the loop observation relative posture between the two images to be used; Taking the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the represented real positions, and the relative postures of the loop observations between the images to be used with a loop relationship as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, so as to obtain the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points as the current mapping data.

2. The method according to claim 1, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with the loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including: Construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used; Taking the minimization of the sum of the first constraint function, the second constraint function and the third constraint function as the optimization goal, data optimization is performed on the three-dimensional coordinates of the real position represented by each valid feature point and the posture to be used corresponding to each image to be used, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each valid feature point.

3. The method according to claim 1, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used, the association relationship between the effective feature points and the real positions represented, and the relative postures of the loop-closed observations between the images to be used with the loop-closed relationship as constraint conditions, performs data optimization on the postures to be used corresponding to the images to be used and the three-dimensional coordinates of the real positions represented by the effective feature points, and obtains the final real posture of the robot when collecting the images to be used and the final real coordinates of the real positions represented by the effective feature points, including: Taking the change relationship between the postures to be used corresponding to the images to be used and the correlation relationship between the effective feature points and the represented real positions as constraint conditions, data optimization is performed on the postures to be used corresponding to the images to be used to obtain the intermediate real posture of the robot when each image to be used is collected; Taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as the constraint conditions, the intermediate real postures corresponding to the images to be used are optimized to obtain the final real posture of the robot when collecting the images to be used; Based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used, the final real coordinates of the real position are calculated.

4. The method according to claim 3, characterized in that The method uses the change relationship between the postures to be used corresponding to the images to be used and the association relationship between the effective feature points and the represented real positions as constraint conditions to optimize the postures to be used corresponding to the images to be used, and obtains the intermediate real posture of the robot when each image to be used is collected, including: Selecting a first specified number of images to be used from the currently unselected images to be used; Taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, data optimization is performed on the postures to be used corresponding to the selected images to be used to obtain the intermediate real posture of the robot when each selected image to be used is collected; Return to the step of selecting a first specified number of images to be used from the currently unselected images to be used, until the intermediate true poses corresponding to the images to be used are obtained.

5. The method according to claim 4, characterized in that The method uses the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions to optimize the postures to be used corresponding to the selected images to be used, and obtain the intermediate real posture of the robot when each selected image to be used is collected, including: Constructing a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting each two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs; Taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization goal, data optimization is performed on the posture to be used corresponding to the selected image to be used, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; The method uses the change relationship between the intermediate real postures corresponding to the images to be used and the loop-closed observation relative postures between the images to be used with a loop-closed relationship as constraint conditions, performs data optimization on the intermediate real postures corresponding to the images to be used, and obtains the final real posture of the robot when collecting the images to be used, including: Constructing a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used that have a loop relationship, and the difference between the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used; Taking the minimization of the sum of the third constraint function and the sixth constraint function as the optimization goal, data optimization is performed on the intermediate real posture corresponding to each image to be used to obtain the final real posture of the robot when each image to be used is collected.

6. The method according to claim 1, characterized in that Before calculating the change in the posture of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each of the two images to be used that have a loop relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtaining the relative posture of the loop observation between the two images to be used, the method further includes: For each image to be used, an image corresponding to the image to be used and having a posture to be used that satisfies the position loop is determined from other images to be used before the image to be used at a collection time, as a rough screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold; From the coarse-screened images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value is determined as an image having a loop relationship with the image to be used.

7. The method according to claim 6, characterized in that The step of determining, from the coarse-screened images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset threshold value as an image having a loop relationship with the image to be used, comprises: From the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold is determined as a similar image corresponding to the image to be used; From the similar images corresponding to the image to be used, an image having a number of feature points matching the image to be used that is greater than a preset number threshold is determined as an image having a loop relationship with the image to be used.

8. The method according to claim 1, characterized in that Obtain the posture to be used corresponding to each image to be used by the following steps: When the posture of the robot changes by a specified angle, performing motion estimation based on the image captured by the image acquisition device, and determining a timestamp of the image captured when the robot changes by the specified angle as a first timestamp; and based on the posture collected by the odometer, determining a timestamp of the posture collected by the odometer when the angle of the robot changes, as a second timestamp; Based on the time difference between the first timestamp and the second timestamp, time aligning the timestamp of each image to be used and the timestamp of the pose collected by the odometer; According to the timestamps of the aligned images to be used and the timestamps of the aligned poses, the collected poses are interpolated so that the timestamps of the interpolated poses are the same as the timestamps of the aligned images to be used, and the pose of the robot collected by the odometer of the robot when collecting each image to be used is obtained as the pose to be used corresponding to the image to be used.

9. The method according to claim 1, characterized in that: Each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference.

10. The method according to claim 1, characterized in that The method further comprises: For each valid feature point, calculate the average of the feature similarities between the valid feature point and other valid feature points matched with the feature, as the average similarity of the valid feature point; For each real position represented by the effective feature point, the effective feature point with the smallest average similarity among the effective feature points representing the real position is used as the stable feature point of the real position, and the characteristics of each determined stable feature point and the association relationship between each stable feature point and the image to be used are recorded in the current mapping data; wherein, the image and stable feature point with an associated relationship indicate that: the image contains the associated stable feature point, or contains feature points that match the characteristics of the associated stable feature point.

11. The method according to claim 10, characterized in that The current mapping data also includes global image features of the image to be used. After obtaining the current mapping data, the method further includes: Extracting the global image features and the contained feature points of the image to be located; Determine an image to be used whose global image features match the global image features of the image to be located as a matching image of the image to be located; Determining, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned; Based on the final real coordinates of the actual position represented by the determined stable feature points, the position and posture of the robot when the image to be positioned is collected is calculated as the positioning result of the image to be positioned.

12. The method according to claim 10, characterized in that The current mapping data also includes global image features of the image to be used, and the method further includes: Acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and association relationships between historically collected images and stable feature points; Perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images; For the matched image to be used and the historically collected image, feature matching is performed on the stable feature points associated with the image to be used and the stable feature points associated with the historically collected image to obtain matched stable feature points; The matched stable feature points are used as stable feature points representing the same real position, and the current mapping data and the historical mapping data are merged.

13. The method according to claim 1, characterized in that The method further comprises: Generate and display a mapping map based on the current mapping data; When receiving an editing instruction for the generated mapping map, performing an editing operation on the generated mapping map according to the editing instruction; The editing operation includes at least one of the following: The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole; Delete at least one pose contained in the current map; Delete the coordinates of at least one real location contained in the current mapping map.

14. The method according to claim 11 or 12, characterized in that: The features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

15. A device for generating mapping data, characterized in that: The device comprises: An information acquisition module is used to acquire each image to be used captured by the image acquisition device of the robot when the robot is driving in the scene to be mapped, and the posture to be used corresponding to each image to be used; wherein the posture to be used corresponding to each image to be used is: the posture of the robot captured by the odometer of the robot when the image to be used is captured; A three-dimensional coordinate calculation module is used to calculate the three-dimensional coordinates of the real position represented by each effective feature point based on the posture to be used corresponding to each image to be used and the pixel coordinates of the effective feature points in each image to be used; wherein any effective feature point has other feature points with feature matching, and the image to be used to which the other feature points with feature matching belong is adjacent to the image to be used to which the effective feature point belongs; A loop closure observation relative posture determination module is used to calculate the posture change of the robot when collecting the two images to be used based on the pixel coordinates of the feature points of feature matching in each two images to be used that have a loop closure relationship and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, so as to obtain the loop closure observation relative posture between the two images to be used; The data optimization module is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point based on the change relationship between the posture to be used corresponding to each image to be used, the association relationship between each effective feature point and the real position represented, and the relative posture of the loop observation between the images to be used with a loop relationship as constraints, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point as the current mapping data.

16. The device according to claim 15, characterized in that The data optimization module comprises: The first function construction submodule is used to construct a first constraint function, a second constraint function and a third constraint function; wherein the first constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used and the pose difference between the poses to be used corresponding to the two images to be used; the second constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point projected in the image to be used and the pixel coordinates of the effective feature point in the image to be used; the third constraint function represents: the difference between the pose difference between the real pose of the robot when collecting every two images to be used with a loop relationship and the loop pose difference; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on the feature points matched by the features in the two images to be used; A first data optimization submodule is used to optimize the data of the posture to be used corresponding to each image to be used and the three-dimensional coordinates of the real position represented by each effective feature point, taking the minimum sum of the first constraint function, the second constraint function and the third constraint function as the optimization target, so as to obtain the final real posture of the robot when collecting each image to be used and the final real coordinates of the real position represented by each effective feature point; and / or, The data optimization module comprises: The intermediate real posture determination submodule is used to optimize the data of the posture to be used corresponding to each image to be used, taking the change relationship between the postures to be used corresponding to each image to be used and the association relationship between each valid feature point and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each image to be used is collected; The final real posture determination submodule is used to optimize the data of the intermediate real postures corresponding to the images to be used, taking the change relationship between the intermediate real postures corresponding to the images to be used and the relative postures of the loop observations between the images to be used with the loop relationship as constraints, so as to obtain the final real posture of the robot when collecting the images to be used; A final real coordinate determination submodule, used to calculate the final real coordinates of the real position based on the final real posture of the robot when each image to be used is collected, and the pixel coordinates of the effective feature points representing the same real position in each image to be used; and / or, The intermediate real pose determination submodule includes: An image selection unit, used for selecting a first specified number of images to be used from the images to be used that are not currently selected; The intermediate real posture determination unit is used to optimize the data of the posture to be used corresponding to the selected images to be used, taking the change relationship between the postures to be used corresponding to the selected images to be used and the correlation relationship between each effective feature point in the selected images to be used and the represented real position as constraint conditions, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; trigger the image selection unit to execute the step of selecting a first specified number of images to be used from the images to be used that are not currently selected, until the intermediate real posture corresponding to each image to be used is obtained; and / or, The intermediate real pose determination unit includes: A function construction subunit, used to construct a fourth constraint function and a fifth constraint function; wherein the fourth constraint function represents: the difference between the posture difference between the real posture of the robot when collecting every two selected images to be used, and the difference between the posture difference between the postures to be used corresponding to the two images to be used; the fifth constraint function represents: the difference between the pixel coordinates of the real coordinates of the actual position represented by each effective feature point in the selected image to be used projected in the image to which it belongs, and the pixel coordinates of the effective feature point in the image to which it belongs; The intermediate real posture determination subunit is used to optimize the posture to be used corresponding to the selected image to be used by taking the minimum sum of the fourth constraint function and the fifth constraint function as the optimization target, so as to obtain the intermediate real posture of the robot when each selected image to be used is collected; The final real pose determination submodule includes: A function construction unit, used to construct a third constraint function and a sixth constraint function; wherein the third constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used with a loop relationship, and the difference between the loop pose differences; the loop pose difference represents: the difference between the poses of the robot when collecting the two images to be used, determined based on feature points of feature matching in the two images to be used; the sixth constraint function represents: the difference between the pose difference between the real poses of the robot when collecting every two images to be used, and the pose difference between the intermediate real poses of the robot when collecting the two images to be used; A final real posture determination unit is used to optimize the data of the intermediate real posture corresponding to each image to be used by taking the minimum sum of the third constraint function and the sixth constraint function as the optimization target, so as to obtain the final real posture of the robot when each image to be used is collected; And / or, the device further comprises: A coarse screening module is used to calculate the posture change of the robot when the two images to be used are collected based on the pixel coordinates of the feature points of feature matching in each two images to be used with a loop relationship, and the three-dimensional coordinates of the real position represented by the feature points of the feature matching, and obtain the relative posture of the loop closure observation between the two images to be used. For each image to be used, from other images to be used before the image to be used at the time of collection, an image to be used corresponding to the image to be used that satisfies the position loop is determined as the coarse screening image corresponding to the image to be used; wherein the distance between the postures to be used corresponding to the two images satisfying the position loop is less than a first distance threshold, and / or the total distance moved by the robot collected by the odometer between the collection times of the two images satisfying the position loop is greater than a second distance threshold; A loop determination module is used to determine, from the coarse screened images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used; and / or, The loop determination module includes: A similar image determination submodule is used to determine, from the coarse-screened images corresponding to the image to be used, an image whose global image feature similarity with the image to be used is greater than a preset similarity threshold, as a similar image corresponding to the image to be used; A loop determination submodule, used to determine, from similar images corresponding to the image to be used, an image whose number of feature points matching the feature of the image to be used is greater than a preset number threshold, as an image having a loop relationship with the image to be used; and / or, The information acquisition module comprises: A timestamp determination submodule is used for, when the posture of the robot changes by a specified angle, performing motion estimation based on the images captured by the image acquisition device, determining the timestamp of the images captured when the robot changes by the specified angle as a first timestamp; and determining the timestamp of the posture captured by the odometer when the robot changes by the specified angle as a second timestamp based on the posture captured by the odometer; A time alignment submodule, configured to perform time alignment on the time stamp of each image to be used and the time stamp of the pose collected by the odometer based on the time difference between the first time stamp and the second time stamp; an interpolation submodule, for interpolating the acquired posture according to the timestamps of the aligned images to be used and the timestamps of the aligned postures, so that the timestamps of the interpolated postures are the same as the timestamps of the aligned images to be used, and obtaining the posture of the robot acquired by the odometer of the robot when acquiring each image to be used as the posture to be used corresponding to the image to be used; and / or, Each image to be used is obtained by extracting frames of images collected when the robot is driving in the scene to be mapped; the difference between the postures to be used corresponding to each two adjacent images to be used is greater than the preset posture difference; and / or, The device also includes: The average similarity calculation module is used to calculate, for each valid feature point, the average value of the feature similarities between the valid feature point and other valid feature points matched with the feature point, as the average similarity of the valid feature point; The stable point determination module is used to, for each real position represented by the effective feature point, use the effective feature point with the smallest average similarity among the effective feature points representing the real position as the stable feature point of the real position, and record the features of each determined stable feature point and the association relationship between each stable feature point and the image to be used in the current mapping data; wherein, the image and stable feature point with an association relationship means: the image contains the associated stable feature point, or contains feature points that match the features of the associated stable feature point; and / or, The current mapping data also includes global image features of the image to be used, and the device also includes: The image processing module to be located is used to extract the global image features and the feature points contained in the image to be located after generating the current mapping data; A matching image determination module, used to determine an image to be used whose global image features match the global image features of the image to be located, as a matching image of the image to be located; A matching feature point determination module, used to determine, from the stable feature points associated with the matching image of the image to be positioned, stable feature points that match the feature points contained in the image to be positioned; A posture calculation module, used to calculate the posture of the robot when collecting the image to be positioned based on the final real coordinates of the actual position represented by the determined stable feature points, as the positioning result of the image to be positioned; and / or, The current mapping data also includes global image features of the image to be used, and the device also includes: A historical mapping data acquisition module, used to acquire historical mapping data; wherein the historical mapping data includes: global image features of historically collected images, features of historically determined stable feature points, final real coordinates of real positions represented by historically determined stable feature points; and associations between historically collected images and stable feature points; A global image feature matching module is used to perform feature matching on the global image features of each image to be used and the global image features of the historically collected images to obtain matching images to be used and historically collected images; A stable feature point matching module is used to perform feature matching on the stable feature points associated with the image to be used and the historically collected image to obtain matched stable feature points. A merging module is used to use the matched stable feature points as stable feature points representing the same real position, and merge the current mapping data with the historical mapping data; and / or, The device also includes: A mapping generation module is used to generate and display a mapping map based on the current mapping data; The editing module is configured to, when receiving an editing instruction for the generated mapping map, perform an editing operation on the generated mapping map according to the editing instruction; the editing operation includes at least one of the following: The coordinates of each pose and each real position contained in the current map are translated and rotated as a whole; Delete at least one pose contained in the current map; Deleting the coordinates of at least one real location contained in the current mapping map; and / or, The features of each feature point and / or the global image features of each image are extracted using a deep learning network model.

17. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any method described in claims 1-14 when executing a program stored in a memory.

18. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 14 is implemented.

19. A computer program product comprising instructions, characterized in that When the computer program product is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 14.

Images