Robot, mapping and positioning method, and storage medium

Abstract

The present invention discloses a robot, a mapping and positioning method, and a storage medium, which include: collecting motion data of the robot within a cruising cycle T to obtain N groups of basic data; determining the real-time posture of the robot based on the instantaneous angular velocity and instantaneous linear velocity in each group of basic data; using each first image to construct a first image set, and based on each second image, constructing a second image set, and using images in the first image set and the second image set to perform feature extraction and matching to obtain basic matching features; determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture; and constructing a positioning map through the position information at each moment and the first image and second image corresponding to the moment. The present invention is conducive to improving mapping quality.

Description

Technical Field

[0001] The present invention relates to the field of robotics, and in particular to a robot, a mapping and positioning method, and a storage medium. Background Art

[0002] With the aging population, the emergence of robotics has addressed the challenge of human labor shortages. A prerequisite for robots to perform complex tasks is their ability to constantly sense their spatial location. Currently, the best approach for robot positioning is to use the robot's camera to construct a priori maps. The robot then uses its sensors to compare these maps with the priori maps and calculate its position within the maps. Therefore, building highly accurate maps is crucial for robots to perform other complex tasks. However, in real indoor environments, constructing priori maps is complicated by varying lighting, low-texture textures, dynamic objects, and repetitive textures. Current priori maps for robot positioning require either environmental modification, such as by pasting QR codes in the environment, or by using texture features to construct semantic maps using feature points within the environment. (Since texture fully represents the surface structural characteristics of objects within an image, texture features have become an important aspect of feature extraction.) However, the quality of maps constructed using texture features is limited by the richness of the environment's texture. Summary of the Invention

[0003] In view of the above problems, embodiments of the present invention provide a robot, a mapping and positioning method, and a storage medium, which are used to solve the problem of low mapping quality in the prior art.

[0004] According to one aspect of an embodiment of the present invention, a robot is provided, comprising a memory and a processor, wherein the memory stores computer-readable instructions executable on the processor, the robot is equipped with a multi-lens camera, and the processor is configured to implement the following steps when calling and executing the computer-readable instructions:

[0005] Collect motion data of the robot during a cruise cycle to obtain N sets of basic data, where each set of basic data is data at the same time point, including instantaneous angular velocity, instantaneous linear velocity, a first image, and a second image, where the first image is obtained by capturing the left camera of the multi-camera, and the second image is obtained by capturing the right camera of the multi-camera;

[0006] Determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data;

[0007] Using each of the first images to construct a first image set, and based on each of the second images, constructing a second image set, and performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features;

[0008] Determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture;

[0009] A positioning map is constructed using the position information at each moment and the first image and the second image corresponding to the moment.

[0010] Optionally, determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data includes:

[0011] Based on the instantaneous angular velocity, calculating the average angular velocity within a preset time t;

[0012] For any moment within the cruising cycle T, the real-time posture of the robot is calculated using the average angular velocity, the instantaneous angular velocity, and the instantaneous linear velocity.

[0013] Optionally, obtaining basic matching features based on feature extraction and matching using images in the first image set and the second image set includes:

[0014] For each of the first image set and the second image set, feature matching is performed on two adjacent frames of images captured at the same time in the same set to obtain a first matching feature, and feature matching is performed on the image at the first moment and the image at the last moment in the same set to obtain a second matching feature;

[0015] The first matching feature and the second matching feature are used as the basic matching features.

[0016] Optionally, determining the real-time spatial position of the robot at each moment based on the basic matching features and the real-time posture as position information includes:

[0017] determining a first pose according to the first matching feature and an image corresponding to the first matching feature;

[0018] Determine a second posture and a third posture according to the image at the first moment and the image at the last moment corresponding to the second matching feature and the second matching feature;

[0019] Based on the first posture, the second posture and the third posture, a triangulation technique is used to calculate the real-time spatial position of the first matching feature at the corresponding moment as the position information corresponding to the first matching feature.

[0020] Optionally, determining the real-time spatial position of the robot at each moment based on the basic matching features and the real-time posture as position information includes:

[0021] Optimizing the real-time posture of the robot using a posture graph optimization method to obtain an optimized posture;

[0022] The real-time spatial position of the robot is determined according to the first matching feature, the second matching feature and the optimized posture.

[0023] Optionally, optimizing the posture of the robot using a posture graph optimization method to obtain the optimized posture includes:

[0024] The optimization equation is as follows:

[0025]

[0026] In the formula, Log represents the logarithmic mapping from matrix to vector, T i represents the pose at the i-th moment.

[0027] Optionally, the left camera and the right camera are symmetrically installed on the robot, and both the left camera and the right camera are tilted upward by 30°.

[0028] Optionally, constructing a positioning map using the position information at each moment and the first image and the second image corresponding to the moment includes:

[0029] Minimize the reprojection of the position information by the LM algorithm to obtain optimized position information;

[0030] A positioning map is constructed using the optimized position information and the first image and the second image corresponding to the moment, and the first image and the second image are stored in a database corresponding to the positioning map.

[0031] According to another aspect of an embodiment of the present invention, a mapping and positioning method is provided, including:

[0032] Collect motion data of the robot within a cruising cycle T to obtain N sets of basic data, where each set of basic data is data at the same time point, including instantaneous angular velocity, instantaneous linear velocity, a first image, and a second image, where the first image is obtained by taking pictures with the robot's left camera, and the second image is obtained by taking pictures with the robot's right camera;

[0033] Determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data;

[0034] Using each of the first images to construct a first image set, and based on each of the second images, constructing a second image set, and performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features;

[0035] Determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture;

[0036] A positioning map is constructed using the position information at each moment and the first image and the second image corresponding to the moment.

[0037] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps implemented by the above-mentioned mapping and positioning method are implemented.

[0038] The embodiment of the present invention collects motion data of the robot within a cruising cycle T to obtain N groups of basic data, and determines the real-time posture of the robot based on the instantaneous angular velocity and instantaneous linear velocity in each group of basic data; uses each first image to construct a first image set, and uses each second image to construct a second image set, and uses the images in the first image set and the second image set to perform feature extraction and matching to obtain basic matching features; determines the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture; constructs a positioning map through the position information at each moment and the first image and second image corresponding to the moment, realizes reference positioning of multiple images, performs feature matching and positioning on the data and images in motion, so that for complex scenes and dynamic objects, the positioning accuracy is effectively improved, which is conducive to improving the mapping quality.

[0039] The above description is only an overview of the technical solutions of the embodiments of the present invention. In order to more clearly understand the technical means of the embodiments of the present invention, they can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the embodiments of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are specifically listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The accompanying drawings are only used to illustrate the embodiments and are not to be considered as limiting the present invention. In addition, the same reference symbols are used to represent the same components throughout the drawings. In the drawings:

[0041] Figure 1 A schematic diagram of the process of mapping and positioning performed by a robot according to an embodiment of the present invention is shown;

[0042] Figure 2 Another schematic diagram of a process for mapping and positioning performed by a robot according to an embodiment of the present invention is shown;

[0043] Figure 3 Another schematic diagram of a process for mapping and positioning performed by a robot according to an embodiment of the present invention is shown;

[0044] Figure 4 Another schematic diagram of the process of mapping and positioning performed by the robot provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0045] The exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0046] Figure 1 A flowchart of an embodiment of the present invention's robot performing mapping and positioning is shown, and the method includes the following steps:

[0047] S100, collecting motion data of the robot within a cruising cycle T to obtain N sets of basic data, where each set of basic data is data at the same time point, including instantaneous angular velocity, instantaneous linear velocity, a first image, and a second image, where the first image is obtained by capturing the left camera of the multi-camera, and the second image is obtained by capturing the right camera of the multi-camera;

[0048] The instantaneous angular velocity is obtained by the inertial sensor, and the instantaneous linear velocity is obtained by the odometer.

[0049] Among them, N is set according to actual application needs (scene complexity, area, etc.) and is not specifically limited here. For example, N is set to 100.

[0050] Wherein, the multi-eye camera is at least a binocular camera.

[0051] S200, determining the real-time position and posture of the robot based on the instantaneous angular velocity and instantaneous linear velocity in each set of basic data;

[0052] Specifically, by integrating the instantaneous angular velocity and the instantaneous linear velocity, the integration result is used as the real-time posture of the robot at the current moment.

[0053] Optionally, in step S200, Figure 2 As shown, based on the instantaneous angular velocity and instantaneous linear velocity in each set of basic data, the real-time posture of the robot is determined including:

[0054] Based on the instantaneous angular velocity, calculate the average angular velocity within a preset time t;

[0055] For any moment within the cruise cycle T, the average angular velocity, instantaneous angular velocity, and instantaneous linear velocity are used to calculate the real-time posture of the robot.

[0056] Among them, the preset time t can be set according to actual application needs, t ≤T。

[0057] S300, constructing a first image set using each first image, constructing a second image set based on each second image, and performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features;

[0058] S400, determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture;

[0059] Furthermore, based on the basic matching features and the real-time posture, the real-time spatial position of the robot at each moment is determined, and the position information includes:

[0060] Determining a first pose according to the first matching feature and the image corresponding to the first matching feature;

[0061] Determine a second posture and a third posture according to the second matching feature and the image at the first moment and the image at the last moment corresponding to the second matching feature;

[0062] Based on the first pose, the second pose, and the third pose, the triangulation technology is used to calculate the real-time spatial position of the first matching feature at the corresponding moment as the position information corresponding to the first matching feature.

[0063] Specifically, the image corresponding to the first matching feature is the first frame image of two frames of images adjacent at the same time. According to the first matching feature and the first frame image, the real-time posture corresponding to the first frame image is determined as the first posture. The image of the first moment corresponding to the second matching feature is the image at the starting moment, and the image of the last moment corresponding to the second matching feature is the image at the ending moment. According to the second matching feature and the image of the first moment and the image of the last moment corresponding to the second matching feature, the second posture and the third posture are determined. Then, according to the first posture, the second posture and the third posture, the triangulation technology is used to calculate the real-time spatial position at the moment corresponding to the first posture as the position information corresponding to the first matching feature.

[0064] Among them, the pose consists of two variables: attitude and position. The triangulation technology includes but is not limited to linear triangulation, reprojection error triangulation, and deep filters based on epipolar search and block matching technology.

[0065] S500: Construct a positioning map using the position information at each moment and the first image and the second image corresponding to the moment.

[0066] The embodiment of the present invention collects motion data of the robot within a cruising cycle T to obtain N groups of basic data, and determines the real-time posture of the robot based on the instantaneous angular velocity and instantaneous linear velocity in each group of basic data; uses each first image to construct a first image set, and uses each second image to construct a second image set, and uses the images in the first image set and the second image set to perform feature extraction and matching to obtain basic matching features; determines the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture; constructs a positioning map through the position information at each moment and the first image and second image corresponding to the moment, realizes feature matching and positioning of data and images in motion, effectively improves the quality of mapping under low texture, repeated texture, and dynamic objects, and improves the accuracy of positioning mapping.

[0067] It should be understood that this embodiment uses two cameras in order to obtain images with a wider viewing angle range. The above solution can also be implemented when one camera or more than two cameras are used, and it should not be understood as a limitation to the above solution.

[0068] In one optional approach, the robot is also equipped with an inertial sensor, often abbreviated as IMU (Inertial Measurement Unit). In step S100, the average value av is calculated as the average of the inertial sensor's angular velocity over time T. The robot's position is determined by integrating the linear velocity data from the wheel odometer and the angular velocity data from the inertial sensor. The combined calculation of the camera, inertial sensor, and wheel odometer results in more accurate results. Even if the inertial sensor fails, the camera and wheel odometer data can still be used to perform calculations according to the scheme described in the previous paragraph.

[0069] In an optional manner, the robot's posture includes position and attitude. The robot's posture is calculated based on data collected by the inertial sensor and the wheel odometer. There are many ways to calculate the robot's posture. In the present invention, the following formula is further used for calculation:

[0070]

[0071] Where R i represents the posture of the robot at the i-th moment, w i represents the angular velocity output by the inertial sensor gyroscope at the i-th moment, P i represents the position of the robot at the i-th moment, v i represents the speed output by the wheel odometer at time i, t i Represents the time of the i-th time; the integrated position P i , Posture R iand time t i Save the obtained robot pose, av is the average value of the instantaneous angular velocity within a cruise period T. By integrating the time, the robot pose can be calculated conveniently and quickly.

[0072] In an optional manner, the images captured by the left and right cameras are respectively recorded as first images left_images and second images right_images, and the image set first image set captured by the left camera and the image set second image set captured by the right camera are obtained. Figure 3 As shown, for the first image set and the second image set, feature matching is performed on two adjacent frames of images at the same collection moment in the same set to obtain a first matching feature, and feature matching is performed on the image at the first moment and the image at the last moment in the same set to obtain a second matching feature; the first matching feature and the second matching feature are used as the basic matching feature.

[0073] Specifically, ORB feature points of each image in the first image set and the second image set are extracted respectively, and for the same image set, feature matching of the ORB feature points on two adjacent images is performed to obtain the first matching feature between the images, and feature matching is performed on the first image and the last image in the same set to obtain the second matching feature between the images. The obtained first matching feature and second matching feature are used as the basic matching features.

[0074] In an alternative approach, such as Figure 4 As shown, in step S400, the real-time spatial position of the robot at each moment is determined based on the basic matching features and the real-time posture, and the position information includes:

[0075] S410, optimizing the posture of the robot using a posture graph optimization method to obtain an optimized posture;

[0076] S420: Determine the real-time spatial position of the robot according to the first matching feature, the second matching feature, and the optimized posture.

[0077] Preferably, this embodiment uses poseGraph as the pose graph.

[0078] The posture of the robot is optimized by using the posture graph optimization method, and the optimization equation used for optimization is as follows:

[0079]

[0080] In the formula, Log represents the logarithmic mapping from matrix to vector, T i represents the pose at the i-th moment.

[0081] In an optional manner, two cameras are symmetrically installed on the robot, and the two cameras are tilted upward by 20° to 40°. Preferably, the angle is 30°. It should be noted that this embodiment adopts this preferred method to maximize the viewing angle range, which is conducive to improving the quality of the captured image.

[0082] In an optional manner, the time T is greater than or equal to 5 seconds, preferably, the time T is equal to 10 seconds.

[0083] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, any one of the steps described above is implemented.

[0084] The algorithm or demonstration provided herein are not inherently relevant to any particular computer, virtual system or other equipment. Various general-purpose systems can also be used together with the teachings based on this. According to the above description, it is apparent that the structure required for constructing this type of system. In addition, the embodiment of the present invention is not directed to any specific programming language yet. It should be understood that various programming languages can be utilized to realize the content of the present invention described herein, and the above description of specific languages is for the purpose of disclosing the best mode of the present invention.

[0085] In the description provided herein, numerous specific details are described. However, it is understood that embodiments of the present invention may be practiced without these specific details. In some instances, well-known methods, structures, and techniques are not shown in detail so as not to obscure the understanding of this description.

[0086] Similarly, it should be understood that in order to streamline the present invention and aid in understanding one or more of the various inventive aspects, in the above description of exemplary embodiments of the present invention, various features of the embodiments of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, this disclosed method should not be interpreted as reflecting an intention that the claimed invention requires more features than are expressly recited in each claim. Rather, as reflected in the claims below, inventive aspects lie in less than all the features of the individual embodiments disclosed above. Accordingly, the claims that follow the detailed description are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate embodiment of the present invention.

[0087] Those skilled in the art will appreciate that the modules in the devices in the embodiments may be adaptively changed and arranged in one or more devices different from the embodiments. The modules or units or components in the embodiments may be combined into one module or unit or component, and in addition may be divided into multiple submodules or subunits or subcomponents. All features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed herein may be combined in any combination, except that at least some of such features and / or processes or units are mutually exclusive. Unless expressly stated otherwise, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0088] Furthermore, those skilled in the art will appreciate that although some embodiments herein include certain features included in other embodiments but not other features, combinations of features from different embodiments are intended to be within the scope of the present invention and to form different embodiments. For example, in the claims below, any of the claimed embodiments may be used in any combination.

[0089] It should be noted that the above embodiments illustrate rather than limit the invention, and that alternative embodiments may be devised by a person skilled in the art without departing from the scope of the appended claims. In the claims, any reference signs placed between brackets should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising several different elements and by means of appropriately programmed computers. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third etc. does not indicate any order. These words may be interpreted as names. The steps in the above embodiments should not be understood as limiting the order of execution unless otherwise specified.

Claims

1. A robot comprising a memory and a processor, wherein the memory stores computer-readable instructions that can be run on the processor, and the robot is equipped with a multi-camera, characterized in that: The processor is configured to implement the following steps when calling and executing the computer-readable instructions: Collect motion data of the robot during a cruise cycle to obtain N sets of basic data, where each set of basic data is data at the same time point, including instantaneous angular velocity, instantaneous linear velocity, a first image, and a second image, where the first image is obtained by capturing the left camera of the multi-camera, and the second image is obtained by capturing the right camera of the multi-camera; Determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data; Using each of the first images to construct a first image set, and based on each of the second images, constructing a second image set, and performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features; Determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture; Constructing a positioning map using the position information at each moment and the first image and the second image corresponding to the moment; The step of performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features includes: For each of the first image set and the second image set, feature matching is performed on two adjacent frames of images captured at the same time in the same set to obtain a first matching feature, and feature matching is performed on the image at the first moment and the image at the last moment in the same set to obtain a second matching feature; using the first matching feature and the second matching feature as the basic matching features; Determining the real-time spatial position of the robot at each moment based on the basic matching features and the real-time posture, as position information, includes: determining a first pose according to the first matching feature and an image corresponding to the first matching feature; Determine a second posture and a third posture according to the image at the first moment and the image at the last moment corresponding to the second matching feature and the second matching feature; Based on the first posture, the second posture and the third posture, a triangulation technique is used to calculate the real-time spatial position of the first matching feature at the corresponding moment as the position information corresponding to the first matching feature.

2. The robot according to claim 1, characterized in that Determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data includes: Based on the instantaneous angular velocity, calculating the average angular velocity within a preset time t; For any moment within the cruising cycle T, the real-time posture of the robot is calculated using the average angular velocity, the instantaneous angular velocity, and the instantaneous linear velocity.

3. The robot according to claim 1, characterized in that Determining the real-time spatial position of the robot at each moment based on the basic matching features and the real-time posture, as position information, includes: Optimizing the real-time posture of the robot using a posture graph optimization method to obtain an optimized posture; The real-time spatial position of the robot is determined according to the first matching feature, the second matching feature and the optimized posture.

4. The robot according to claim 3, characterized in that The method of optimizing the real-time posture of the robot by using the posture graph optimization method to obtain the optimized posture includes: The optimization equation is as follows: Where Log represents the logarithmic mapping from matrix to vector, represents the pose at the i-th moment.

5. The robot according to claim 1, characterized in that The left camera and the right camera are symmetrically installed on the robot, and both the left camera and the right camera are tilted upward by 30 degrees.

6. The robot according to claim 1, characterized in that The constructing of a positioning map by using the position information at each moment and the first image and the second image corresponding to the moment includes: Minimize the reprojection of the position information by the LM algorithm to obtain optimized position information; A positioning map is constructed using the optimized position information and the first image and the second image corresponding to the moment, and the first image and the second image are stored in a database corresponding to the positioning map.

7. A mapping and positioning method, characterized in that: include: Collect motion data of the robot within a cruising cycle T to obtain N sets of basic data, where each set of basic data is data at the same time point, including instantaneous angular velocity, instantaneous linear velocity, a first image, and a second image, where the first image is obtained by taking pictures with the robot's left camera, and the second image is obtained by taking pictures with the robot's right camera; Determining the real-time posture of the robot based on the instantaneous angular velocity and the instantaneous linear velocity in each set of basic data; Using each of the first images to construct a first image set, and based on each of the second images, constructing a second image set, and performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features; Determining the real-time spatial position of the robot at each moment as position information based on the basic matching features and the real-time posture; Constructing a positioning map using the position information at each moment and the first image and the second image corresponding to the moment; The step of performing feature extraction and matching using images in the first image set and the second image set to obtain basic matching features includes: For each of the first image set and the second image set, feature matching is performed on two adjacent frames of images captured at the same time in the same set to obtain a first matching feature, and feature matching is performed on the image at the first moment and the image at the last moment in the same set to obtain a second matching feature; using the first matching feature and the second matching feature as the basic matching features; Determining the real-time spatial position of the robot at each moment based on the basic matching features and the real-time posture, as position information, includes: determining a first pose according to the first matching feature and an image corresponding to the first matching feature; Determine a second posture and a third posture according to the image at the first moment and the image at the last moment corresponding to the second matching feature and the second matching feature; Based on the first posture, the second posture and the third posture, a triangulation technique is used to calculate the real-time spatial position of the first matching feature at the corresponding moment as the position information corresponding to the first matching feature.

8. A computer-readable storage medium, characterized in that Computer-readable instructions are stored thereon, and when the computer-readable instructions are executed by a processor, the mapping and positioning method according to claim 7 is implemented.

Images