Robot and positioning method, device, equipment, server and storage medium thereof

Abstract

The present application discloses a robot positioning method. The robot positioning method comprises the following steps: obtaining current environment data while the robot moves in a physical space based on a first map; matching the environment data with the first map, and selecting, based on the matching result, to obtain the current positioning information of the robot mapped in the first map based on the environment data and the first map, or to obtain the current positioning information of the robot mapped in the first map based on the environment data and the second map; wherein the second map has an association relationship with the first map, and obtaining the positioning information of the robot mapped in the first map based on the environment data and the second map comprises: converting the current positioning information of the robot mapped in the second map into the positioning information in the first map based on the association relationship.

Description

Technical Field

[0001] The present application relates to the field of robot technology, and specifically to a robot and a positioning method thereof, a robot control device, a computer storage medium, an electronic device, and a server. Background Art

[0002] With the development of automation technology and artificial intelligence, robots are widely used in various occasions to replace manual work. For example, robots replace manual cleaning of floor surfaces. As a machine device that performs work automatically, robots need to determine their position and posture in order to perform the next action during the execution of tasks.

[0003] Robots usually use SLAM (Simultaneous Localization and Mapping) technology to build and locate maps. However, SLAM technology positioning and mapping are carried out simultaneously. When the surrounding environment changes little or infrequently, in order to reduce the amount of calculation and improve positioning efficiency, the map built once is usually saved for positioning in the next mission. In some scenarios, the saved map is, for example, a visual map built based on visual SLAM technology. Since visual SLAM technology relies on image data, it is difficult for robots to locate through the saved visual map when the lighting conditions in the environment change. In other scenarios, the robot is equipped with a laser device, and the saved map is, for example, a laser map built based on laser SLAM technology. Although the data collected by the laser device can better reflect the real physical environment, when the objects in the environment change, for example, the position of the original objects in the environment changes or new objects are added to the environment, the robot will not be able to locate or the positioning accuracy will be unreliable based on the saved laser map.

[0004] Therefore, there is an urgent need to provide a positioning method that can both improve positioning accuracy and ensure positioning efficiency. Summary of the invention

[0005] In view of the shortcomings of the above-mentioned related technologies, the purpose of the present application is to provide a robot and a positioning method thereof, a robot control device, a computer storage medium, an electronic device, and a server, so as to overcome the technical problems of poor positioning accuracy and low efficiency in the above-mentioned related technologies.

[0006] To achieve the above-mentioned purpose and other related purposes, the first aspect disclosed in the present application discloses a robot positioning method, comprising the following steps: while the robot moves in a physical space based on a first map, obtaining current environmental data; matching the environmental data with the first map to select, based on the matching result, obtaining the positioning information of the current robot mapped in the first map based on the environmental data and the first map, or obtaining the positioning information of the current robot mapped in the first map based on the environmental data and the second map; wherein the second map has an association relationship with the first map, and obtaining the positioning information of the robot mapped in the first map based on the environmental data and the second map includes: converting the positioning information of the current robot mapped in the second map into positioning information in the first map based on the association relationship.

[0007] The second aspect disclosed in the present application discloses a control device for a robot, comprising an interface device for communicating data with the robot; a storage device for storing at least one program; and a processing device connected to the storage device and the interface device, for executing the at least one program to execute and implement the robot positioning method as described in any embodiment disclosed in the first aspect of the present application.

[0008] The third aspect disclosed in the present application discloses a robot, comprising: a sensor device for acquiring environmental data; a mobile device for performing mobile operations; a storage device for storing at least one program; a processing device connected to the sensor device, the mobile device, and the storage device, and used to execute the at least one program to execute the robot positioning method as described in any embodiment disclosed in the first aspect of the present application.

[0009] The fourth aspect disclosed in the present application discloses a computer storage medium storing at least one computer program, which, when executed by a processor, controls a device where the storage medium is located to execute a robot positioning method as described in any embodiment disclosed in the first aspect of the present application.

[0010] The fifth aspect disclosed in the present application discloses an electronic device, comprising: an interface device for communicating data with a robot; a storage device for storing at least one program; a processing device connected to the storage device and the interface device, for executing the at least one program to execute and implement the robot positioning method as described in any embodiment disclosed in the first aspect of the present application.

[0011] The sixth aspect disclosed in the present application discloses a server, comprising: an interface device for communicating data with a robot; a storage device for storing at least one program; a processing device connected to the storage device and the interface device, for executing the at least one program to execute and implement the robot positioning method as described in any embodiment disclosed in the first aspect of the present application.

[0012] In summary, the present application discloses a robot and a positioning method thereof, a robot control device, a computer storage medium, an electronic device, and a server. By setting a first map as a basic map to provide a navigation and movement basis for the robot, when the positioning accuracy of the first map is high, that is, when the environmental data and the feature matching consistency of the first map are high, the environmental data are matched with the first map to perform positioning and navigation movement, etc. When the positioning accuracy of the first map is low, that is, when the matching consistency of the environmental data and the first map is low, the robot obtains positioning information based on the environmental data and the second map, and the association between the first map and the second map, and the more accurate positioning information on the second map is mapped to the positioning information in the first map, so that the robot continues to navigate and move based on the first map with more accurate positioning. In this way, when the positioning based on the first map is inaccurate, the positioning information obtained based on the second map is used as a supplement, so that it can be achieved while improving the positioning efficiency and reducing the amount of calculation, and also reducing the probability of positioning failure or error.

[0013] Those skilled in the art can easily perceive other aspects and advantages of the present application from the detailed description below. In the detailed description below, only exemplary embodiments of the present application are shown and described. As will be appreciated by those skilled in the art, the content of the present application enables those skilled in the art to modify the disclosed specific embodiments without departing from the spirit and scope of the invention to which the present application relates. Accordingly, the description in the drawings and specification of the present application is merely exemplary and not restrictive. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The specific features of the invention involved in this application are shown in the attached claims. The features and advantages of the invention involved in this application can be better understood by referring to the exemplary embodiments and drawings described in detail below. A brief description of the drawings is as follows:

[0015] Figure 1 Shown is a flow chart of a robot positioning method in one embodiment of the present application.

[0016] Figure 2 It is a flowchart of step S120 in one embodiment of the present application.

[0017] Figure 3Shown is a flow chart of determining the coordinate conversion relationship between the first map and the second map in one embodiment of the present application.

[0018] Figure 4 It is a schematic diagram showing how to reposition each key frame in a first map in a second map in one embodiment of the present application.

[0019] Figure 5 It is a schematic diagram showing how to reposition each key frame in the second map in the first map in one embodiment of the present application.

[0020] Figure 6 Shown is a flow chart of a method for a robot to construct a map in one embodiment of the present application.

[0021] Figure 7 It is a flowchart of step S320 in one embodiment of the present application.

[0022] Figure 8 Shown is a flowchart of step S330 in one embodiment of the present application.

[0023] Fig. 9 Shown is a schematic diagram of determining a key frame image and simultaneously recording the positioning information of the robot mapped onto the first map in one embodiment of the present application.

[0024] Fig.10 FIG. 4 is a schematic diagram showing a re-projection error in one embodiment of the present application.

[0025] Fig.11 Shown is a flow chart of a method for updating a map by a robot in one embodiment of the present application.

[0026] Fig.12 Shown is a flowchart of step S430 in one embodiment of the present application.

[0027] Fig.13 Shown is a schematic diagram of the structure of a robot in one embodiment of the present application.

[0028] Fig.14 Shown is a schematic structural diagram of an electronic device in one embodiment of the basic application.

[0029] Fig.15 Shown is a schematic diagram of the structure of a server in an embodiment of the basic application. DETAILED DESCRIPTION

[0030] The following is an explanation of the implementation of the present application by means of specific embodiments. People familiar with the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification.

[0031] In the following description, reference is made to the accompanying drawings, which describe several embodiments of the present application. It should be understood that other embodiments may also be used, and that changes in module or unit composition, electrical and operational aspects may be made without departing from the spirit and scope of the present disclosure. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present application is limited only by the claims of the published patents. The terms used herein are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0032] Although the terms first, second, etc. are used to describe various elements or parameters in this article in some instances, these elements or parameters should not be limited by these terms. These terms are only used to distinguish one element or parameter from another element or parameter. For example, the first map can be referred to as the second map, and similarly, the second map can be referred to as the first map, without departing from the scope of the various described embodiments. The first map and the second map are both describing a map, but unless the context clearly indicates otherwise, they are not the same map.

[0033] Furthermore, as used in this article, the singular forms "one", "an" and "the" are intended to include plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, steps, operations, elements, components, projects, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, steps, operations, elements, components, projects, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions, steps or operations is inherently mutually exclusive in some way.

[0034] As described in the background technology, considering the amount of calculation and positioning efficiency, the method of using the constructed and saved map to locate the robot will usually result in unreliable or impossible positioning of the robot due to environmental changes. Specifically, in the method of positioning based on the saved visual map built based on SLAM technology, it is more sensitive to changes in light in the environment. In the same scene with different light, the frame images obtained by the robot are not the same, which makes it difficult to locate; and in the method of positioning based on the saved laser map, it is more sensitive to changes in the position of the object. Once an object moves or a new object is added in the space, the robot will not be able to determine its own position based on the data detected by the laser device.

[0035] In order to solve the impact of light changes on positioning based on visual maps, in some embodiments, the pre-built and saved map is constructed by extracting image features through machine learning methods. In positioning, the frame image is also processed through machine learning. Although this reduces the impact of light changes on positioning, the amount of calculation is large and the positioning speed is very slow, resulting in the problem of large computing load and non-instant positioning.

[0036] In view of this, the present application discloses a robot and a positioning method thereof, a robot control device, a computer storage medium, an electronic device, and a server. In the present application, a first map is set as a basic map to provide a navigation and movement basis for the robot. When the positioning accuracy of the first map is high, that is, when the environmental data and the feature matching consistency of the first map are high, the environmental data are matched with the first map to perform positioning and navigation movement, etc. When the positioning accuracy of the first map is low, that is, when the matching consistency of the environmental data and the first map is low, the robot obtains positioning information based on the environmental data and the second map, and the association between the first map and the second map, and the more accurate positioning information on the second map is mapped to the positioning information in the first map, so that the robot continues to navigate and move based on the first map with more accurate positioning. In this way, when the positioning based on the first map is inaccurate, the positioning information obtained based on the second map is used as a supplement, so that it can be achieved while improving the positioning efficiency and reducing the amount of calculation, and also reducing the probability of positioning failure or error.

[0037] The physical space described in this application refers to the actual three-dimensional space where the mobile robot is located, which can be described by abstract data constructed in the spatial coordinate system. For example, the physical space includes but is not limited to home residences, public places (such as offices, shopping malls, hospitals, underground parking lots, and banks), etc. For mobile robots, the physical space usually refers to indoor space, that is, the space has boundaries in length, width, and height. In particular, it includes physical spaces with large spatial ranges and high scene repetition, such as shopping malls and terminal halls.

[0038] The first map described in the present application may, for example, be a pre-built laser map or visual map corresponding to the physical space. The second map described in the present application may also, for example, be a laser map or visual map corresponding to the physical space. In order to distinguish the first map from the second map, in some embodiments or examples where the first map corresponds to a laser map or a visual map, the laser map is also referred to as a first laser map, and the visual map is also referred to as a first visual map. In some embodiments or examples where the second map corresponds to a laser map or a visual map, the laser map is also referred to as a second laser map, and the visual map is also referred to as a second visual map.

[0039] In some embodiments, the first laser map is a map constructed by the robot based on laser SLAM technology when it moves in the physical space, such as a grid map or a feature map. For example, the robot is equipped with a laser device. When the robot needs to construct a first map, such as when it is placed in the current physical space for the first time, the robot controls its laser device to scan the surrounding environment during movement and constructs the first laser map of the current physical space based on the point cloud data detected by the laser device. After the first laser map is constructed, it is saved for use in subsequent work of the robot. Of course, the first laser map can also be constructed using other technologies, and this application does not limit this. Since the laser map can more accurately reflect the real physical space and has high positioning efficiency, in an embodiment where the first map is set as the first laser map, the robot moves according to the first map during navigation, which can achieve efficient and high-precision positioning of most locations in the physical space based on the first map and environmental data.

[0040] In some embodiments, the first visual map is a map constructed based on VSLAM (Visual SLAM, visual-based SLAM, full name Visual Simultaneous Localization and Mapping) technology. Specifically, the robot is equipped with an image capture device. When the robot needs to build a first map, such as being placed in the current physical space for the first time, the robot controls the image capture device to obtain image data of the surrounding environment during movement and builds a first visual map of the current physical space based on the image data. After the first visual map is built, it is saved for use in subsequent work of the robot. Therefore, in the example where the first map is set as the first visual map, since the map built based on VSLAM has the advantage of fast speed in robot positioning, the robot moves according to the map built based on VSLAM as the first map during navigation, which can ensure efficient positioning based on the first map and environmental data in most scenarios.

[0041] In some embodiments, the second map is a pre-constructed second laser map or a second visual map. In this embodiment, the coordinate transformation relationship between the second map and the first map is determined as the association relationship between the first map and the second map. Among them, the specific construction method of the second laser map can refer to the aforementioned description of the first laser map, which will not be repeated here. Among them, in some examples, the specific construction method of the second visual map can also refer to the aforementioned description of the first visual map; in other examples, the second visual map is set to a map constructed based on machine learning to extract image features. The map is highly robust to environmental changes, but because it involves processing frame images in a machine learning manner, the amount of calculation in composition and positioning is very large. Therefore, using the second visual map as a supplement to the positioning of the first map can improve the accuracy of robot positioning.

[0042] In some other embodiments, the second map is a second laser map or a second visual map constructed based on the first map. Thus, in this embodiment, the association relationship between the first map and the second map can be formed during the process of constructing the second map.

[0043] In some embodiments, the key frame described in the present application refers to the frame where the key action in the robot's motion change is located or a representative frame in the motion change. For example, the information redundancy between similar frames is high, and the key frame is a representative frame among the local similar frames. For example, the robot does not move in the original place. Although it will obtain ordinary frames, its key frames will not increase due to the lack of motion changes. Specifically, according to the different types of collected environmental data, the key frames correspond to different types of key frames. For example, the collected environmental data includes image data obtained by an image acquisition device, then the key frames correspond to visual key frames. In some embodiments or examples disclosed in the present application, visual key frames are also referred to as key frame images. For another example, the collected environmental data includes point cloud data obtained by a laser device, then the key frames correspond to key frame point clouds. It should be noted that the point cloud data that constitute the key frame point cloud are not generated at the same time point. Generally, the point cloud data accumulated within a fixed time length (such as 100ms) or a fixed rotation angle (such as one rotation of a single-line laser device) is used as a frame of point cloud. The key frame point cloud is the frame of point cloud where the key action is located or a representative frame of point cloud.

[0044] In some embodiments, the feature information described in the present application includes feature points, feature lines, etc. For example, the feature information can be represented by a descriptor. The positioning information described in the present application is used to represent the position and posture of the robot in the physical space. Considering that the robot has the same or different positions and postures in the physical space at different moments, the positioning information of the frame image, frame point cloud, key frame, key frame image, or key frame point cloud is also used to represent the position and posture of the robot in the physical space at a certain moment. For example, the positioning information of the frame image is used to represent the position and posture of the robot in the physical space at the moment corresponding to the frame image, or it can also be understood that the positioning information of the frame image is used to represent the position and posture of the robot in the physical space when the frame image is acquired. Taking into account the different reference bases, the positioning information mapped on the first map or the second map is also used to indicate which map is used as the basis for mapping the robot's position and posture in the physical space. At the same time, taking into account the above-mentioned different moments and different reference bases, the robot's position in the physical space at a certain moment can also be mapped to the position on the first map or the second map through the positioning information of the frame image, frame point cloud, key frame, key frame image, or key frame point cloud on the first map or the second map. For example, the positioning information of the key frame image on the first map indicates that the robot's position in the physical space at the moment corresponding to the key frame image is mapped to the position in the first map. Alternatively, it can also be understood that the robot's position is mapped to the first map when the key frame image is acquired.

[0045] The robot described in the present application is a mobile device equipped with a laser device and / or an image capture device, which is used to perform operations based on the laser device and / or the image capture device. The operations performed include work tasks in application scenarios, as well as the robot positioning method disclosed in the present application, the method for robot mapping, and the method for robot updating maps. The robot refers to an autonomous mobile device with the ability to build a map in a physical space, including but not limited to: a drone, an industrial robot, a family companion mobile device, a medical mobile device, a household cleaning robot, a commercial cleaning robot, an intelligent vehicle, and a patrol robot. In different application scenarios, the robot can be configured to perform corresponding tasks. For example, the robot is used to complete floor cleaning work indoors. In this application scenario, the robot can also be called a cleaning robot, a floor washing robot, an automatic cleaning robot, etc. In other indoor scenarios, the robot can also be a family companion mobile robot, a patrol robot, or a robot for delivering meals / items.

[0046] In some embodiments, the present application discloses a robot positioning method. In some examples, the robot positioning method is executed by a robot, and further, can be executed by a control device configured on the robot. In some examples, the robot positioning method can also be executed by a processing device configured on a server, and the server and the robot can communicate data to control the robot to perform corresponding actions when executing the robot positioning method disclosed in the present application. In other examples, the robot positioning method can also be executed by a processing device configured on an electronic device, and the electronic device communicates data with the robot to control the robot to perform corresponding operations when executing the robot positioning method disclosed in the present application. The following embodiments are described by taking the robot positioning method being executed by a control device configured on the robot as an example.

[0047] See also Figure 1 , which is a flow chart of a robot positioning method in one embodiment of the present application. As shown in the figure, the robot positioning method includes: step S110 and step S120.

[0048] In step S110 , the control device obtains current environment data while the robot moves in the physical space based on the first map.

[0049] Corresponding to the first map and the second map type settings, the environmental data may include at least one of point cloud data or image data. In one embodiment, the first map and the second map are both set as laser maps, and the environmental data includes point cloud data acquired by a laser device. In another embodiment, the first map and the second map are both set as visual maps, and the environmental data includes image data acquired by an image capture device. In some other embodiments, one of the first map and the second map is set as a laser map and the other is set as a visual map, and the environmental data includes point cloud data and image data acquired by a laser device and an image capture device, respectively.

[0050] In one embodiment, the laser device is configured on the robot, and the laser device includes: a single-line laser radar or a multi-line laser radar, etc. The laser device can be controlled by the control device. During the movement of the robot, the control device controls the laser device to emit a laser beam at a certain rotation angle, so that point cloud data of the surrounding environment can be obtained. Considering that the single-point point cloud data corresponding to a laser beam cannot usually be used alone to reflect the real physical space situation, in some examples, the point cloud data accumulated within a fixed time length (such as 100ms) or the point cloud data accumulated within a fixed rotation angle (such as a single-line laser radar rotates one circle) is used as a frame of point cloud, and each frame of point cloud is set with a timestamp. The control device obtains point cloud data in a frame manner, and when the control device obtains each frame of point cloud, it also obtains the timestamp corresponding to the frame of point cloud.

[0051] In one embodiment, the image capture device is configured on the robot, including: a camera, a video camera, a camera module integrated with an optical system or a CCD chip, a camera module integrated with an optical system and a CMOS chip, etc. The image capture device can be controlled by the control device. During the movement of the robot, the control device controls the image capture device to capture the surrounding environment, thereby obtaining image data of the surrounding environment. The image data includes frame images, and each frame image is provided with a timestamp. The control device obtains the image data in the form of frame images, that is, the image capture device sends the obtained image data to the control device in the form of frames. In some examples, the image capture device sends all acquired frame images to the control device, such as the image capture device sends each acquired frame image to the control device; in some examples, the image capture device sends frame images at fixed time intervals to the control device, such as the image capture device acquires a frame image at t0, t1, t2, t3, and t4 respectively, and it only sends the frame images acquired at t0, t2, and t4 to the control device; in other examples, the image capture device sends frame images to the control device at a fixed frame interval, such as the image capture device acquires continuous frame images, and it sends the first frame image, the fifth frame image, the ninth frame image, the thirteenth frame image, ... to the control device at a frame interval of three frames. The above are only examples, and this application does not limit how the image capture device sends frame images to the control device.

[0052] It should be understood that in step S110, the current environmental data includes at least one of the current point cloud data and the current image data. Among them, the current point cloud data includes at least one frame of point cloud, and the current image data includes at least one frame of image, that is to say, the current environmental data includes at least one frame of point cloud and at least one frame of image. In an embodiment where both the first map and the second map are set as laser maps, the current environmental data includes at least one frame of point cloud. In an embodiment where both the first map and the second map are set as visual maps, the current environmental data includes at least one frame of image. In an embodiment where one of the first map and the second map is set as a laser map and the other is set as a visual map, the current environmental data includes at least one frame of point cloud and at least one frame of image.

[0053] Since the laser device and the image capture device have their own sampling frequency characteristics, the at least one frame of point cloud and / or at least one frame of image included in the current environmental data is not necessarily obtained at the current moment, nor is it necessarily obtained at the same moment. It only needs to be guaranteed that it is obtained within a time period with the current moment as the cutoff point to be used as the current environmental data. In other words, during the movement of the robot in the physical space based on the first map, the current environmental data acquired by the control device includes at least one frame of point cloud and / or at least one frame of image acquired within a period of time with the current moment as the cutoff point. Wherein, the time period can be, for example, a fixed value. In one example, the current environmental data includes at least one frame of point cloud and at least one frame of image. The fixed value can be set to the maximum value in the frame sampling period of the laser device and the image capture device, and the frame of point cloud and the frame of image collected closest to the current moment within the time range are used as the current environmental data. Of course, those skilled in the art can design the range of the time period by themselves according to the different hardware devices actually used and the different specific application scenarios, and this application does not limit this.

[0054] It should be noted that the environmental data acquired before a period of time with the current moment as the end point is the historical environmental data. In some embodiments or examples of the present application, in order to indicate the time relationship between the environmental data of a certain time period in the historical environmental data and the current environmental data, an identifier reflecting the sequence is added to the environmental data. For example, the previous environmental data is used to indicate the historical environmental data acquired before the current environmental data. Furthermore, depending on the type of environmental data, the environmental data may be replaced with the corresponding data it contains, such as the previous frame image, the previous frame point cloud, etc., without changing the added identifier reflecting the sequence.

[0055] It should be noted that the type of data included in the current environment data is not limited to the above embodiment, and the environment data only needs to include data that is consistent with the first map and the second map type respectively.

[0056] Please continue reading Figure 1 In step S120, the control device matches the environmental data with the first map to select, according to the matching result, to obtain the positioning information of the current robot mapping in the first map based on the environmental data and the first map, or to obtain the positioning information of the current robot mapping in the first map based on the environmental data and the second map.

[0057] See also Figure 2 , which is a flowchart of step S120 in one embodiment of the present application. As shown in the figure, step S120 includes step S121, step S122, and step S123 or step S124.

[0058] In step S121, the control device matches the current environmental data with the first map. Specifically, when the control device matches the current environmental data with the first map, the data of the type consistent with the first map in the current environmental data is matched with the first map. For example, in an example where the first map is a laser map, the control device matches at least one frame of point cloud in the current environmental data with the first map; in an example where the first map is a visual map, the control device matches at least one frame of image in the current environmental data with the first map.

[0059] Taking the first map set as the first visual map as an example, the control device matches the current environmental data with the first map, that is, feature matching of the at least one frame of image with the positioning data set of the first map. The positioning data set includes keyframe images and landmark points. The keyframe images include feature information and positioning information. The positioning information is used to indicate the position and posture of the robot when acquiring the keyframe image. In other words, it can indicate the observation relationship between the landmark point and its corresponding keyframe image. Among them, the landmark point is a three-dimensional point in the physical space mapped from the feature point in the keyframe image, and the feature points of different keyframe images can be mapped to the same landmark point.

[0060] Specifically, the control device performs feature matching on the at least one frame image and the positioning data set of the first map to determine the key frame image in the positioning data set that matches the frame image and the feature matching relationship between the frame image and the key frame image. The matched key frame image refers to a key frame that has similarity with the at least one frame image, for example, a key frame that has an overlapping area with the at least one frame image is determined to have similarity, that is, a key frame that has an overlapping area with the at least one frame image is determined to be a matched key frame image. The feature matching relationship includes a one-to-one pairing relationship of feature points, or a one-to-one pairing relationship between other descriptors that embody feature information, which is not limited in this application.

[0061] Taking the first map as the first laser map as an example, the current environment data acquired by the control device includes at least one frame of point cloud, and the control device matches the current environment data with the first map, that is, matches the at least one frame of point cloud with the first map. In one example, the laser map is set as a grid map, and the control device compares the at least one frame of point cloud with the grid map, for example, by brute force matching, so as to obtain the confidence of the frame of point cloud on the grid map, and the confidence is used to represent the probability of the accurate posture obtained based on the comparison between the frame of point cloud and the grid map.

[0062] Please continue reading Figure 2In step S122, the control device determines whether the matching result of the feature matching between the current environment data and the first map meets the preset matching condition. If it is determined that the matching condition is met, step S123 is executed, and if it is determined that the matching condition is not met, step S124 is executed.

[0063] In some embodiments, the matching condition includes that the number of key frames in the first map that match the current environment data reaches a set threshold. For example, in an embodiment in which the first map is set as the first visual map in step S121, a matching key frame image can be obtained, and the control device compares the number of matching key frame images with the set threshold. When the number reaches the threshold, it is considered that the matching condition is met, and when the number does not reach the threshold, it is considered that the matching condition is not met. Of course, the first map can also be set as the first laser map, and the control device compares the number of matching key frame point clouds with the set threshold. When the number reaches the threshold, it is considered that the matching condition is met, and when the number does not reach the threshold, it is considered that the matching condition is not met.

[0064] In some embodiments, the matching condition includes that the number of pairs of feature points reaches a set threshold. For example, in an embodiment in which the first map is set as the first visual map in step S121, the feature matching relationship between a frame image and a key frame image can be determined, and the control device determines that the number of pairs of feature points reaches the set threshold based on the feature matching relationship, that is, it is considered that the matching condition is met, otherwise, it is considered that the matching condition is not met. For example, if the set threshold is 100 pairs, then when it is determined that the number of pairs of feature points reaches 100 pairs, it is considered that the matching condition is met, and when the number of pairs of feature points does not reach 100 pairs, it is considered that the matching condition is not met.

[0065] In some embodiments, the matching condition includes that the confidence reaches a set threshold. For example, in an embodiment in which the first map is set as a grid map in step S121, the confidence of a frame of point cloud in the current environment data on the first map can be obtained, and the control device further compares the confidence with the set threshold. When the confidence reaches the set threshold, it is determined that the matching condition is met, otherwise, it is considered that the matching condition is not met.

[0066] The matching conditions described in any of the embodiments of step S122 above are only examples. Those skilled in the art may set different matching conditions or combine the matching conditions of any of the above embodiments to set new matching conditions according to the type of map actually set and the different matching methods. This application does not impose any restrictions on this.

[0067] Please continue reading Figure 2When the matching result between the current environment data and the first map is determined to meet the matching condition based on step S122, it means that the positioning of the robot according to the first map is reliable, and then step S123 is executed. In step S123, the control device obtains the positioning information of the robot currently mapped in the first map based on the current environment data and the first map.

[0068] In one embodiment, in step S123, the control device matches the current environment data with the first map to obtain the positioning information of the robot currently mapped in the first map. Here, the process of matching the current environment data with the first map may include part of the process in step S121. For example, in the embodiment in which the first map is set as the first visual map in step S121, a matching key frame image and a feature matching relationship can be obtained. Then, in step S123, the control device will further determine the position of the robot when acquiring the current environment data according to the matching key frame image, the feature matching relationship, and the landmark points corresponding to the key frame image in the positioning data set of the first map, using the PnP (Perspective-n-Point) method, that is, the positioning information of the robot currently mapped in the first map. For another example, in the embodiment in which the first map is set as the first laser map in step S121, the current environment data will be compared with the first map. Then, in step S123, the control device will further determine the positioning information of the current robot mapped on the first map according to the comparison result.

[0069] It should be noted that, in some embodiments, the step S123 further includes optimizing the positioning information of the current robot mapped in the first map. In some examples, the control device further optimizes the positioning information of the current robot mapped in the first map with the reprojection error as a constraint item, so that the optimized positioning information minimizes the reprojection error, wherein the definition of the reprojection error will be described in detail later.

[0070] Please continue reading Figure 2 When it is determined according to step S122 that the matching condition is not met, it means that the positioning of the robot according to the first map is unreliable, and then step S124 is executed. In step S124, the control device obtains the positioning information of the robot currently mapped in the first map based on the current environment data and the second map. The second map is associated with the first map.

[0071] In one embodiment, the step S124 includes: based on the association relationship, converting the positioning information of the robot currently mapped in the second map into the positioning information in the first map. The positioning information of the robot currently mapped in the second map is determined by the control device based on matching the current environment data with the second map.

[0072] In one embodiment, the control device matches the current environment data with the second map by matching the data of the type consistent with the second map in the current environment data with the second map. For example, in an example where the second map is a laser map, the control device matches at least one frame of point cloud in the current environment data with the second map; in an example where the second map is a visual map, the control device matches at least one frame of image in the current environment data with the second map.

[0073] In one embodiment, the second map is set as a second visual map, and the control device matches the current environmental data with the second map by performing feature matching on the at least one frame of image and the positioning data set of the second map. Specifically, the control device performs feature matching on the at least one frame of image and the positioning data set of the second map to obtain the posture of the robot in obtaining the at least one frame of image in the current environmental data based on the PnP method, that is, the positioning information of the robot currently mapped in the second map. In other embodiments where the second map is set as a second visual map, the control device matches the at least one frame of image with the second map by machine learning, thereby obtaining the positioning information of the robot currently mapped in the second map.

[0074] In one embodiment, the second map is set as a second laser map, and the control device matches the current environment data with the second map, that is, matches the at least one frame of point cloud with the second map. In one example, the second laser map is set as a grid map, and the control device compares the at least one frame of point cloud with the grid map, for example, by brute force matching, so as to obtain the pose information of the frame of point cloud on the grid map, that is, the positioning information of the current robot mapped in the second map.

[0075] It should be understood that there are various ways to obtain the positioning information of the current robot mapped in the second map based on the current environmental data and the second map. What is described in the above embodiments of the present application is only an example and should not be understood as a limitation to the present application.

[0076] In one embodiment, the association relationship between the first map and the second map is set as a coordinate conversion relationship between the first map and the second map. In this embodiment, the control device converts the current positioning information of the robot in the second map into the positioning information in the first map according to the coordinate conversion relationship. The coordinate conversion relationship is determined by relocating each key frame of the first map and each key frame of the second map. In view of this, in some embodiments, the robot positioning method disclosed in the present application also includes: relocating each key frame of the first map and each key frame of the second map to determine the coordinate conversion relationship between the first map and the second map.

[0077] See also Figure 3 , which is a flowchart of determining the coordinate transformation relationship between the first map and the second map in one embodiment of the present application. As shown in the figure, each key frame of the first map and each key frame of the second map are repositioned to determine the coordinate transformation relationship between the first map and the second map, including step S210 and step S220.

[0078] In step S210, the control device relocates each key frame in the first map in the second map, and relocates each key frame in the second map in the first map, so as to obtain a paired data set. According to the types of the first map and the second map, each key frame in the first map and the second map may be, for example, a key frame image or a key frame point cloud.

[0079] The paired data set includes positioning information of the key frame that is successfully relocated on the first map and positioning information on the second map. Considering that the relocation in the second map and the relocation in the first map are performed separately in step S210, in some embodiments, the paired data set includes first paired data and second paired data, the first paired data includes positioning information of the key frame that is successfully relocated in the second map on the first map and the second map respectively, and the second paired data includes positioning information of the key frame that is successfully relocated in the first map on the first map and the second map respectively.

[0080] The following describes the relocation on the first map and the second map respectively with reference to the figures.

[0081] See also Figure 4, showing a schematic diagram of relocating each key frame in the first map in the second map in one embodiment of the present application, in which the key frames in the first map are illustrated as three frames. As shown in the figure, node 1 in the first map is used to represent the position (i.e., positioning information) of the first key frame in the first map, and the position of the first key frame after relocation in the second map (i.e., the positioning information of the first key frame on the second map) is represented by node 1'; node 2 is used to represent the position of the second key frame in the first map, and the position of the second key frame after relocation in the second map (i.e., the positioning information of the second key frame on the second map) is represented by node 2'; node 3 is used to represent the position of the third key frame in the first map, and the position of the third key frame after relocation in the second map (i.e., the positioning information of the third key frame on the second map) is represented by node 3', then Figure 4 The first pairing data includes: (node ​​1, node 1'), (node ​​2, node 2'), (node ​​3, node 3').

[0082] See also Figure 5 , showing a schematic diagram of relocating each key frame in the second map in the first map in one embodiment of the present application, as shown in the figure, the figure takes the key frames in the second map as an example for explanation, as shown in the figure, node 4 in the second map is used to represent the posture (i.e. positioning information) of the fourth key frame in the second map, and the posture of the fourth key frame after relocation in the first map (i.e. positioning information of the fourth key frame on the first map) is represented by node 4'; node 5 is used to represent the posture of the fifth key frame in the second map, and the posture of the fifth key frame after relocation in the first map (i.e. positioning information of the fifth key frame on the first map) is represented by node 5'; node 6 is used to represent the posture of the sixth key frame in the second map, and the posture of the sixth key frame after relocation in the first map (i.e. positioning information of the sixth key frame on the first map) is represented by node 6', then Figure 5 The second pairing data includes: (node ​​4, node 4'), (node ​​5, node 5'), (node ​​6, node 6').

[0083] Please continue reading Figure 3 In step S220, the control device determines a coordinate conversion relationship between the first map and the second map based on the paired data set.

[0084] In one embodiment, the control device determines the coordinate transformation relationship between the first map and the second map based on the positioning information of the key frame in the paired data set on the first map and the positioning information on the second map. For example, the control device determines the transformation matrix between the point pair (i.e., the positioning information of the key frame on the first map and the positioning information on the second map are a point pair) through the ICP (Iterative Closest Point) algorithm as the coordinate transformation relationship between the first map and the second map. Of course, under the inspiration of this application, those skilled in the art may also use other data registration algorithms to determine the coordinate transformation relationship between the two maps, and this application does not limit this calculation method.

[0085] In the case where the difference between the first map and the second map is too large, even if the coordinate conversion relationship between the two maps is obtained to convert the positioning information on the second map into the positioning information on the first map, the positioning information on the first map obtained is unreliable. In view of this, in some embodiments, before the step S220, it also includes: a step of judging whether the paired data set meets the preset condition, and when the paired data set is judged to meet the preset condition, step S220 is executed. Specifically, in one example, the preset condition is set to the number of key frames of successful relocation in the paired data set reaches a preset threshold. When the number of key frames of successful relocation is judged to reach the preset threshold according to the paired data set, the control device continues to execute step S220. When the number of key frames of successful relocation is judged to not reach the preset threshold according to the paired data set, the control device no longer executes step S220. Further, in step S120, when the control device cannot obtain the association relationship between the second map and the first map (that is, there is no coordinate conversion relationship between the first map and the second map), it still chooses to obtain the positioning information of the current robot mapped in the first map based on the environmental data and the first map.

[0086] In another embodiment, the association between the first map and the second map is established during the construction of the second map based on the first map, so that the control device can use the positioning information of the robot currently mapped in the second map as the positioning information in the first map.

[0087] In view of this, in some embodiments, the present application also discloses a method for constructing a map by a robot, which can be completed before executing step S110 or in executing step 110 and step S120 as part of the robot positioning method disclosed in the present application, or it can be executed separately, and the second map successfully constructed is used as the second map in the robot positioning method disclosed in the present application. In some examples, the method for constructing a map by a robot is executed by a robot, and further, it can be executed by a control device configured on the robot. In some examples, the method for constructing a map by a robot can also be executed by a processing device configured on a server, and the server and the robot can communicate data to control the robot to perform corresponding actions when executing the method for constructing a map by a robot disclosed in the present application. In other examples, the method for constructing a map by a robot can also be executed by a processing device configured on an electronic device, and the electronic device communicates data with the robot to control the robot to perform corresponding operations when executing the method for constructing a map by a robot disclosed in the present application. The following embodiments are described by taking the method for constructing a map by a control device configured on a robot as an example.

[0088] See also Figure 6 , which is a flow chart of a method for constructing a map by a robot in one embodiment of the present application. As shown in the figure, the method for constructing a map by a robot includes step S310, step S320, and step S330.

[0089] In step S310, the control device acquires environmental data while the robot moves in the physical space based on the first map. The data types and acquisition methods of the environmental data are described in step S110 in the aforementioned robot positioning method, which will not be repeated here.

[0090] Among them, based on the environmental data, the positioning information of the robot mapped to the first map at different times can be obtained, so that the robot can move in the physical space based on the first map. In view of this, the method for constructing a map by the robot also includes: matching the first map with the current environmental data obtained to obtain the positioning information of the robot currently mapped to the first map. Among them, the process and method of obtaining the current position information of the robot mapped on the first map in this step are similar to step S123 in the aforementioned robot positioning method. Please refer to the description of step S123 in any of the aforementioned embodiments, which will not be repeated here.

[0091] Please continue reading Figure 6 In step S320, the control device analyzes the movement posture of the robot in the physical space based on the environmental data to obtain a key frame and its positioning information.

[0092] In one embodiment, before the step S320, it also includes: when it is determined that there is no initialization map according to the current environmental data, a step of constructing an initialization map. Specifically, taking the second map to be constructed as a visual map as an example, when it is determined that there is no initialization map according to the current environmental data, the step of constructing an initialization map includes: when it is determined that there is no initialization map according to the frame image in the current environmental data, an initialization map is constructed based on at least two frames of images. Among them, the initialization map refers to the constructed initial map, which provides the initial coordinate system and initial landmark points of the second map, and may also provide the correspondence between the coordinate system of the second map and the image coordinate system. In one example, the control device can perform feature recognition and matching on the frame image in the current environmental data (which can also be referred to as the current frame image) to determine whether there is an initialization map. When it is determined that there is no initialization map, the control device constructs the initialization map according to the position of the matching features of the current frame image and the previous frame image, and the movement information obtained from the previous frame image to the current frame image. It should be understood that when the current frame image is the first frame image, the posture of the robot corresponding to the frame image should be used as the starting point of the coordinate system for constructing the second map. The control device continues to acquire the current environmental data during the movement. At this time, the first frame image is the previous frame image, and the frame image in the current environmental data that continues to be acquired is the current frame image. The control device initializes the map construction based on the current frame image and the previous frame image at this time.

[0093] In order to make the coordinate system of the second map consistent with the coordinate system of the first map, in some embodiments, the step of constructing the initialization map based on at least two frames of images includes: using the positioning information of the robot mapped to the first map at the time corresponding to the first frame of image as the positioning information of the first frame of image. Specifically, as mentioned above, the position of the robot corresponding to the first frame of image is the starting point for constructing the coordinate system of the second map, and the positioning information of the robot mapped to the first map at the time corresponding to the first frame of image is used as the positioning information of the first frame of image to construct the initialization map, so that the initial coordinate system constructed by the initialization map corresponding to the second map is consistent with the coordinate system of the first map. In this way, it is ensured that the coordinate system of the second map formed by continuing to construct the local map based on the initialization map and finally forming the second map can be consistent with the coordinate system of the first map.

[0094] See also Figure 7 , which is a flow chart of step S320 in one embodiment of the present application. As shown in the figure, step S320 includes: step S321 and step S322.

[0095] In step S321, the control device analyzes the movement posture of the robot based on the current frame image and the previous frame image to determine the positioning information of the current frame image. In some examples, the control device first obtains the initial positioning information corresponding to the current frame image based on the current frame image and the previous frame image; and then optimizes the initial positioning information to obtain the positioning information of the current frame image. In some examples, the control device can track the previous frame image based on the current frame image to obtain the initial positioning information corresponding to the current frame image, or determine the relative displacement of the current frame image and the previous frame image according to the movement information of the robot to determine the initial positioning information corresponding to the current frame image. In some examples, the initial positioning information can be optimized by matching the current frame image with the constructed part of the second map (also referred to as a local map) to optimize the initial positioning information in a map optimization manner to obtain the positioning information corresponding to the current frame image.

[0096] In step S322, the control device judges the current frame image to determine whether to use it as the key frame image.

[0097] Specifically, in some examples, the control device determines that the current frame image is a key frame image when judging that the difference between the current frame image and the key frame image determined last time is a preset number of frames or a preset duration. For example, the preset number of frames is 20 frames. If the difference between the current frame image and the key frame image determined last time reaches 20 frames, the current frame image is determined as a key frame image. In this example, setting the preset number of frames or the preset duration as a judgment condition can indicate that the scene of the robot movement has changed, so it is necessary to add a key frame. In other examples, the control device determines that the current frame image is a key frame image when judging that the feature points in the current frame image exceed the preset number of feature points. In this example, the preset number of feature points as a judgment condition can indicate that the reliability of the current frame image is high and can be added as a key frame. In some other examples, the control device can also determine whether to use the current frame image as a key frame image according to the busyness of the local map construction thread. Using the current frame image as a key frame image requires that the local map construction thread is idle. It should be noted that the judgment methods in the above examples are all examples. Technical personnel in this field can design the standard for the current frame image to be used as a key frame image according to actual needs, or combine the judgment conditions in the above examples. This application does not impose any restrictions on this.

[0098] Please continue reading Figure 6 In step S330, the control device constructs a local map of the physical space based on the key frame and its positioning information, and optimizes the local map according to a constraint item so that the coordinate system of the local map is consistent with the coordinate system of the first map.

[0099] Among them, the local map is a description method of the intermediate form map when the second map is not completed. It refers to the map corresponding to part of the physical space that is incrementally constructed based on the initialization map during the movement of the robot in the physical space. That is, the local map can reflect part of the information in the physical space.

[0100] See also Figure 8 , which is a flowchart of step S330 in one embodiment of the present application. As shown in the figure, step S330 includes: step S331 and step S332.

[0101] In step S331, the control device inserts the key frame into the local map, and updates the landmark points in the local map based on the inserted key frame and its positioning information. The following describes various embodiments or examples of step S330 by taking the key frame as a key frame image as an example.

[0102] Inserting the key frame image into the local map is to prepare for updating the landmark points in the local map. In one embodiment, updating the landmark points in the local map includes: deleting the landmark points and generating new landmark points. In one example, the control device clears out useless landmark points in the local map. For example, if the key frame image in which the landmark point is observed is less than 3 frames, that is, the key frame image corresponding to the landmark point is less than 3 frames, then such landmark points are considered useless and the control device will delete them. In one example, the control device performs feature matching between the inserted key frame image and the historical key frame image in the local map and triangulates according to the positioning information of the key frame image to generate a new landmark point.

[0103] In step S332, the control device optimizes the local map according to the constraint item so that the positioning information of the key frame in the optimized local map can make the constraint item within the ideal threshold. The constraint item at least includes the error between the positioning information of the key frame in the local map and the positioning information of the robot mapped to the first map at the same time. The key frame is set as a key frame image as an example for explanation.

[0104] In order to obtain the constraint item described in step S332, in some embodiments, the method for constructing a map by a robot further includes: determining a key frame and recording the positioning information of the robot mapped to the first map. Continuing to take the key frame as a key frame image as an example for explanation.

[0105] In one embodiment, the first map and the second map to be constructed are maps of the same type, for example, both are set as visual maps, then the image data based on the two maps are frame images of the same image capture device, because the robot will continuously determine its position mapped on the first map according to the collected frame images during its movement in the physical space based on the first map, therefore, the control device only needs to determine the frame image in the environmental data at the same time according to the timestamp of each key frame image in the local map, and then can directly obtain the positioning information of the robot mapped to the first map at the same time.

[0106] Considering that in some embodiments, the first map and the second map are set as different types of maps, the acquired environmental data may contain the same type of data or different types of data. Different types of data are usually acquired by different types of devices, and different types of devices usually have different sampling frequency characteristics. Therefore, the timestamp of the key frame image in the local map does not necessarily directly acquire another type of data in the environmental data that is simultaneously captured, such as the frame point cloud captured simultaneously, that is, the positioning information of the robot mapped to the first map at the same time cannot be acquired.

[0107] In view of this, in an embodiment in which the first map is set as a first laser map and the second map is set as a second visual map, the step of determining the key frame image and simultaneously engraving the positioning information of the robot mapped onto the first map includes: determining two frames of point clouds adjacent to the timestamp of the key frame image in the environmental data, and engraving the positioning information of the robot mapped onto the first map simultaneously with the key frame image based on the determination of the two adjacent frames of point clouds.

[0108] See also Fig. 9 , which is a schematic diagram of determining a key frame image and simultaneously recording the positioning information of the robot mapped onto the first map in one embodiment of the present application. As shown in the figure, during the movement of the robot in the physical space, the environmental data acquired include a frame point cloud acquired by a laser device at a frequency of 1 / T1 (i.e., period T1), and a frame image acquired by an image acquisition device at a frequency of 1 / T2 (i.e., period T2). The timestamp of the key frame image in the local map is T2_n. The control device determines two frame point clouds in the environmental data that are temporally adjacent to the timestamp T2_n based on the timestamp T2_n, i.e., Fig. 9 The frame point cloud with timestamp T1_n-1 and the frame point cloud with timestamp T1_n are used, and the positioning information of the robot mapped on the first map at the corresponding moments of the two frame point clouds is further deduced through the interpolation algorithm.

[0109] It should be noted that, in some embodiments, if the control device cannot obtain two frames of point clouds adjacent to the timestamp of the key frame image in the environmental data, the control device waits for a period of time before obtaining, or the control device deletes the current key frame image. In some examples, if the control device cannot obtain the adjacent frame of point clouds after the timestamp of the key frame image in the environmental data based on the timestamp of the key frame image, the control device waits for a period of time before continuing to obtain. For example, when the control device controls the laser device and the image acquisition device to obtain the current environmental data respectively, the frame image is first obtained, and then the frame point cloud is obtained. The control device can obtain an adjacent frame point cloud before the timestamp of the key frame image and an adjacent frame point cloud after the timestamp of the key frame image after waiting for a period of time. In other examples, if the control device cannot obtain the adjacent frame point cloud before the timestamp of the key frame image in the environmental data based on the timestamp of the key frame image, the current key frame image is deleted. For example, in the acquisition of environmental data, the key frame image is the first environmental data to appear, that is, there is no frame point cloud data before the timestamp of the key frame image, and the control device abandons the key frame image.

[0110] In one embodiment, in step S332, optimizing the local map includes: optimizing the positioning information of the key frame image in the local map. Specifically, the control device performs the optimization with the goal of making the constraint item (i.e., including: the positioning information of the key frame image and the error between the positioning information of the robot mapped to the first map at the same time) within the ideal threshold. In other words, in essence, in the process of constructing the local map, the positioning error between the second map and the first map is continuously corrected, that is, the coordinate system error between the coordinate system of the local map and the first map is continuously corrected, so that the positioning information on the second map formed when the local map is finally constructed can be directly used as the positioning information on the first map. For example, the local map includes a total of 10 key frame images including the newly inserted key frame image. The control device can obtain the positioning information of the robot mapped on the first map at the same time based on the timestamps of the 10 key frame images, and use the errors between all 10 pairs of positioning information paired at the same time (i.e., the positioning information of the key frame image in the local map and the positioning information of the robot mapped on the first map at the same time) as the constraint item for optimizing the local map, so as to obtain the optimal positioning information of each key frame image so that the error between each pair of positioning information is within the ideal threshold. The ideal threshold is used to represent the tolerance range of the constraint item, and within this tolerance range, it means that the optimal solution (i.e., the optimal positioning information of the key frame image) is within the accuracy range.

[0111] In one embodiment, the constraint item further includes: a reprojection error of the landmark point in the local map. Fig.10, which is a schematic diagram of a reprojection error in an embodiment of the present application. The reprojection error refers to the projection of a real landmark point Pj in a three-dimensional space on an image, such as Fig. 9 The projection points in the two frames are X1j and X2j (projection points are also called pixel points), and the reprojection points are obtained by performing a second projection on the image based on the positioning information of the calculated landmark point Pj' and its corresponding frame image, as shown in Fig. 9 The landmark points Pj' calculated in the above formula are respectively reprojected at the points X1j' and X2j' of the two frames. That is, in this embodiment, in the process of optimizing the local map, the constraint term also includes the reprojection error, so that the positioning information of each key frame image optimized can make the reprojection error, the positioning information of each key frame image and the positioning information of the robot mapped to the first map at the same time within the ideal threshold.

[0112] In one embodiment, on the basis that the constraint items include the positioning information of the aforementioned key frame images and the error between the positioning information of the robot mapped to the first map at the same time, it also includes: the error between the inertial navigation positioning information and the positioning information corresponding to the key frame images in the local map. That is, in this embodiment, in the process of optimizing the local map, the constraint items also include the error between the inertial navigation positioning information and the positioning information corresponding to the key frame images, so that the positioning information of each optimized key frame image can make the error between the inertial navigation positioning information and the positioning information corresponding to the key frame image, and the error between the positioning information of the aforementioned key frame images and the positioning information mapped to the first map by the robot at the same time within the ideal threshold. The above-mentioned constraint items are only examples. Of course, in other embodiments, the constraint items may also include the three errors described in the aforementioned embodiments, that is, in the process of optimizing the local map, the positioning information of each optimized key frame image can make the three errors and within the ideal threshold. Those skilled in the art may also add other constraint items according to actual needs, and this application does not limit this. It should be understood that the "inertial navigation" in the "inertial navigation positioning information" described in this application refers to the inertial navigation system (English: Inertial navigation system, abbreviated as INS, referred to as inertial navigation), which is an auxiliary navigation system that uses accelerometers and gyroscopes to measure the acceleration and angular velocity of objects, and continuously estimates the position, attitude and speed of moving objects.

[0113] In some other embodiments, in step S332, optimizing the local map includes optimizing the landmark points in the local map in addition to optimizing the positioning information of the key frame images in the local map. Optimizing the landmark points in the local map refers to optimizing the spatial coordinate positions of the landmark points. For example, the control device optimizes the landmark points based on the optimized positioning information of each key frame.

[0114] In some embodiments, step S330 also includes step S333, in which the control device deletes some of the key frames based on the feature matching results between the inserted key frames and the historical key frames in the local map. In this way, data redundancy can be avoided and computing efficiency can be improved. The key frames are continued to be set as key frame images for explanation. For example, the control device performs feature matching between the inserted key frame image and the historical key frame image. If it is determined that, for example, 90% of the landmark points corresponding to the inserted key frame image can be observed by at least three frames of historical key frame images, the key frame image or one of the three frames of historical key frame images is selected for deletion.

[0115] In order to obtain a second map corresponding to the physical space, the present application needs to incrementally construct the map based on the local map. In view of this, the method for constructing a map by a robot disclosed in the present application includes: repeating the steps S310, S320, and S330 described in any of the above embodiments to obtain the second map corresponding to the physical space. Since, in the loop of executing steps S310, S320, and S330, the coordinate system between the local map and the first map is aligned, the coordinate system of the obtained second map is consistent with the coordinate system of the first map. In other words, the error between the coordinate system of the second map and the coordinate system of the first map is within the accuracy range, and the positioning information mapped by the robot on the second map can be directly used as the positioning information mapped by the robot on the first map.

[0116] As described in the robot positioning method in any of the aforementioned embodiments, the control device will choose to use the positioning information of the first map or the second map as a reference for positioning based on the matching result of the environmental data and the first map. However, in some cases, the robot does not have a complete second map while moving in the physical space. For example, the control device is constructing a second map based on the robot map construction method disclosed in the present application, and it is unreliable for the control device to use the positioning information of the second map as a reference for positioning.

[0117] In view of this, in some embodiments, the robot positioning method described in the present application also includes: a step of determining the integrity of the second map. In some examples, this step can be performed before step 120. When the second map is determined to be complete, the control device executes step S120. When the second map is determined to be incomplete, the control device directly matches the environmental data with the first map to obtain the current positioning information of the robot in the first map, and no longer selects based on the matching result. In some examples, this step can also be performed in step S120. It only needs to be performed before selecting according to the matching result in step S120. When the second map is determined to be incomplete in step S120, the control device chooses to obtain the current positioning information of the robot in the first map based on the environmental data and the first map. When the second map is determined to be complete in step S120, the control device continues to select according to the matching result.

[0118] In one embodiment, the step of determining the integrity of the second map includes determining the integrity of the complete second map, but this will result in an excessive amount of calculation. Therefore, in order to reduce the amount of calculation, in one embodiment, the step of determining the integrity of the second map includes determining the integrity of the sub-area map where the current robot is mapped, and when the sub-area map where the current robot is mapped is determined to be complete, the second map is considered to be complete. In view of this, in some embodiments, the robot positioning method disclosed in the present application also includes: a step of dividing the second map into a plurality of sub-area maps. For example, the second map can be divided according to the first map, such as dividing the second map according to the size of the first map, the distribution of obstacles, etc.; the second map can also be divided according to a preset size; the second map can also be divided according to the amount of calculation, and the present application does not impose any restrictions on this.

[0119] In one embodiment, the step of determining the integrity of the sub-region map corresponding to the current robot location includes: determining the sub-region map corresponding to the current robot location. In one example, the control device first determines the positioning information of the current robot on the first map based on the environmental data and the first map, and then converts the positioning information of the current robot on the first map into the positioning information on the second map according to the association between the first map and the second map. The sub-region map corresponding to the positioning information can be determined according to the positioning information of the current robot on the second map, that is, the sub-region map corresponding to the current robot location. For the association between the first map and the second map and the conversion of the positioning information of the two maps, please refer to the description of any of the aforementioned embodiments, which will not be repeated here.

[0120] For example, in the above embodiment, the judgment of the integrity of the map can be determined by the number of key frames, for example, when the number of key frames reaches a certain scale, the map is considered to be complete; it can also be determined by using the first map as a reference, etc., and the present application does not impose any restrictions on this.

[0121] In some embodiments, when the robot is executing the robot positioning method described in any of the above embodiments, such as executing any of the above steps S110 and S120, in order to ensure the consistency of the coordinate systems of the second map and the first map, the robot positioning method disclosed in the present application also includes: a step of updating the second map.

[0122] In view of this, in some embodiments, the present application also discloses a method for updating a map of a robot, which can be completed as part of the robot positioning method disclosed in the present application during the execution of step S110 and step S120, or can be executed separately, and the updated second map continues to be used as the second map in the robot positioning method disclosed in the present application. In some examples, the method for updating a map of a robot is executed by a robot, and further, can be executed by a control device configured on the robot. In some examples, the method for updating a map of a robot can also be executed by a processing device configured on a server, and the server and the robot can communicate data to control the robot to perform corresponding actions when executing the method for updating a map of a robot disclosed in the present application. In other examples, the method for updating a map of a robot can also be executed by a processing device configured on an electronic device, and the electronic device communicates data with the robot to control the robot to perform corresponding operations when executing the method for updating a map of a robot disclosed in the present application. The following embodiments are described by taking the method for updating a map of a robot executed by a control device configured on a robot as an example.

[0123] See also Fig.11 , which is a flow chart of a method for updating a map of a robot in one embodiment of the present application. As shown in the figure, the method for updating a map of a robot includes step S410, step S420, and step S430.

[0124] In step S410, the control device acquires environmental data while the robot moves in the physical space based on the first map. In some embodiments, step S140 and step S110 are steps reused, that is, the environmental data acquired by step S110 can be used for positioning the robot in step S120, and step S110 can also be used as step S410 in the robot map updating method, and the environmental data acquired is used in the subsequent steps of the robot map updating method. Specifically, the data types and acquisition methods of the environmental data acquired by step S410 refer to the relevant description of any embodiment of step S110 in the aforementioned robot positioning method, which will not be repeated here.

[0125] In step S420, the control device analyzes the movement posture of the robot in the physical space based on the environmental data to obtain key frames and positioning information thereof.

[0126] In some embodiments, the execution process of step S420 is similar to step S320 in the method for constructing a map by a robot described in any of the aforementioned embodiments. The specific execution process of step S420 can refer to any embodiment of step S320 in the method for constructing a map by a robot. The difference lies in that, since the robot updates the map as an update of the second map, in the method for constructing a map by the robot, the steps related to the existence and construction of the initialization map before step S320 may not be necessary before step S420, and may also be adaptively changed to the judgment and construction of whether the second map exists.

[0127] In step S430, the control device updates the second map based on the key frame and its positioning information so that the coordinate system of the updated second map is consistent with the coordinate system of the first map. Wherein, updating the second map based on the key frame and its positioning information includes: optimizing the second map according to a constraint item.

[0128] In one embodiment, updating the second map in step S430 includes updating the complete second map, but this will result in excessive computational effort, and updating the second map will not change the portion of the second map corresponding to the physical space to which the robot has not yet moved. Therefore, updating the complete second map will also result in a waste of computing resources.

[0129] In view of this, in one embodiment, updating the second map in step S430 includes: the control device updates the sub-area map where the current robot is mapped based on the key frame and its positioning information. The sub-area map is set to be obtained after dividing the robot. For example, the second map can be divided according to the first map, such as the size of the first map, the obstacle distribution, etc.; the second map can also be divided according to a preset size; the second map can also be divided according to the calculation amount, and this application does not limit this.

[0130] In one embodiment, the step of the control device updating the sub-area map mapped to the current robot based on the key frame and its positioning information includes: the step of determining the sub-area map corresponding to the current robot's location. In one example, the control device first determines the positioning information of the current robot on the first map based on the environmental data and the first map, and then converts the positioning information of the current robot on the first map into the positioning information on the second map according to the association between the first map and the second map. The sub-area map corresponding to the positioning information can be determined according to the positioning information of the current robot on the second map, that is, the sub-area map corresponding to the current robot's location. For the association between the first map and the second map and the conversion of the positioning information of the two, please refer to the description of any embodiment of the robot positioning method and the robot map construction method, which will not be repeated here.

[0131] As mentioned above, the updating of the second map described in any of the above or following step S430 can be understood as updating the complete second map, or can be understood as updating the sub-region map thereof. Unless otherwise specified, this is understood and will not be described in detail later.

[0132] In one embodiment, see Fig.12 , which is a flow chart of step S430 in one embodiment of the present application. As shown in the figure, step S430 includes: step S431 and step S432.

[0133] In step S431, the control device inserts the key frame into the second map, and updates the landmark points in the second map based on the inserted key frame and its positioning information. In some embodiments, the execution process of step S431 is similar to step S331 in the method for building a map by a robot described in any of the aforementioned embodiments. The specific execution process of step S431 can refer to any of the embodiments of step S331 in the method for building a map by a robot. The difference is that since the construction of a map by a robot is the construction process of a second map, in any of the embodiments of step S331 in the method for building a map by a robot, the form in the process of building the second map is called a local map, and in step S431 in the method for updating a map by a robot, the description related to building a local map in step S331 needs to be adaptively modified to a description related to updating the second map.

[0134] In step S432, the control device optimizes the second map according to the constraint item so that the positioning information of the key frame in the optimized second map can make the constraint item within the ideal threshold. The constraint item includes at least the error between the positioning information of the key frame in the second map and the positioning information of the robot mapped to the first map at the same time.

[0135] In some embodiments, the execution process of step S432 is similar to step S332 in the method for robot constructing a map described in any of the aforementioned embodiments. The specific execution process of step S432 can refer to any of the embodiments of step S332 in the method for robot constructing a map. The difference lies in that, since the robot constructing a map is a process of constructing a second map, in any of the embodiments of step S332 in the method for robot constructing a map, the form in the process of constructing the second map is called a local map, and in step S432 in the method for robot updating a map, the description related to constructing a local map in step S332 needs to be adaptively modified to a description related to updating the second map.

[0136] In some embodiments, step S430 also includes step S433, in which the control device deletes some of the key frames based on the feature matching results between the inserted key frames and the historical key frames in the second map. In this way, data redundancy can be avoided and computing efficiency can be improved. The key frame is set as a key frame image for illustration. For example, the control device performs feature matching between the inserted key frame image and the historical key frame image. If it is determined that the inserted key frame image corresponds to, for example, 90% of the landmark points that can be observed by at least three frames of historical key frame images, the key frame image or one of the three frames of historical key frame images is selected for deletion.

[0137] The present application also discloses a robot, which is used to execute the robot positioning method, the robot map construction method, or the robot map update method described in any of the above embodiments. Fig.13 , which is a schematic diagram of the structure of a robot in one embodiment of the present application. As shown in the figure, the robot 1 includes a moving device 10, a control device 11, and a sensor device 12.

[0138] In one embodiment, the mobile device 10 is used to perform a moving operation. For example, the mobile device 10 is disposed at the bottom of the robot 1 to drive the robot 1 to move. In some embodiments, the mobile device 10 includes a driving component and driving wheels disposed on opposite sides of the bottom of the robot 1, and the driving wheels are driven by the driving component to drive the robot 1 to move. Specifically, the driving wheels are driven to drive the robot 1 to perform forward and backward reciprocating motion, rotational motion, or curvilinear motion according to a planned moving trajectory, or to drive the robot 1 to adjust its posture, and provide two contact points between the robot 1 and the walking surface. In other embodiments, the mobile device 10 also includes a driven wheel, which is located in front of the driving wheel, and the driven wheel and the driving wheel together maintain the balance of the robot 1 in a moving state.

[0139] In one embodiment, the sensor device 12 is used to collect environmental data, including: a laser device. The laser device is horizontally arranged on the top of the robot 1, so that the control device 11 of the robot 1 can control the laser device to rotate and project the laser line without being blocked by the body of the robot 1, so that the laser device can scan the surrounding environment to the maximum extent. Of course, in some other embodiments, according to different application scenarios or functions provided, the laser device can also be set at a certain inclination angle at the front or top of the robot 1, and the present application does not limit this. For example, the laser device can be set as a single-line laser radar or a multi-line laser radar.

[0140] In one embodiment, the sensor device 12 further includes: an image capture device. The image capture device is configured on the robot, including: a camera, a video camera, a camera module integrated with an optical system or a CCD chip, a camera module integrated with an optical system and a CMOS chip, etc. The image capture device can be controlled by the control device. During the movement of the robot, the control device controls the image capture device to capture the surrounding environment, thereby obtaining image data of the surrounding environment. The power supply system of the image capture device can be controlled by the power supply system of the robot. When the robot is powered on and moving, the image capture device starts to capture image frames and provides them to the control device 11. The image capture device can be set on the top of the robot. Taking a cleaning robot as an example, in some examples, the image capture device of the robot is set in the middle or on the edge of its top cover. The optical axis of the field of view of the image capture device is ±30° relative to the vertical line or 60-120° relative to the horizontal line. In some examples, the angle between the optical axis of the camera device of the cleaning robot and the vertical line is -30°, -29°, -28°, -27°...-1°, 0°, 1°, 2°...29°, or 30°. In some other examples, the angle between the optical axis of the camera device of the robot and the horizontal line is 60°, 61°, 62°...119°, 120°. It should be noted that those skilled in the art should understand that the angle between the above-mentioned optical axis and the vertical or horizontal line is only an example, and the angle accuracy is not limited to the range of 1°. According to the actual design requirements of the robot, the accuracy of the angle can be higher, such as reaching 0.1°, 0.01° or above, etc., and no endless examples are given here.

[0141] In one embodiment, the control device 11 is disposed on the robot 1, and is used to control the mobile device 10 to drive the robot 1 to move so as to execute the robot positioning method, the robot map construction method, or the robot map update method described in any of the above embodiments. In some examples, the control device 11 cooperates with the mobile device 10 by controlling at least one of the laser device and the image capture device to execute the robot positioning method, the robot map construction method, or the robot map update method described in any of the above embodiments. The control device 11 can also control the robot 1 to perform work tasks, such as cleaning the ground, cyclic security inspection, etc.

[0142] In some embodiments, the control device 11 of the robot disclosed in the present application includes an interface device, a memory, and a processing device. The interface device is used to communicate data with the robot 1, for example, to communicate data with the laser device and the image acquisition device of the robot 1. The storage device is used to store at least one program. In some examples, the storage device can also store data acquired through the interface device, such as image data and point cloud data. The processing device is connected to the storage device and the interface device, and is used to execute the at least one program to coordinate the storage device and the interface device to execute and implement the robot positioning method, the robot map construction method, or the robot map update method as described in any of the above embodiments.

[0143] In an embodiment, the processing device can be used to read and execute computer-readable instructions. In a specific implementation, the processing device may mainly include a controller, an operator and a register. Among them, the controller is mainly responsible for instruction decoding and sends a control signal for the operation corresponding to the instruction. The operator is mainly responsible for performing fixed-point or floating-point arithmetic operations, shift operations, and logical operations, etc., and can also perform address operations and conversions. The register is mainly responsible for saving register operands and intermediate operation results temporarily stored during the execution of instructions. In a specific implementation, the hardware architecture of the processing device can be an application-specific integrated circuit (ASIC) architecture, a MIPS architecture, an ARM architecture, or an NP architecture, etc. The processing device may include one or more processing units, for example: the processing device may include an application processor (AP), a modulation and demodulation processing device, a graphics processing unit (GPU), an image signal processing device (ISP), a controller, a video codec, a digital signal processing device (DSP), a baseband processing device, and / or a neural-network processing unit (NPU), etc. Among them, different processing units may be independent devices or integrated in one or more processing devices.

[0144] In an embodiment, the storage device is coupled to the processing device and is used to store various software programs and / or multiple sets of instructions. In a specific implementation, the storage device may include a high-speed random access storage device, and may also include a non-volatile storage device, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. The storage device may store an operating system, such as an embedded operating system such as uCOS, VxWorks, RTLinux, etc. The storage device may also store a communication program, which can be used to communicate with an intelligent terminal, an electronic device, one or more servers, or an additional device.

[0145] The present application also discloses an electronic device, which is used to communicate with a robot to control the movement of the robot and implement the robot positioning method, the robot map construction method, or the robot map updating method as described in any of the above embodiments.

[0146] In some embodiments, the electronic device is a device capable of performing digital calculations, logical processing, and information processing on data, including but not limited to: personal computers, servers, server clusters, smart terminals, cloud-based server systems, etc.

[0147] See also Fig.14 , which is a schematic diagram of the structure of an electronic device in an embodiment of the basic application, wherein the electronic device 2 at least includes an interface device 20, a storage device 21, and a processing device 22. In some embodiments, the electronic device 2 may also include a display device (not shown) connected to data through the interface device.

[0148] In some embodiments, the storage device 21 is used to store at least one program, and the at least one program can be executed by the processing device 22 to coordinate the storage device 21 and the interface device 20, etc. to implement the robot positioning method, the robot map construction method, or the robot map update method described in any of the above embodiments. Here, the storage device 21 includes but is not limited to: Read-Only Memory (ROM), Random Access Memory (RAM), Nonvolatile RAM (NVRAM). For example, the storage device 21 includes a flash memory device or other non-volatile solid-state storage device. In some embodiments, the storage device 21 may also include a memory away from one or more processing devices 22, such as a network attached memory accessed via an RF circuit or an external port and a communication network, wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as the CPU and peripheral interfaces.

[0149] In some embodiments, the interface device 20 includes at least one interface unit, each of which is used to output a visual interface, receive a human-computer interaction event generated according to the operation of a technician, etc. For example, the interface device 20 includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In one embodiment, the interface device 20 also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0150] In some embodiments, the display device is used to display a visual interface, i.e., an operation interface. The display device includes, for example, a display, which, when integrated with a touch sensor, can be used as a hardware device for displaying and generating input events. The display device can be connected to the processing device 22 via the interface device 20.

[0151] According to the hardware devices actually included in the computer device, the processing device 22 is connected to the interface device 20, the storage device 21, and the display device. The processing device 22 includes one or more processors. The processing device 22 can perform data read and write operations with the storage device 21. The processing device 22 includes one or more general-purpose microprocessors, one or more application-specific processors (ASICs), one or more digital signal processors (Digital Signal Processors, referred to as DSPs), one or more field programmable gate arrays (Field Programmable Gate Arrays, referred to as FPGAs), or any combination thereof.

[0152] The present application also discloses a server, which is used to communicate with a robot to control the movement of the robot and implement the robot positioning method, or the robot map construction method, or the robot map update method as described in any of the above embodiments. Here, the server system is an electronic device capable of digital calculation, logical processing, and information processing of data, including but not limited to: a central computer, a server, a server cluster, a server system based on a cloud architecture, etc. In the example where the server system is a cloud server system, the cloud server system is, for example, arranged on one or more physical servers according to various factors such as function and load. For example, a server based on a cloud architecture includes a public cloud service end and a private cloud service end, wherein the public or private cloud service end includes SaaS, PaaS, and IaaS, etc. The private cloud service end is, for example, Meituan cloud computing service platform, Alibaba cloud computing service platform, Amazon cloud computing service platform, Baidu cloud computing platform, Tencent cloud computing platform, etc. In the example where the server system is composed of a distributed or centralized server cluster, the server cluster is, for example, composed of at least one physical server, each physical server is configured with multiple virtual servers, each virtual server runs at least some steps of the robot positioning method described in any of the above embodiments, or the robot map building method, or the robot map updating method, and the virtual servers communicate with each other through a network.

[0153] See also Fig.15 , which is a schematic diagram of the structure of a server in an embodiment of the basic application, wherein the server 3 at least includes an interface device 30, a storage device 31, and a processing device 32. In some embodiments, the server 3 may also include a display device (not shown) connected to data through the interface device.

[0154] In an embodiment, the at least one storage device 31 is used to store at least one program; in an embodiment, the storage device 31 includes a storage server or a memory, and the memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage devices, flash memory devices or other non-volatile solid-state storage devices. In some embodiments, the memory may also include a memory away from one or more processors, such as a network attached memory accessed via an RF circuit or an external port and a communication network (not shown), wherein the communication network may be the Internet, one or more intranets, local area networks, wide area networks, storage area networks, etc., or a suitable combination thereof. The memory controller may control access to the memory by other components of the device such as the CPU and peripheral interfaces.

[0155] In an embodiment, the processing device 32 is connected to the storage device 31, and is used to execute and implement the robot positioning method, the robot map construction method, or the robot map update method described in any of the above embodiments when running the at least one program. The processing device 32 is, for example, a server including a processor, such as an application server, etc., and the processor is operably coupled to a memory and / or a non-volatile storage device.

[0156] In an embodiment, the interface device 30 includes at least one interface unit, each of which is used to output a visual interface, receive a human-computer interaction event generated according to the operation of a technician, etc. For example, the interface device 30 includes but is not limited to: a serial interface such as an HDMI interface or a USB interface, or a parallel interface, etc. In an embodiment, the interface device 30 also includes a network communication unit, which is a device for data transmission using a wired or wireless network, examples of which include but are not limited to: an integrated circuit including a network card, a local area network module such as a WiFi module or a Bluetooth module, a wide area network module such as a mobile network, etc.

[0157] The present application also provides a computer storage medium storing at least one program, which, when executed by a processor, controls the device where the storage medium is located to execute the robot positioning method, the robot map building method, or the robot map updating method as described in any of the above embodiments.

[0158] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a robot equipped with the storage medium to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0159] In the embodiments provided in the present application, the computer storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0160] In one or more exemplary aspects, the functions described in the computer program of the robot positioning method, the robot map construction method, or the robot map update method described in the present application can be implemented in hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored or transmitted as one or more instructions or codes to a computer-readable medium. The steps of the method or algorithm disclosed in the present application can be embodied in a processor-executable software module, wherein the processor-executable software module can be located on a tangible, non-temporary computer storage medium. Tangible, non-temporary computer storage media can be any available media that can be accessed by a computer.

[0161] The flowchart and block diagram in the accompanying drawings described in the present application illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. Based on this, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or the flowchart, and the combination of the boxes in the block diagram and / or the flowchart can be realized by a dedicated hardware-based system that performs a specified function or operation, or can be realized by a combination of dedicated hardware and computer instructions.

[0162] In summary, the present application discloses a robot and a positioning method thereof, a robot control device, a computer storage medium, an electronic device, and a server. In the present application, a first map is set as a basic map to provide a navigation and movement basis for the robot. When the positioning accuracy of the first map is high, that is, when the feature matching consistency of the environmental data and the first map is high, the environmental data is matched with the first map for positioning and navigation and movement, etc. When the positioning accuracy of the first map is low, that is, when the matching consistency of the environmental data and the first map is low, the robot obtains positioning information based on the environmental data and the second map, and the association between the first map and the second map, and the more accurate positioning information on the second map is mapped to the positioning information in the first map, so that the robot continues to navigate and move based on the first map with more accurate positioning. In this way, when the positioning based on the first map is inaccurate, the positioning information obtained based on the second map is used as a supplement, so that the probability of positioning failure or error can be reduced while improving the positioning efficiency (that is, reducing the amount of calculation).

[0163] The above embodiments are merely illustrative of the principles and effects of the present application and are not intended to limit the present application. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present application. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed in the present application shall still be covered by the claims of the present application.

Claims

1. A robot positioning method, characterized in that: The following steps are involved: Acquiring current environmental data while the robot moves in a physical space based on a first map; the first map is used as a basic map to provide a navigation and movement basis for the robot; Matching the environmental data with the first map, so as to select, according to the matching result, to obtain the positioning information of the current robot mapping in the first map based on the environmental data and the first map, or to obtain the positioning information of the current robot mapping in the first map based on the environmental data and the second map; wherein the first map and the second map are respectively set to a laser map corresponding to the physical space and a visual map corresponding to the physical space, or the first map is set to a visual map corresponding to the physical space constructed based on VSLAM, and the second map is set to a visual map corresponding to the physical space constructed based on machine learning to extract image features; The second map is associated with the first map, and obtaining the positioning information of the robot mapped in the first map based on the environmental data and the second map includes: converting the positioning information of the robot currently mapped in the second map into the positioning information in the first map based on the association.

2. The robot positioning method according to claim 1, characterized in that: The environmental data includes point cloud data and image data acquired by a laser device and an image pickup device respectively.

3. The robot positioning method according to claim 1, characterized in that: The environmental data includes image data acquired by an image pickup device.

4. The robot positioning method according to claim 1, characterized in that: The association relationship between the first map and the second map is established by any of the following methods: Constructing the second map based on the first map; or Each key frame of the first map and each key frame of the second map are repositioned to determine a coordinate conversion relationship between the first map and the second map.

5. The robot positioning method according to claim 4, characterized in that: The step of repositioning each key frame of the first map and each key frame of the second map to determine a coordinate transformation relationship between the first map and the second map comprises the following steps: Relocating each key frame in the first map in the second map, and relocating each key frame in the second map in the first map, to obtain a paired data set; wherein the paired data set includes positioning information of the successfully relocated key frames on the first map and positioning information on the second map; A coordinate transformation relationship between the first map and the second map is determined based on the paired data set.

6. The robot positioning method according to claim 5, characterized in that: Before the step of determining the coordinate conversion relationship between the first map and the second map based on the paired data set, the method further includes: determining whether the paired data set meets a preset condition.

7. The robot positioning method according to claim 4, characterized in that: The constructing the second map based on the first map comprises the following steps: Acquiring environmental data while the robot moves in the physical space based on the first map; Analyzing the movement posture of the robot in the physical space based on the environmental data to obtain key frames and positioning information thereof; A local map of the physical space is constructed based on the key frame and its positioning information, and the local map is optimized according to a constraint item so that the coordinate system of the local map is consistent with the coordinate system of the first map; wherein the constraint item at least includes the error between the positioning information of the key frame in the local map and the positioning information of the robot mapped to the first map at the same time; Repeat the above steps to obtain a second map corresponding to the physical space.

8. The robot positioning method according to claim 7, characterized in that: Also includes: A step of matching the acquired current environment data with the first map to obtain the current positioning information of the robot mapped to the first map.

9. The robot positioning method according to claim 7, characterized in that: Also includes: The step of determining the key frame and simultaneously recording the positioning information of the robot mapped onto the first map.

10. The robot positioning method according to claim 7, characterized in that: Before the step of analyzing the movement posture of the robot in the physical space based on the environmental data to obtain key frames and positioning information thereof, the step further includes: When it is determined that there is no initialization map according to the frame images in the current environment data, a step of constructing an initialization map based on at least two frame images.

11. The robot positioning method according to claim 10, characterized in that: The step of constructing an initialization map based on at least two frames of images includes: mapping the positioning information of the robot on the first map at the time corresponding to the first frame of image as the positioning information of the first frame of image.

12. The robot positioning method according to claim 7, characterized in that: The step of analyzing the movement posture of the robot in the physical space based on the environmental data to obtain a key frame and its positioning information includes: Analyzing the movement posture of the robot based on the current frame image and the previous frame image to determine the positioning information of the current frame image; The current frame image is judged to determine whether to use it as the key frame image.

13. The robot positioning method according to claim 12, characterized in that: The step of analyzing the movement posture of the robot based on the current frame image and the frame image at the previous moment to determine the positioning information of the current frame image includes: Acquire initial positioning information of the current frame image based on the current frame image and the previous frame image; The initial positioning information is optimized to obtain the positioning information of the current frame image.

14. The robot positioning method according to claim 7, characterized in that: The step of constructing a local map of the physical space based on the key frames and their positioning information, and optimizing the local map according to a constraint item so that the coordinate system of the local map is consistent with the coordinate system of the first map comprises: Inserting the key frame into the local map, and updating the landmark points in the local map based on the inserted key frame and its positioning information; The local map is optimized according to the constraint item so that the positioning information of the key frame in the optimized local map can make the constraint item within the ideal threshold.

15. The robot positioning method according to claim 14, characterized in that: The step of constructing a local map of the physical space based on the key frames and their positioning information, and optimizing the local map according to a constraint so that the coordinate system of the local map is consistent with the coordinate system of the first map also includes: Based on the feature matching results between the inserted key frames and the historical key frames in the local map, some of the key frames are deleted.

16. The robot positioning method according to claim 7, characterized in that: The constraint items also include: reprojection errors of landmark points in the local map.

17. The robot positioning method according to claim 7, characterized in that: The constraint items also include: the error between the inertial navigation positioning information and the positioning information corresponding to the key frames in the local map.

18. The robot positioning method according to claim 1, characterized in that: Converting the current positioning information of the robot in the second map into the positioning information in the first map based on the association relationship includes: Using the current positioning information of the robot mapped in the second map as the positioning information in the first map; According to the coordinate conversion relationship between the first map and the second map, the positioning information of the robot currently mapped in the second map is converted into the positioning information in the first map.

19. The robot positioning method according to claim 1, characterized in that: Also includes: The step of determining the completeness of the second map.

20. The robot positioning method according to claim 19, characterized in that: The determining the integrity of the second map includes: determining the integrity of the sub-area map where the robot is currently mapped.

21. The robot positioning method according to claim 20, characterized in that: Also includes: The step of dividing the second map into multiple sub-area maps.

22. The robot positioning method according to claim 1, characterized in that: The robot is a cleaning robot.

23. A robot control device, characterized in that: include: An interface device for data communication with the robot; A storage device storing at least one program; A processing device, connected to the storage device and the interface device, is used to execute the at least one program to execute and implement the robot positioning method as described in any one of claims 1-22.

24. A robot, characterized in that: include: A sensor device for acquiring environmental data; A mobile device for performing a moving operation; A storage device for storing at least one program; A processing device, connected to the sensor device, the moving device, and the storage device, is used to execute the at least one program to execute the robot positioning method as described in any one of claims 1-22.

25. A computer storage medium, characterized in that At least one computer program is stored, and when the computer program is executed by a processor, the device where the storage medium is located controls the device to execute the robot positioning method as described in any one of claims 1-22.

26. An electronic device, characterized in that: include: An interface device for data communication with the robot; A storage device storing at least one program; A processing device, connected to the storage device and the interface device, is used to execute the at least one program to execute and implement the robot positioning method as described in any one of claims 1-22.

27. A server, characterized in that: include: An interface device for data communication with the robot; A storage device storing at least one program; A processing device, connected to the storage device and the interface device, is used to execute the at least one program to execute and implement the robot positioning method as described in any one of claims 1-22.

Images