A mapping method, device and robot

Abstract

The present invention provides a mapping method, device and robot, wherein the mapping method comprises: acquiring at least one frame of target image of a target area, determining the pose of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; then determining the pose of the corresponding mark in a world coordinate system according to the pose of the corresponding mark in the camera coordinate system; and then constructing a target map of the target area according to the pose of the corresponding mark in the world coordinate system. The present invention maps the target area based on the pose of the preset mark in the world coordinate system, and the generated map is only related to the absolute pose of the preset mark, and has no requirements on the texture and lighting environment of the target area, nor on the installation accuracy of the preset mark, thereby avoiding the problem of map matching and solving the problems of loop accumulation error and textureless matching.

Description

Technical Field

[0001] The present invention relates to the field of robot technology, and in particular to a mapping method, a mapping device, a robot, and a computer-readable storage medium. Background Art

[0002] With the rapid development of computer technology and network technology, various intelligent robots have emerged. In actual use, robots need to move in the working environment to complete corresponding tasks. However, the actual working environment of robots is generally unknown. In an unknown environment, robots must correctly and effectively detect the current working environment through mapping.

[0003] In the existing technology, the means of mapping are mainly divided into two types: vision and laser. Vision is to obtain massive and redundant texture information from an unknown working environment, and to build a map through methods such as view geometry and graph optimization; while laser can collect scattered point clouds with accurate angle and distance information, and calculate relative motion through point cloud matching, so as to locate the robot and update the map.

[0004] However, mapping through visual means performs poorly in textureless environments or low-light environments; mapping through laser means discards most of the information, making it difficult to eliminate the accumulated positioning errors, and thus requires more convenient and accurate methods for mapping operations or processing. Summary of the invention

[0005] In view of this, an embodiment of the present invention provides a mapping method. The present invention also relates to a mapping device, a robot, and a computer-readable storage medium to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of the present invention, a mapping method is provided, including:

[0007] Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system;

[0008] Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system;

[0009] A target map of the target area is constructed according to the position and posture of the corresponding marker in the world coordinate system.

[0010] Optionally, determining the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system includes:

[0011] The camera coordinate system of the first target image in the at least one target image frame is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system is converted into the position and posture of the corresponding mark in the world coordinate system.

[0012] Optionally, the target area includes preset value groups of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and corresponding marks of a group of the preset marks in the at least one frame of the target image also include a target value of marking points;

[0013] The step of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system includes:

[0014] According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

[0015] Optionally, the step of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system according to the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system includes:

[0016] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0017]

[0018] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marking point in the marking coordinate system, is the coordinate of the marker point in the pixel coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0019] Optionally, determining the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system includes:

[0020] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0021] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0022] Optionally, after determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the method further includes:

[0023] The position and posture of the corresponding marker in the world coordinate system are corrected to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0024] Optionally, the correcting the position and posture of the corresponding marker in the world coordinate system includes:

[0025] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0026] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0027] Optionally, constructing a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system includes:

[0028] Determine the boundary of the movable area corresponding to the corresponding mark;

[0029] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0030] A target map of the target area is constructed according to the coordinates of the movable area and the corresponding mark in the world coordinate system.

[0031] Optionally, determining a boundary of a movable area corresponding to the corresponding mark includes:

[0032] The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0033] Optionally, determining a boundary of a movable area corresponding to the corresponding mark includes:

[0034] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0035]

[0036] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0037] Optionally, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the method further includes:

[0038] Saving the group number of the corresponding marker and its position in the world coordinate system;

[0039] When a map loading request is detected, the initial map is loaded, and the corresponding markers are saved in the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, so as to construct a target map of the target area.

[0040] Optionally, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the method further includes:

[0041] Acquire an image captured at the current position, and determine at least one set of preset marks included in the image captured at the current position;

[0042] Selecting a target preset mark from the at least one group of preset marks;

[0043] The camera position is determined according to the position of the selected target preset marker in the world coordinate system.

[0044] Optionally, determining the camera pose according to the pose of the selected preset target marker in the world coordinate system includes:

[0045] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0046] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0047] According to a second aspect of an embodiment of the present invention, there is provided a mapping device, comprising:

[0048] A first determination module is configured to obtain at least one frame of target image of a target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system;

[0049] A second determination module is configured to determine the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system;

[0050] The first construction module is configured to construct a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system.

[0051] Optionally, the second determining module is further configured to:

[0052] The camera coordinate system of the first frame of the target image in the at least one frame of the target image is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system are converted into the coordinates of the corresponding mark in the world coordinate system.

[0053] Optionally, the target area includes preset value groups of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and corresponding marks of a group of the preset marks in the at least one frame of the target image also include a target value of marking points;

[0054] The second determining module is further configured to:

[0055] According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

[0056] Optionally, the second determining module is further configured to:

[0057] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0058]

[0059] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marking point in the marking coordinate system, is the coordinate of the marker point in the pixel coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0060] Optionally, the first determining module is further configured to:

[0061] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0062] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0063] Optionally, the device further comprises:

[0064] The correction module is configured to correct the position and posture of the corresponding marker in the world coordinate system to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0065] Optionally, the correction module is further configured to:

[0066] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0067] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0068] Optionally, the first building module is further configured as follows:

[0069] Determine the boundary of the movable area corresponding to the corresponding mark;

[0070] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0071] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0072] Optionally, the first building module is further configured as follows:

[0073] The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0074] Optionally, the first building module is further configured as follows:

[0075] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0076]

[0077] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0078] Optionally, the device further comprises:

[0079] A saving module, configured to save the group number of the corresponding mark and the position and posture in the world coordinate system;

[0080] The second construction module is configured to load the initial map when a map loading request is detected, save the corresponding markers to the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, and construct a target map of the target area.

[0081] Optionally, the device further comprises:

[0082] A third determining module is configured to obtain an image taken at the current position, and determine at least one set of preset marks included in the image taken at the current position;

[0083] A selection module is configured to select a target preset mark from the at least one group of preset marks;

[0084] The fourth determination module is configured to determine the camera posture according to the posture of the selected target preset mark in the world coordinate system.

[0085] Optionally, the fourth determining module is further configured to:

[0086] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0087] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0088] According to a third aspect of an embodiment of the present invention, there is provided a robot, comprising:

[0089] A mechanical body, wherein at least one processor and at least one memory storing computer instructions are disposed on the mechanical body;

[0090] The at least one processor is used to execute computer instructions to implement the following method:

[0091] Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system;

[0092] Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system;

[0093] A target map of the target area is constructed according to the position and posture of the corresponding marker in the world coordinate system.

[0094] Optionally, in the process of determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the at least one processor is further used to:

[0095] The camera coordinate system of the first target image in the at least one target image frame is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system is converted into the position and posture of the corresponding mark in the world coordinate system.

[0096] Optionally, the target area includes preset value groups of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and corresponding marks of a group of the preset marks in the at least one frame of the target image also include a target value of marking points;

[0097] In the process of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system, the at least one processor is further used to:

[0098] According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

[0099] Optionally, in the process of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system according to the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the at least one processor is further used to:

[0100] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0101]

[0102] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marker point in the imaging plane coordinate system, is the coordinate of the marker point in the image coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0103] Optionally, in the process of determining the position of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system, the at least one processor is further configured to:

[0104] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0105] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0106] Optionally, after determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the at least one processor is further configured to:

[0107] The position and posture of the corresponding marker in the world coordinate system are corrected to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0108] Optionally, in the process of correcting the posture of the corresponding marker in the world coordinate system, the at least one processor is further used to:

[0109] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0110] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0111] Optionally, in the process of constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further used to:

[0112] Determine the boundary of the movable area corresponding to the corresponding mark;

[0113] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0114] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0115] Optionally, in the process of determining the boundary of the movable area corresponding to the corresponding mark, the at least one processor is further configured to:

[0116] The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0117] Optionally, in the process of determining the boundary of the movable area corresponding to the corresponding mark, the at least one processor is further configured to:

[0118] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0119]

[0120] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0121] Optionally, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further configured to:

[0122] Saving the group number of the corresponding marker and its position in the world coordinate system;

[0123] When a map loading request is detected, the initial map is loaded, and the corresponding markers are saved in the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, so as to construct a target map of the target area.

[0124] Optionally, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further configured to:

[0125] Acquire an image captured at the current position, and determine at least one set of preset marks included in the image captured at the current position;

[0126] Selecting a target preset mark from the at least one group of preset marks;

[0127] The camera position is determined according to the position of the selected target preset marker in the world coordinate system.

[0128] Optionally, in the process of determining the camera pose according to the selected pose of the preset target marker in the world coordinate system, the at least one processor is further configured to:

[0129] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0130] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0131] According to a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, which stores computer-executable instructions, and when the instructions are executed by a processor, the steps of any of the mapping methods are implemented.

[0132] The present invention provides a mapping method, which first obtains at least one frame of target image of a target area, determines the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; then determines the position and posture of the corresponding mark in a world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system; and then constructs a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system. In the present invention, it is not necessary to construct a map through texture information, but multiple groups of preset marks are set in the target area in advance. After the camera captures an image of the target area, the image will include the above-mentioned preset marks, and then the map can be constructed directly through the position and posture of the preset mark. In this way, the texture information of the target area is not required, so that even in a textureless environment or a weak light environment, accurate mapping can be performed. In addition, since the installation of the preset marker is unpredictable and there is uncertainty, the present invention first determines the position and posture of the preset marker in the camera coordinate system, and then converts the position and posture of the preset marker in the camera coordinate system into the position and posture in the world coordinate system, so as to map the target area based on the absolute position and posture of the preset marker. The generated map is only related to the absolute position and posture of the preset marker in the world coordinate system. There is no requirement for the installation accuracy of the preset marker, which avoids the problem of map matching and solves the problems of loop accumulation error and textureless matching. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 is a flow chart of a mapping method provided by an embodiment of the present invention;

[0134] Figure 2 is a flow chart of another mapping method provided by an embodiment of the present invention;

[0135] Figure 3 is an example of an infrared image provided by an embodiment of the present invention;

[0136] Figure 4 is a schematic diagram of a common viewing state provided by an embodiment of the present invention;

[0137] Figure 5 is a schematic diagram of a two-dimensional grid map provided by an embodiment of the present invention;

[0138] Figure 6 is a schematic diagram of another two-dimensional grid map provided by an embodiment of the present invention;

[0139] Figure 7 is a structural schematic diagram of a mapping device provided by an embodiment of the present invention;

[0140] Figure 8 It is a structural block diagram of a computing device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0141] Many specific details are set forth in the following description to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of the present invention, so the present invention is not limited to the specific implementation disclosed below.

[0142] The terms used in one or more embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of the present invention. The singular forms of "a", "said" and "the" used in one or more embodiments of the present invention and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of the present invention refers to and includes any or all possible combinations of one or more associated listed items.

[0143] It should be understood that, although the terms first, second, etc. may be used to describe various information in one or more embodiments of the present invention, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of the present invention, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0144] First, the terms involved in one or more embodiments of the present invention are explained.

[0145] Camera coordinate system: It is a three-dimensional rectangular coordinate system with the focal center of the camera as the origin and the optical axis as the Z axis. That is, the origin is the optical center of the camera, the x-axis and y-axis are parallel to the X and Y axes of the image, and the z-axis is the optical axis of the camera, which is perpendicular to the graphic plane.

[0146] World coordinate system: The camera can be placed anywhere in the environment. A reference coordinate system is selected in the environment to describe the position of the camera, and it is used to describe the position of any object in the environment. This coordinate system is called the world coordinate system. The relationship between the camera coordinate system and the world coordinate system can be described by the rotation matrix and the translation vector.

[0147] Image coordinate system: The digital image captured by the camera can be stored as an array in the computer. The value of each element in the array is the brightness of the image point. A rectangular coordinate system is defined on the image. The coordinates of each pixel are the total number of columns and rows of the pixel in the array. That is, the image coordinate system is a coordinate system based on pixels, which is a two-dimensional rectangular coordinate system.

[0148] Imaging plane coordinate system: Since the image coordinate system only indicates the number of columns and rows of pixels in the digital image, and does not use physical units to represent the physical position of the pixel in the image, it is necessary to establish an imaging plane coordinate system represented by physical units (for example, centimeters). The origin is defined at the intersection of the camera optical axis and the image plane, which is called the principal point of the image. This point is generally located at the center of the image.

[0149] Homography: It is a reversible transformation from a real projective plane to a projective plane, under which a straight line is still mapped to a straight line. In the field of computer vision, any two images of the same plane in space are related together through homography. For example, an object can be obtained by rotating the camera lens to obtain two different photos (the contents of these two photos do not have to be completely corresponding, but partially corresponding). The homography can be set as a two-dimensional matrix M, then photo 1 multiplied by M is photo 2.

[0150] Next, the self-moving device involved in the present invention is schematically explained.

[0151] The self-moving device may refer to a robot used in the field of artificial intelligence that can move autonomously in a target area, i.e., a self-moving robot. The self-moving robot may include, but is not limited to, various intelligent robots with cleaning functions, public service robots, sweeping robots, and movable air purifiers, and may also be used for driverless cars, etc. The self-moving robot is provided with sensors for following functions, such as cameras, pulse radio (Ultra Wideband, UWB), sensors, etc. These sensors, in combination with appropriate algorithms, can realize effective and robust mapping functions.

[0152] Among them, the sensors include multiple types, such as visual sensors, laser ranging sensors, infrared sensors, laser sensors, etc. More specifically, the sensor can be one or more. The sensors can be all set on the top of the self-propelled robot, or they can be respectively set on the sides of the self-propelled robot, so as to obtain more complete images in all directions of the self-propelled robot.

[0153] In addition, the usage scenarios of the self-moving robot of the present application for performing area recognition include the floors of places such as homes, shopping malls, and schools, and may also include the surfaces of various objects, such as flat boards.

[0154] The present application does not limit the various shapes of the self-moving robot, such as ellipse, circle, convex polygon, etc., and the self-moving robot can implement the method logic of controlling the movement of the self-moving robot by installing software or APP in a controller used in conjunction with the self-moving robot or writing a program in a corresponding device inside the self-moving robot.

[0155] Next, the basic concept of the mapping method provided by the present invention is briefly described:

[0156] Simultaneous localization and mapping (SLAM) is the current mainstream mapping idea. With the help of SLAM algorithm, a self-propelled robot can move from an unknown position in an unknown environment, and locate itself according to position estimation and sensor data during the movement, while using an incremental method to build a map.

[0157] Currently, the means of mapping and positioning are mainly divided into two types: vision and laser. Vision is to obtain massive and redundant texture information from unknown working environments, and to build maps through methods such as view geometry and graph optimization; while laser can collect scattered point clouds with accurate angle and distance information, and calculate relative motion through point cloud matching, so as to locate the self-propelled robot and update the map. Vision methods perform poorly in textureless environments or low-light environments, and laser methods discard most of the information, making it difficult to eliminate the accumulated errors in positioning.

[0158] Marks are widely used in the fields of AR and camera calibration. Their form is similar to that of QR codes. They are identified by decoding them and their relative position is calculated after 3D-2D matching. However, they lack the absolute position in the world coordinate system and can only be used as an auxiliary positioning method. The absolute position generally requires high installation accuracy of the marker, and there is currently a lack of methods to obtain the absolute position of the mark.

[0159] In view of the basic requirements of poor accuracy and large errors in mapping and positioning in the prior art, the present invention provides a mapping method, device and robot, first obtaining at least one frame of target image of the target area, determining the position and posture of the corresponding mark of the preset mark in the target area in the at least one frame of target image in the camera coordinate system; then determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system; and then constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system. In the present invention, the position and posture of the preset mark in the camera coordinate system are first determined, and then the position and posture of the preset mark in the camera coordinate system are converted into the position and posture in the world coordinate system, so as to map the target area based on the position and posture of the preset mark in the world coordinate system. The generated map is only related to the position and posture of the preset mark in the world coordinate system, and there is no requirement for the texture and lighting environment of the target area, and there is no requirement for the installation accuracy of the preset mark, thereby avoiding the problem of map matching and solving the problems of loop accumulation error and textureless matching.

[0160] In the present invention, a mapping method is provided. The present invention also relates to a mapping device, a robot, and a computer-readable storage medium, which are described in detail one by one in the following embodiments.

[0161] Figure 1 A flowchart of a mapping method provided according to an embodiment of the present invention is shown, which specifically includes the following steps:

[0162] Step 102: Acquire at least one frame of target image of a target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system.

[0163] In one or more implementations of this embodiment, determining the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in the camera coordinate system may be specifically implemented as follows:

[0164] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0165] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0166] Step 104: Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system.

[0167] Specifically, on the basis of acquiring at least one frame of target image of the target area and determining the position and posture of the corresponding mark of the preset mark in the target area in the at least one frame of target image in the camera coordinate system, further, the position and posture of the corresponding mark in the world coordinate system will be determined according to the position and posture of the corresponding mark in the camera coordinate system.

[0168] In one or more implementations of this embodiment, the camera coordinate system of the first frame target image in the at least one frame target image can be used as the world coordinate system, and then the posture of the corresponding mark in the camera coordinate system can be converted into the posture of the corresponding mark in the world coordinate system.

[0169] In one or more implementations of the present embodiment, the target area includes preset value groups of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of mark points, and a group of corresponding marks of the preset marks in the at least one frame of the target image also includes a target value of mark points. The position of the corresponding mark in the camera coordinate system can be converted into the position of the corresponding mark in the world coordinate system according to the coordinates of the target value of mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system.

[0170] In one or more implementations of this embodiment, the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system, can be specifically implemented as follows:

[0171] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0172]

[0173] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marker point in the imaging plane coordinate system, is the coordinate of the marker point in the image coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0174] In one or more implementations of this embodiment, after determining the position and posture of the corresponding marker in the world coordinate system based on the position and posture of the corresponding marker in the camera coordinate system, the position and posture of the corresponding marker in the world coordinate system can also be corrected to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0175] In one or more implementations of this embodiment, the correction of the position and posture of the corresponding marker in the world coordinate system may be specifically implemented as follows:

[0176] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0177] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0178] Step 106: construct a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system.

[0179] Specifically, based on determining the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system, a target map of the target area is further constructed according to the position and posture of the corresponding marker in the world coordinate system.

[0180] In one or more implementations of this embodiment, a target map of the target area is constructed according to the position and posture of the corresponding mark in the world coordinate system. The specific implementation process may be:

[0181] Determine the boundary of the movable area corresponding to the corresponding mark;

[0182] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0183] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0184] In one or more implementations of this embodiment, the boundary of the movable area corresponding to the corresponding mark can be determined based on the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0185] In one or more implementations of this embodiment, the boundary of the movable area corresponding to the corresponding mark is determined, and the specific implementation process may be:

[0186] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0187]

[0188] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0189] In one or more implementations of this embodiment, after constructing the target map of the target area according to the position of the corresponding marker in the world coordinate system, the group number of the corresponding marker and the position in the world coordinate system can also be saved; thereafter, when a map loading request is detected, the initial map is loaded, and the corresponding marker is saved in the initial map according to the group number of the corresponding marker and the position in the world coordinate system to construct the target map of the target area.

[0190] In one or more implementations of this embodiment, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the current position of the camera can also be located. The specific implementation process is as follows:

[0191] Acquire an image captured at the current position, and determine at least one set of preset marks included in the image captured at the current position;

[0192] Selecting a target preset mark from the at least one group of preset marks;

[0193] The camera position is determined according to the position of the selected target preset marker in the world coordinate system.

[0194] In one or more implementations of this embodiment, the camera pose is determined according to the pose of the selected target preset marker in the world coordinate system, and the specific implementation process may be:

[0195] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0196] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0197] The present invention provides a mapping method, which first obtains at least one frame of target image of a target area, determines the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; then determines the position and posture of the corresponding mark in a world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system; and then constructs a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system. In the present invention, it is not necessary to construct a map through texture information, but multiple groups of preset marks are set in the target area in advance. After the camera captures an image of the target area, the image will include the above-mentioned preset marks, and then the map can be constructed directly through the position and posture of the preset mark. In this way, the texture information of the target area is not required, so that even in a textureless environment or a weak light environment, accurate mapping can be performed. In addition, since the installation of the preset marker is unpredictable and there is uncertainty, the present invention first determines the position and posture of the preset marker in the camera coordinate system, and then converts the position and posture of the preset marker in the camera coordinate system into the position and posture in the world coordinate system, so as to map the target area based on the absolute position and posture of the preset marker. The generated map is only related to the absolute position and posture of the preset marker in the world coordinate system. There is no requirement for the installation accuracy of the preset marker, which avoids the problem of map matching and solves the problems of loop accumulation error and textureless matching.

[0198] The following combination Figure 2 , for the above-mentioned Figure 1 The mapping method shown in FIG. Figure 2 A flowchart of another mapping method provided by an embodiment of the present invention is shown, which specifically includes the following steps:

[0199] Step 202: Acquire at least one frame of target image of a target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system.

[0200] Specifically, the target area is the actual working scene of the robot, that is, the area where the robot can move, which is also the area where the map needs to be constructed in the end; the target image is the image of the target area taken by the camera; the preset mark is the mark pre-installed in the target area; the corresponding mark is the mark corresponding to the preset mark in the image obtained by shooting the target area.

[0201] In actual applications, the user can install a preset value group of preset marks on the top wall of the target area in advance, and ensure that the top wall where the preset marks are installed is flat and has no protrusions. In addition, each group of preset marks includes a target value of marking points, and has a corresponding group number. The distance between each marking point must ensure that the camera can observe at least two marking points when shooting a frame of target image. Correspondingly, the corresponding marks of a group of the preset marks in the at least one frame of target image also include a target value of marking points.

[0202] It should be noted that a set of preset marks includes multiple marking points, which can form a plane with an inclination angle. If the preset marks are installed on a horizontal roof, the inclination angle of the plane formed by the multiple marking points is 0 degrees. Therefore, it is necessary to determine the position of the corresponding mark of the preset mark in the target area in the at least one frame of the target image in the camera coordinate system, and the position includes coordinates and inclination angle. For a single marking point included in a set of preset marks, it only has coordinates in the coordinate system.

[0203] The robot is equipped with a camera and moves in the target area, taking images while moving, and at least one frame of the target image can be obtained. When moving, the robot moves in one direction to traverse all preset marks while ensuring that there are mark observations, and the image taken during the entire movement is the at least one frame of the target image; in addition, the robot can also move back and forth multiple times in the observable area of ​​the preset mark, and the image taken during the entire movement is the at least one frame of the target image.

[0204] It should be noted that the present invention does not specifically limit the movement process of the robot, as long as the robot movement process can traverse all preset marks, and in order to provide sufficient constraints, a back-off loop can be inserted in the movement process, that is, after moving to a preset mark, it moves back to another observed mark; or the number of traversals can be increased. In this case, enough target images can be obtained, and the corresponding marks in the target images can provide sufficient constraints, so that the coordinates of the corresponding marks in the target images determined later in the world coordinate system are more accurate, thereby improving the accuracy of mapping.

[0205] In specific implementation, in order to prevent interference from overhead spotlights, an infrared global shutter camera can be used to acquire the target image to improve the detection rate and avoid motion blur. An example of target image acquisition is shown in the attached figure. Figure 3 shown.

[0206] In one or more implementations of this embodiment, determining the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in the camera coordinate system may be specifically implemented as follows:

[0207] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0208] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0209] In practical applications, existing image analysis methods can be used to determine the pixel positions of each marking point included in the corresponding marking in the at least one frame of the target image, and decode them to the group number of the corresponding marking, and then the position and posture of the corresponding marking in the camera coordinate system can be calculated by homography.

[0210] In the present invention, by moving a robot carrying a camera in a target area, at least one target image of the target area can be captured, and then the position and posture of the corresponding mark of a preset mark in the target area in the at least one frame of the target image in the camera coordinate system can be determined, so as to facilitate the subsequent further determination of the position and posture of the corresponding mark in the at least one frame of the target image in the world coordinate system, thereby constructing a map of the target area.

[0211] Step 204: Taking the camera coordinate system of the first target image in the at least one target image frame as the world coordinate system, converting the position and posture of the corresponding mark in the camera coordinate system into the position and posture of the corresponding mark in the world coordinate system.

[0212] Specifically, on the basis of acquiring at least one frame of target image of the target area and determining the position and posture of the corresponding mark of the preset mark in the target area in the at least one frame of target image in the camera coordinate system, further, taking the camera coordinate system of the first frame of target image in the at least one frame of target image as the world coordinate system, the position and posture of the corresponding mark in the camera coordinate system is converted into the position and posture of the corresponding mark in the world coordinate system.

[0213] In actual applications, the camera is generally installed with the z-axis facing upward. At this time, the camera coordinate system of the first frame of the target image in at least one frame of the target image can be used as the world coordinate system. Subsequently, the position and posture of the corresponding mark in the camera coordinate system can be converted into the position and posture of the corresponding mark in the world coordinate system.

[0214] In one or more implementations of this embodiment, the position and posture of the corresponding mark in the camera coordinate system may be converted into the position and posture of the corresponding mark in the world coordinate system according to the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system. The specific implementation process is as follows:

[0215] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula (1):

[0216]

[0217] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marker point in the imaging plane coordinate system, is the coordinate of the marker point in the image coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0218] In practical applications, if a set of preset markers includes 6 marker points, that is, a set of markers uses 6 pixel positions, 6 constraints can be generated on the marker position and the camera position, and the preset value group markers will generate plane constraints. Specifically, the position and posture of the corresponding marker in the world coordinate system can be determined by the above formula (1), so that ξ in formula (1) is minimized, and the corresponding

[0219] In addition, cameras taking images of the target area at different positions may observe the same mark, i.e., common view state, e.g. Figure 4 As shown, camera 1 can observe marker 1, camera 2 can observe marker 1 and marker 2, camera 3 can observe marker 1, marker 2, and marker 3, camera 4 can observe marker 2 and marker 3, and camera 5 can observe marker 3.

[0220] In the present invention, the camera coordinate system of the first frame target image in the at least one frame target image is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system is converted into the position and posture of the corresponding mark in the world coordinate system, so that the target area is mapped based on the position and posture of the preset mark in the world coordinate system. The generated map is only related to the position and posture of the preset mark in the world coordinate system, and has no requirements on the texture and lighting environment of the target area, and has no requirements on the installation accuracy of the mark, thereby avoiding the problem of map matching and solving the problems of loop accumulation error and textureless matching.

[0221] Step 206: Correct the position and posture of the corresponding marker in the world coordinate system to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0222] Specifically, on the basis of converting the posture of the corresponding marker in the camera coordinate system into the posture of the corresponding marker in the world coordinate system, the posture of the corresponding marker in the world coordinate system is further corrected to obtain the corrected posture of the corresponding marker in the world coordinate system.

[0223] In practical applications, the position and posture of the corresponding marker in the world coordinate system are corrected, that is, the position and posture of the corresponding marker in the world coordinate system is optimized (filtered, corrected, etc.). In specific implementation, there are two possible ways to correct the position and posture of the corresponding marker in the world coordinate system. The first way is to correct the position and posture of the corresponding markers of all acquired target images in the world coordinate system. The specific implementation process can be:

[0224] The poses of all target images and determined markers in the world coordinate system are globally optimized by BA. The pose optimization vertex of the marker in the world coordinate system is used as the benchmark, and the pose of the marker in the target image frame of the marker with the corresponding group number in the world coordinate system is used as another optimization vertex. The pixel coordinates in the target image are used to construct pixel reprojection optimization edge constraints for graph optimization.

[0225] Specifically, BA optimization uses three-dimensional points for projection to find the most suitable camera pose, and iteratively adjusts the light beam so that the light beam satisfies the constrained plane conditions.

[0226] The second method is to correct the position and posture of the corresponding markers of some target images in all the acquired target images in the world coordinate system. The specific implementation process can be:

[0227] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0228] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0229] In the present invention, after converting the posture of the corresponding mark in the camera coordinate system to the posture of the corresponding mark in the world coordinate system, the posture of the corresponding mark in the world coordinate system is not directly used for the subsequent map construction process, but the posture of the corresponding mark in the world coordinate system is further corrected to correct the posture error of the corresponding mark in the world coordinate system to a certain extent, which can improve the accuracy of the subsequent construction of the target map of the target area. In addition, the posture of the corresponding mark in the world coordinate system can be corrected by obtaining all or part of the target image, and the correction method is more flexible and suitable for most practical application scenarios.

[0230] Step 208: construct a target map of the target area according to the corrected position and posture of the corresponding marker in the world coordinate system.

[0231] Specifically, after correcting the position and posture of the corresponding marker in the world coordinate system to obtain the corrected position and posture of the corresponding marker in the world coordinate system, a target map of the target area is further constructed according to the corrected position and posture of the corresponding marker in the world coordinate system.

[0232] In one or more implementations of this embodiment, a target map of the target area is constructed according to the corrected position and posture of the corresponding mark in the world coordinate system. The specific implementation process may be:

[0233] Determine the boundary of the movable area corresponding to the corresponding mark;

[0234] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0235] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0236] Specifically, a group of markers has an external rectangular boundary, which corresponds to the minimum range in which the markers can be observed in various rotation and position states, that is, the movable area.

[0237] In practical applications, the boundary of the movable area corresponding to the corresponding mark can be determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark. The specific implementation process is as follows:

[0238] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula (2):

[0239]

[0240] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0241] In actual implementation, Figure 5 As shown, a set of markers can be projected into a two-dimensional grid map according to the installation height H of the camera, and further, the boundary grid state of the area can be set as an obstacle, and the middle area can be set to a free state. In addition, multiple sets of markers can be projected to obtain multiple two-dimensional grid maps, such as Figure 6 As shown, marker 1 and marker 2 are projected to obtain two two-dimensional grid maps. At this time, the overlapping area can be reset to a free state to obtain a two-dimensional movable grid map, that is, a movable area, and a feasible area grid map is generated to facilitate subsequent navigation.

[0242] In addition, after determining the movable area (two-dimensional movable grid map), the boundary of the movable area can also be set as a virtual wall. When the robot touches the virtual wall, an out-of-bounds alarm is issued to control the movement of the robot.

[0243] In one or more implementations of this embodiment, after the mapping is completed, only the marks of some specific position areas can be used to assist the laser and vision to eliminate the cumulative errors.

[0244] Step 210: Acquire an image taken at the current position, determine at least one group of preset marks included in the image taken at the current position, select a target preset mark from the at least one group of preset marks, and determine the camera pose according to the pose of the selected target preset mark in the world coordinate system.

[0245] It should be noted that when the map of the target area is completed, when the robot moves in the target area, it can avoid obstacles through distance sensors such as lasers and ultrasound, and determine the camera's position and posture based on the coordinates of the markers installed in the target area in the world coordinate system. Since the camera is moved by the robot, the coordinates in the camera's position and posture are the coordinates of the robot, which realizes the positioning of the camera (robot) based on the coordinates of the preset markers in the world coordinate system.

[0246] In practical applications, the camera pose is determined according to the pose of the selected target preset marker in the world coordinate system. The specific implementation process may be:

[0247] The pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position may be determined first;

[0248] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points, so as to determine the camera posture.

[0249] In one or more implementations of this embodiment, after determining the position and posture of the corresponding mark in the world coordinate system (or the corrected position and posture of the corresponding mark in the world coordinate system), the target map of the target area may not be constructed first, but the group number and the position and posture of the corresponding mark in the world coordinate system may be saved first; when a map loading request is detected later, the target map is constructed again, specifically, the initial map is loaded, and according to the group number and the position and posture of the corresponding mark in the world coordinate system, the corresponding mark is saved to the initial map to construct the target map of the target area. The saving mechanism can choose to save text or binary files for easy loading.

[0250] The present invention provides a mapping method, which first determines the position and posture of a preset mark in a camera coordinate system, and then converts the position and posture of the preset mark in the camera coordinate system into a position and posture in a world coordinate system, thereby mapping a target area based on the position and posture of the preset mark in the world coordinate system. The generated map is only related to the position and posture of the preset mark in the world coordinate system, and has no requirements for the texture and lighting environment of the target area, nor for the installation accuracy of the preset mark. It only requires the attached top surface to be a plane, thereby avoiding the problem of map matching and solving the problems of loop accumulation error and textureless matching. In addition, after the mapping is completed, it is only necessary to save the position and posture of the preset mark and the corresponding group number. After loading the map during positioning, a two-dimensional grid map available for navigation can be generated, which consumes less storage resources and is convenient and fast. Furthermore, after constructing the target map, the position of the camera can be directly obtained according to the absolute position of the preset mark in space (position and posture in the world coordinate system).

[0251] When the scene changes frequently and there are glass objects or textures that are not clearly distinguishable, the marking mapping and positioning method provided by the present invention can be used. By installing preset markers to determine the feasible area of ​​the robot and establishing a corresponding map, the absolute position can be given by the preset markers, and other sensors can assist in navigation by obstacle avoidance.

[0252] Corresponding to the above method embodiment, the present invention also provides a mapping device embodiment, Figure 7 FIG. 1 is a schematic diagram showing the structure of a mapping device provided by an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0253] The first determination module 702 is configured to obtain at least one frame of target image of the target area, and determine the position and posture of the corresponding mark of the preset mark in the target area in the at least one frame of target image in the camera coordinate system;

[0254] A second determination module 704 is configured to determine the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system;

[0255] The first construction module 706 is configured to construct a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system.

[0256] In one or more implementations of this embodiment, the second determining module 704 is further configured to:

[0257] The camera coordinate system of the first frame of the target image in the at least one frame of the target image is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system are converted into the coordinates of the corresponding mark in the world coordinate system.

[0258] In one or more implementations of this embodiment, the target area includes a preset value group of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and a corresponding mark of a group of the preset marks in the at least one frame of the target image also includes a target value of marking points;

[0259] The second determining module 704 is further configured to:

[0260] According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

[0261] In one or more implementations of this embodiment, the second determining module 704 is further configured to:

[0262] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0263]

[0264] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marking point in the marking coordinate system, is the coordinate of the marker point in the pixel coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0265] In one or more implementations of this embodiment, the first determining module 702 is further configured to:

[0266] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0267] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0268] In one or more implementations of this embodiment, the device further includes:

[0269] The correction module is configured to correct the position and posture of the corresponding marker in the world coordinate system to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0270] In one or more implementations of this embodiment, the correction module is further configured to:

[0271] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0272] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0273] In one or more implementations of this embodiment, the first building module 706 is further configured to:

[0274] Determine the boundary of the movable area corresponding to the corresponding mark;

[0275] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0276] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0277] In one or more implementations of this embodiment, the first building module 706 is further configured to:

[0278] The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0279] In one or more implementations of this embodiment, the first building module 706 is further configured to:

[0280] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0281]

[0282] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0283] In one or more implementations of this embodiment, the device further includes:

[0284] A saving module, configured to save the group number of the corresponding mark and the position and posture in the world coordinate system;

[0285] The second construction module is configured to load the initial map when a map loading request is detected, save the corresponding markers to the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, and construct a target map of the target area.

[0286] In one or more implementations of this embodiment, the device further includes:

[0287] A third determining module is configured to obtain an image taken at the current position, and determine at least one set of preset marks included in the image taken at the current position;

[0288] A selection module is configured to select a target preset mark from the at least one group of preset marks;

[0289] The fourth determination module is configured to determine the camera posture according to the posture of the selected target preset mark in the world coordinate system.

[0290] In one or more implementations of this embodiment, the fourth determining module is further configured to:

[0291] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0292] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0293] The above is a schematic scheme of a mapping device of this embodiment. It should be noted that the technical scheme of the mapping device and the technical scheme of the mapping method described above are of the same concept, and the details not described in detail in the technical scheme of the mapping device can be referred to the description of the technical scheme of the mapping method described above.

[0294] An embodiment of the present application discloses a robot, see Figure 8 , comprising a mechanical body 80, on which at least one processor 801 and at least one memory 802 for storing computer instructions are arranged.

[0295] The memory 802 is used to store a program that supports the robot to execute the control method in the aforementioned embodiment, and the processor 801 is configured to execute the program stored in the memory.

[0296] In addition to one or more processors 801 and one or more memories 802, the mechanical body 80 is also provided with some basic components of the robot, such as a drive component, a cleaning component, a camera, a sensor component, a power supply component, etc. Optionally, the drive component may include a drive wheel, a drive motor, a universal wheel, etc. Optionally, the cleaning component may include a cleaning motor, a floor brush, a dust brush, a dust suction fan, etc. These basic components contained in different robots and the composition of the basic components may vary, and the embodiments of the present application list only some examples.

[0297] It is worth noting that one or more processors 801 and one or more memories 802 may be disposed inside the mechanical body or on the surface of the mechanical body 80 .

[0298] The mechanical body 80 is an actuator for the robot to complete the task, and can perform the operation specified by the processor in a certain environment. The mechanical body 80 reflects the appearance of the robot to a certain extent. In this embodiment, the appearance of the robot is not limited, for example, it can be a circle, an ellipse, a triangle, a convex polygon, etc.

[0299] One or more memories 802 are mainly used to store computer instructions, which can be executed by one or more processors 801, so that one or more processors 801 can control the mechanical body 800 of the robot to perform the robot control method. In addition to storing computer instructions, one or more memories 802 can also be configured to store various other data to support operations on the robot. Examples of such data include instructions for any application or method operating on the robot, map data of the environment / scene where the robot is located, information about the area to be cleaned, cleaning time, etc.

[0300] One or more memories 802 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0301] One or more processors 801 can be regarded as a robot control system, which can be used to execute computer instructions stored in one or more memories 802 to implement a robot control method.

[0302] The at least one processor 801 is used to execute computer instructions to implement the following method:

[0303] Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system;

[0304] Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system;

[0305] A target map of the target area is constructed according to the position and posture of the corresponding marker in the world coordinate system.

[0306] Optionally, in the process of determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the at least one processor is further used to:

[0307] The camera coordinate system of the first frame of the target image in the at least one frame of the target image is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system are converted into the coordinates of the corresponding mark in the world coordinate system.

[0308] In one or more implementations of this embodiment, the target area includes a preset value group of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and a corresponding mark of a group of the preset marks in the at least one frame of the target image also includes a target value of marking points;

[0309] In the process of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system, the at least one processor is further used to:

[0310] According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

[0311] In one or more implementations of this embodiment, in the process of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system according to the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the at least one processor is further used to:

[0312] The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula:

[0313]

[0314] in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marker point in the imaging plane coordinate system, is the coordinate of the marker point in the image coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

[0315] In one or more implementations of this embodiment, in the process of determining the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system, the at least one processor is further configured to:

[0316] Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image;

[0317] According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

[0318] In one or more implementations of this embodiment, after determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the at least one processor is further configured to:

[0319] The position and posture of the corresponding marker in the world coordinate system are corrected to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

[0320] In one or more implementations of this embodiment, in the process of correcting the position and posture of the corresponding marker in the world coordinate system, the at least one processor is further configured to:

[0321] Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image;

[0322] The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

[0323] In one or more implementations of this embodiment, in the process of constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further used to:

[0324] Determine the boundary of the movable area corresponding to the corresponding mark;

[0325] Determining the movable area corresponding to the corresponding mark according to the boundary of the movable area;

[0326] A target map of the target area is constructed according to the positions and postures of the movable area and the corresponding markers in a world coordinate system.

[0327] In one or more implementations of this embodiment, in the process of determining the boundary of the movable area corresponding to the corresponding mark, the at least one processor is further configured to:

[0328] The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

[0329] In one or more implementations of this embodiment, in the process of determining the boundary of the movable area corresponding to the corresponding mark, the at least one processor is further configured to:

[0330] The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula:

[0331]

[0332] Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

[0333] In one or more implementations of this embodiment, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further configured to:

[0334] Saving the group number of the corresponding marker and its position in the world coordinate system;

[0335] When a map loading request is detected, the initial map is loaded, and the corresponding markers are saved in the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, so as to construct a target map of the target area.

[0336] In one or more implementations of this embodiment, after constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the at least one processor is further configured to:

[0337] Acquire an image captured at the current position, and determine at least one set of preset marks included in the image captured at the current position;

[0338] Selecting a target preset mark from the at least one group of preset marks;

[0339] The camera position is determined according to the position of the selected target preset marker in the world coordinate system.

[0340] In one or more implementations of this embodiment, in the process of determining the camera pose according to the pose of the selected target preset marker in the world coordinate system, the at least one processor is further configured to:

[0341] Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position;

[0342] The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

[0343] The above is a schematic scheme of a robot of this embodiment. It should be noted that the technical scheme of the robot and the technical scheme of the above-mentioned mapping method belong to the same concept, and the details not described in detail in the technical scheme of the robot can be referred to the description of the technical scheme of the above-mentioned mapping method.

[0344] An embodiment of the present invention further provides a computer-readable storage medium storing computer instructions, which are used when executed by a processor to:

[0345] Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system;

[0346] Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system;

[0347] A target map of the target area is constructed according to the position and posture of the corresponding marker in the world coordinate system.

[0348] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the above-mentioned mapping method belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the above-mentioned mapping method.

[0349] The above describes specific embodiments of the present invention. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0350] The computer instructions include computer program codes, which may be in source code form, object code form, executable files or some intermediate forms, etc. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer-readable medium may be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.

[0351] It should be noted that, for the convenience of description, the aforementioned method embodiments are all described as a series of action combinations, but those skilled in the art should be aware that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0352] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0353] The preferred embodiments of the present invention disclosed above are only used to help explain the present invention. The optional embodiments do not describe all the details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the present invention. The present invention selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A mapping method, characterized in that: The method comprises: Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system; Constructing a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system; Wherein, constructing a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system includes: Determine the boundary of the movable area corresponding to the corresponding marker, wherein the movable area is the minimum range of the marker observed in various rotation and position states; determine the movable area corresponding to the corresponding marker based on the boundary of the movable area; and construct a target map of the target area based on the poses of the movable area and the corresponding marker in the world coordinate system.

2. The mapping method according to claim 1, characterized in that: Determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system includes: The camera coordinate system of the first target image in the at least one target image frame is used as the world coordinate system, and the position and posture of the corresponding mark in the camera coordinate system is converted into the position and posture of the corresponding mark in the world coordinate system.

3. The mapping method according to claim 2, characterized in that: The target area includes a preset value group of preset marks, each group of the preset marks corresponds to a group number, a group of the preset marks includes a target value of marking points, and a corresponding mark of a group of the preset marks in the at least one frame of the target image also includes a target value of marking points; The step of converting the pose of the corresponding mark in the camera coordinate system into the pose of the corresponding mark in the world coordinate system includes: According to the coordinates of the target numerical marking points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system, the posture of the corresponding mark in the camera coordinate system is converted into the posture of the corresponding mark in the world coordinate system.

4. The mapping method according to claim 3, characterized in that: The step of converting the position and posture of the corresponding mark in the camera coordinate system into the position and posture of the corresponding mark in the world coordinate system according to the coordinates of the target numerical mark points included in the corresponding mark in the imaging plane coordinate system and the coordinates in the image coordinate system comprises: The position and posture of the corresponding marker in the world coordinate system are determined according to the following formula: in, represents the coordinate transformation from the world coordinate system to the camera coordinate system corresponding to the j-th frame image, represents the marker pose of group number k in the world coordinate system, P i is the position of the marker point in the imaging plane coordinate system, is the coordinate of the marker point in the image coordinate system, i represents the specified marker point among the target numerical marker points, π c Refers to projecting the marker points in the camera coordinate system to the pixel plane.

5. The mapping method according to claim 1, characterized in that: The determining of the position of a corresponding mark of a preset mark in the target area in the at least one frame of the target image in the camera coordinate system comprises: Determine the pixel position of each mark point included in the corresponding mark in the at least one frame of the target image; According to the pixel position, the position and posture of the corresponding mark in the camera coordinate system are determined.

6. The mapping method according to claim 1, characterized in that: After determining the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system, the method further includes: The position and posture of the corresponding marker in the world coordinate system are corrected to obtain the corrected position and posture of the corresponding marker in the world coordinate system.

7. The mapping method according to claim 6, characterized in that: The correcting the position and posture of the corresponding marker in the world coordinate system includes: Selecting a frame of target image from valid frame of target image in the at least one frame of target image as a first target image; The position of the corresponding mark in the first target image in the world coordinate system is taken as the first correction vertex, and the second target image including the first correction vertex is determined in the valid frame target image. The position of the corresponding mark in the second target image that is not included in the first target image in the world coordinate system is taken as a fixed constraint condition for correction.

8. The mapping method according to claim 1, characterized in that: The determining the boundary of the movable area corresponding to the corresponding mark includes: The boundary of the movable area corresponding to the corresponding mark is determined according to the width of the target image, the horizontal coordinates and focal length of the center point of the target image, the installation height of the preset mark relative to the camera, and the side length of the circumscribed rectangle of the preset mark.

9. The mapping method according to claim 8, characterized in that: The determining the boundary of the movable area corresponding to the corresponding mark includes: The boundary of the movable area corresponding to the corresponding mark is determined according to the following formula: Among them, s is the boundary of the movable area, v is the width of the target image, h is the installation height of the preset mark relative to the camera, a is the side length of the circumscribed rectangle of the preset mark, cy and fy are the coordinates and focal length of the center point of the target image in the y-axis direction, respectively.

10. The mapping method according to claim 1, characterized in that: After constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the method further includes: Saving the group number of the corresponding marker and its position in the world coordinate system; When a map loading request is detected, the initial map is loaded, and the corresponding markers are saved in the initial map according to the group numbers of the corresponding markers and the positions in the world coordinate system, so as to construct a target map of the target area.

11. The mapping method according to claim 1, characterized in that: After constructing the target map of the target area according to the position and posture of the corresponding mark in the world coordinate system, the method further includes: Acquire an image captured at the current position, and determine at least one set of preset marks included in the image captured at the current position; Selecting a target preset mark from the at least one group of preset marks; The camera position is determined according to the position of the selected target preset marker in the world coordinate system.

12. The mapping method according to claim 11, characterized in that: The step of determining the camera position and posture according to the selected coordinates of the preset target mark in the world coordinate system includes: Determine the pixel points corresponding to each mark point included in the target preset mark in the image taken at the current position; The projection relationship is calculated through the coordinates of each mark point included in the preset mark of the target in the world coordinate system and the corresponding pixel points to determine the camera posture.

13. A mapping device, characterized in that: The device comprises: A first determination module is configured to obtain at least one frame of target image of a target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; A second determination module is configured to determine the position and posture of the corresponding mark in the world coordinate system according to the position and posture of the corresponding mark in the camera coordinate system; A first construction module is configured to construct a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system; Wherein, the first building module is further configured as follows: Determine the boundary of the movable area corresponding to the corresponding marker, wherein the movable area is the minimum range of the marker observed in various rotation and position states; determine the movable area corresponding to the corresponding marker based on the boundary of the movable area; and construct a target map of the target area based on the poses of the movable area and the corresponding marker in the world coordinate system.

14. A robot, characterized in that: The invention comprises a mechanical body, wherein at least one processor and at least one memory storing computer instructions are arranged on the mechanical body; The at least one processor is used to execute computer instructions to implement the following method: Acquire at least one frame of target image of the target area, and determine the position and posture of a corresponding mark of a preset mark in the target area in the at least one frame of target image in a camera coordinate system; Determine the position and posture of the corresponding marker in the world coordinate system according to the position and posture of the corresponding marker in the camera coordinate system; Constructing a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system; Wherein, constructing a target map of the target area according to the position and posture of the corresponding mark in the world coordinate system includes: Determine the boundary of the movable area corresponding to the corresponding marker, wherein the movable area is the minimum range of the marker observed in various rotation and position states; determine the movable area corresponding to the corresponding marker based on the boundary of the movable area; and construct a target map of the target area based on the poses of the movable area and the corresponding marker in the world coordinate system.

15. A computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the steps of the mapping method according to any one of claims 1 to 12.

Images