Human and mobile robot global positioning method in indoor environment

Abstract

The invention provides a human and mobile robot global positioning method in an indoor environment, and the method is characterized in that the method comprises the following steps: S1, setting a plurality of human landmarks on a ceiling in the indoor environment in advance before work, and carrying out the accurate calibration of the coordinates of the human landmarks in a world coordinate system, a first camera with a vertically upward optical axis and a second camera with a forward optical axis are arranged on the mobile robot; s2, during working, the mobile robot shoots an artificial landmark through a first camera, and then precise positioning of the mobile robot is achieved under the assistance of the artificial landmark; and S3, the mobile robot shoots a front image through a second camera to determine the position of the pedestrian. According to the global positioning method for the person and the mobile robot in the indoor environment, accurate positioning of the mobile robot is achieved based on the artificial landmark, accurate positioning of the pedestrian is achieved based on a mask centroid estimation method, and therefore accurate global positioning of the person and the mobile robot in the indoor environment is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of global positioning, in particular to a method for global positioning of a person and a mobile robot in an indoor environment. BACKGROUND

[0002] At present, mobile robots have been widely used in various places to provide various services for users. Positioning and navigation technology is an important prerequisite for indoor mobile robots to autonomously complete work, and the difference in positioning and navigation technology directly affects the use experience of mobile robots. Realizing pedestrian detection in a complex indoor environment is a research difficulty and hotspot in this field.

[0003] In the prior art, an indoor mobile robot mainly calculates the distance accurately through an inertial sensor, Wi-Fi, a laser radar or an image sensor to complete positioning and navigation work, and determines the human target to be detected by calibrating and training the image of the camera of the mobile robot. However, such a scheme still has the problem of insufficient accuracy, and it is difficult to realize accurate global positioning of a person and a mobile robot. Therefore, it is necessary to design a method for global positioning of a person and a mobile robot in an indoor environment. SUMMARY

[0004] The purpose of the present application is to provide a method for global positioning of a person and a mobile robot in an indoor environment, which is used to solve the technical problems in the background art.

[0005] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0006] A method for global positioning of a person and a mobile robot in an indoor environment, comprising the following steps:

[0007] S1, before work, a plurality of artificial landmarks are set on the ceiling in the indoor environment, and the coordinates of the artificial landmarks in the world coordinate system are accurately calibrated, and a first camera with an optical axis vertically upward and a second camera with an optical axis toward the front are set on the mobile robot;

[0008] S2, during work, the mobile robot captures the artificial landmarks through the first camera, and then realizes accurate positioning of the mobile robot itself with the aid of the artificial landmarks;

[0009] S3, the mobile robot captures the front view through the second camera to determine the position of the pedestrian.

[0010] Further, in step S1, an AprilTag two-dimensional code pattern is selected as the artificial landmark, the world coordinates and physical size of each two-dimensional code pattern are known information, and a unique ID is encoded.

[0011] Further, before step S2, the conversion relationship between the pixel coordinate system of the camera and the world coordinate system is also derived by the following method:

[0012] The pixel coordinate system of the camera takes the upper left corner of the image as the origin, the image coordinate system takes the center point of the image as the origin, and the pixel coordinates where the origin of the image coordinate system is marked as The conversion relationship between the pixel coordinates and the image coordinates can be written as:

[0013] ;

[0014] wherein, is the pixel coordinate, is the image coordinate, is the pixel size, is the actual size of each pixel of the imaging sensor of the camera;

[0015] In addition, the camera coordinate system takes the projection focal point as the origin, the coordinate system whose axis coincides with the optical axis of the camera, according to the pinhole model, the coordinate conversion between the image coordinate system and the camera coordinate system is represented as:

[0016] ;

[0017] wherein, is the coordinate in the camera coordinate system, is the imaging focal length of the pinhole model;

[0018] Let the rotation matrix and the translation matrix of the world coordinate system to the camera coordinate system be R and T respectively, and the conversion relationship between the camera coordinates and the world coordinates is:

[0019] ;

[0020] wherein, is the coordinate in the world coordinate system, and the conversion relationship between the pixel coordinates and the world coordinates can be obtained by comprehensively using the above formulas as:

[0021] ;

[0022] wherein, , Let the camera intrinsic matrix be , and the conversion matrix between the camera coordinate system and the world coordinate system be , then:

[0023] ;

[0024] Thus, the conversion relationship between the world coordinates of a certain point and the pixel coordinates after imaging through the pinhole model and the photosensitive element is obtained, wherein the camera intrinsic matrix can be obtained by calibration in advance, and only the conversion matrix is unknown.

[0025] In step S2, the conversion matrix is determined with the aid of artificial landmarks, and the above formula is used to convert the pixel coordinates into world coordinates, and the conversion between the mobile robot coordinate system and the camera coordinate system can be obtained according to the camera installation position, thereby realizing accurate positioning of the mobile robot.

[0026] Further, in step S2, the following steps are specifically included:

[0027] S21, the first camera adopts a monocular RGB camera to capture an RGB image of an AprilTag two-dimensional code pattern as an artificial landmark;

[0028] S22, the captured RGB image is grayed by adaptive threshold processing;

[0029] S23, the image edge and contour information are extracted by gradient detection;

[0030] S24, the quadrilateral pattern of the AprilTag two-dimensional code is extracted by adjacent edge splicing;

[0031] S25, the content surrounded by the quadrilateral pattern is decoded, and compared with the AprilTag information saved in the database, and if the ID comparison is successful, the pixel coordinates of the four corners of the quadrilateral are output;

[0032] S26, the conversion matrix is solved by re-projection iteration method using the pixel coordinates of the four corners and the world coordinates calibrated in advance, thereby realizing accurate positioning of the mobile robot.

[0033] Further, in step S3, the following steps are specifically included:

[0034] S31, the second camera adopts an RGB-D camera to simultaneously capture an RGB image and a depth image;

[0035] S32, for the RGB image, a mask covering the pedestrian is obtained by instance segmentation, and then mask centroid coordinate estimation is performed to obtain the pixel coordinates of the mask centroid;

[0036] S33, the centroid coordinates are corresponded to the depth image to obtain the corresponding depth value at the mask centroid, complete the depth estimation, and realize pedestrian positioning.

[0037] Further, in step S32, when performing mask centroid coordinate estimation, specifically, the binary image of the mask covering the pedestrian is recorded in the form of a matrix as Wherein m and n are mask pixel resolution;

[0038] The pixel point value of the target individual in the binary image is 1, and the pixel point value of other pixel points is 0. The pixel coordinate in the binary image is The pixel point value of the target individual in the binary image is 1, and the pixel point value of other pixel points is 0. That is:

[0039] ;

[0040] The expression of the mask centroid coordinate calculation can be written as:

[0041] ;

[0042] That is the pixel coordinate of the mask centroid.

[0043] Further, in step S33, the depth value corresponding to the mask centroid is represented by Then, according to the coordinate system conversion relationship, the coordinates of the pedestrian in the world coordinate system are obtained by the following formula:

[0044] ;

[0045] Wherein, The pedestrian coordinates are obtained, K is the internal parameter matrix of the second camera, which needs to be obtained by pre-calibration, and T is the coordinate conversion matrix from the camera coordinate system to the world coordinate system.

[0046] Compared with the prior art, the global positioning method of the person and the mobile robot in the indoor environment provided by the present application realizes the accurate positioning of the mobile robot based on the artificial landmark, and realizes the accurate positioning of the pedestrian based on the mask centroid estimation method, so as to realize the accurate global positioning of the person and the mobile robot in the indoor environment. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0048] Figure 1 The schematic diagram of the global positioning method of the person and the mobile robot in the indoor environment provided by the present application is configured in the indoor environment.

[0049] Figure 2 The schematic diagram of processing the two-dimensional code RGB image in the specific embodiment is shown.

[0050] Figure 3 This is a schematic diagram comparing the center of the rectangle and the centroid of the mask in a specific embodiment;

[0051] Figure 4 This is a schematic diagram illustrating the process of mask centroid estimation and depth estimation in a specific embodiment. Detailed Implementation

[0052] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following description, in conjunction with the accompanying drawings and specific embodiments, further explains how this invention is implemented.

[0053] In one specific embodiment, the present invention provides a method for global localization of humans and mobile robots in an indoor environment, comprising the following steps:

[0054] In step S1, before starting work, refer to Figure 1 As shown, several artificial landmarks 2 are pre-set on the ceiling in the indoor environment, and the coordinates of the artificial landmarks 2 in the world coordinate system are accurately calibrated. A first camera 4 with the optical axis pointing vertically upward and a second camera 5 with the optical axis pointing forward are set on the mobile robot 1.

[0055] In this embodiment, the AprilTag QR code pattern is selected as the artificial landmark 2. The world coordinates and physical size of each QR code pattern are known information, and it is encoded with a unique ID. In addition, multiple artificial landmarks 2 can be evenly laid on the ceiling.

[0056] In step S2, during operation, the mobile robot 1 uses the first camera 4 to photograph the artificial landmark 2, thereby achieving precise positioning of the mobile robot 1 with the assistance of the artificial landmark 2.

[0057] Before step S2, the transformation relationship between the camera's pixel coordinate system and the world coordinate system is derived using the following method:

[0058] The camera's pixel coordinate system has its origin at the top left corner of the image, while the image coordinate system has its origin at the center of the image. The pixel coordinates of the origin of the image coordinate system are denoted as... The transformation relationship between pixel coordinates and image coordinates can be written as:

[0059] ;

[0060] in, For pixel coordinates, For image coordinates, and This refers to the actual size of each pixel on the camera's imaging sensor.

[0061] In addition, the camera coordinate system has its origin at the projection focus. The coordinate system in which the axis coincides with the camera optical axis, according to the pinhole model, the coordinate conversion between the image coordinate system and the camera coordinate system is represented as:

[0062] ;

[0063] wherein, is the coordinate in the camera coordinate system, is the imaging focal length of the pinhole model;

[0064] Let the rotation matrix and the translation matrix of the world coordinate system to the camera coordinate system be R and T respectively, then the conversion relationship between the camera coordinate and the world coordinate is:

[0065] ;

[0066] In the formula, is the coordinate in the world coordinate system, and the conversion relationship between the pixel coordinate and the world coordinate can be obtained by comprehensively considering the above formula as:

[0067] ;

[0068] wherein, , Let the camera intrinsic matrix be , and the conversion matrix between the camera coordinate system and the world coordinate system be , then:

[0069] ;

[0070] Thus, the conversion relationship between the world coordinate of a certain point and the pixel coordinate after imaging through the pinhole model and the photosensitive element is obtained, wherein the camera intrinsic matrix can be obtained in advance through calibration, and therefore only the conversion matrix is unknown;

[0071] In step S2, the conversion matrix is determined with the aid of artificial landmarks, and the pixel coordinate can be converted into the world coordinate by bringing the above formula, and the conversion between the mobile robot coordinate system and the camera coordinate system can be obtained according to the camera installation position, thereby realizing the accurate positioning of the mobile robot.

[0072] In step S2, the following steps are specifically included:

[0073] S21, the first camera adopts a monocular RGB camera to shoot an RGB image of an AprilTag two-dimensional code pattern as an artificial landmark.

[0074] S22, the RGB image obtained by shooting is grayed through adaptive threshold processing.

[0075] S23, extract image edge and contour information through gradient detection.

[0076] S24, extract the quadrilateral pattern of the AprilTag two-dimensional code through adjacent edge splicing.

[0077] S25, decode the content surrounded by the quadrilateral pattern, and compare with the AprilTag information saved in the database, if the ID comparison is successful, output the pixel coordinates of the four corners of the quadrilateral.

[0078] S26, using the pixel coordinates of the four corners and the pre-calibrated world coordinates, the conversion matrix is solved by re-projection iteration method , and the accurate positioning of the mobile robot is realized. The principle of re-projection iteration method is to continuously iterate and optimize the model parameters, and minimize the error between the calculated position of three-dimensional points projected to the image plane and the actual observed position.

[0079] In this embodiment, the processing flow of the RGB image is shown in Figure 2 . The AprilTag two-dimensional code used in this embodiment has good encoding mode and analysis speed, and can provide four coplanar corner points as feature points, so that the conversion matrix can be calculated.

[0080] In step S3, the mobile robot 1 shoots the front image through the second camera 5 to determine the position of the pedestrian 3. Specifically, the following steps are included:

[0081] S31, the second camera uses an RGB-D camera to simultaneously acquire an RGB image and a depth image.

[0082] Due to the lack of depth information, it is difficult to determine the position of the pedestrian in space only using the RGB image, therefore, the present application uses an RGB-D camera to combine the RGB image and the depth image for pedestrian positioning.

[0083] S32, for the RGB image, the mask covering the pedestrian is obtained through instance segmentation, and then the mask centroid coordinate estimation is performed to obtain the pixel coordinates of the mask centroid.

[0084] Referring to Figure 3 , the pedestrian area can be detected from the image through a traditional target detection algorithm first, and the output of the target detection algorithm is generally a rectangular bounding box. The traditional method uses the center of the rectangular box output by the pedestrian detection model as the depth value feature point, which may be offset when the pedestrian has large body movements, resulting in incorrect depth estimation and reducing the positioning accuracy. Therefore, the present application adopts a pedestrian positioning method based on mask centroid estimation, which can adapt to scenes where the pedestrian has large body movements.

[0085] When the mask centroid coordinate is estimated, specifically, the binary image of the mask covering the pedestrian is recorded in the form of a matrix as wherein m and n are the mask pixel resolution;

[0086] The pixel point value of the binary image belonging to the target individual is 1, and the pixel point value of other pixel points is 0, and the pixel point value of The pixel coordinate in the pixel point is is recorded as that is:

[0087] ;

[0088] The expression of the mask centroid coordinate calculation can be written as:

[0089] ;

[0090] That is, the pixel coordinate of the mask centroid.

[0091] S33, referring to Figure 4 The centroid coordinate is corresponded to the depth image to obtain the depth value corresponding to the mask centroid, the depth estimation is completed, and the pedestrian positioning is realized.

[0092] The depth value corresponding to the mask centroid is represented by Then, according to the conversion relationship of the coordinate system, the pedestrian coordinates in the world coordinate system are obtained by the following formula:

[0093] ;

[0094] wherein is the obtained pedestrian coordinate, K is the internal parameter matrix of the second camera, which needs to be obtained by prior calibration, and T is the coordinate conversion matrix of the camera coordinate system to the world coordinate system.

[0095] In summary, the indoor environment human and mobile robot global positioning method provided by the present application realizes the accurate positioning of the mobile robot itself based on artificial landmarks, and realizes the accurate positioning of the pedestrian based on the mask centroid estimation method, so as to realize the accurate global positioning of the human and the mobile robot in the indoor environment.

[0096] Finally, it should be pointed out that the above-mentioned only for the embodiment of the present application, and not limited to the patent range of the present application, any equivalent structure or equivalent process transformation using the content of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection range of the present application.

Claims

1. A method for global localization of humans and mobile robots in an indoor environment, characterized in that, Includes the following steps: S1. Before starting work, set up several artificial landmarks on the ceiling in the indoor environment and accurately calibrate the coordinates of the artificial landmarks in the world coordinate system. Also, set up a first camera with the optical axis pointing vertically upward and a second camera with the optical axis pointing forward on the mobile robot. S2. During operation, the mobile robot uses the first camera to photograph artificial landmarks, thereby achieving precise positioning of the mobile robot with the assistance of the artificial landmarks. S3. The mobile robot uses a second camera to take pictures of the area in front of it to determine the location of pedestrians.

2. The method for global localization of humans and mobile robots in an indoor environment according to claim 1, characterized in that, In step S1, AprilTag QR code patterns are selected as artificial landmarks. The world coordinates and physical dimensions of each QR code pattern are known information, and a unique ID is encoded.

3. The method for global positioning of humans and mobile robots in an indoor environment according to claim 2, characterized in that, Before step S2, the transformation relationship between the camera's pixel coordinate system and the world coordinate system is derived using the following method: The camera's pixel coordinate system has its origin at the top left corner of the image, while the image coordinate system has its origin at the center of the image. The pixel coordinates of the origin of the image coordinate system are denoted as... The transformation relationship between pixel coordinates and image coordinates can be written as: ； in, For pixel coordinates, For image coordinates, and This refers to the actual size of each pixel on the camera's imaging sensor. In addition, the camera coordinate system has its origin at the projection focus. For a coordinate system where the optical axis coincides with the camera's optical axis, the coordinate transformation between the image coordinate system and the camera coordinate system, based on the pinhole model, is expressed as: ； in, The coordinates are in the camera coordinate system. The focal length for pinhole model imaging; Let R and T be the rotation and translation matrices from the world coordinate system to the camera coordinate system, respectively. Then the transformation relationship between camera coordinates and world coordinates is: ； In the formula, Given the coordinates in the world coordinate system, combining the above formulas, we can obtain the transformation relationship between pixel coordinates and world coordinates as follows: ； in, , Let the camera intrinsic parameter matrix be denoted as The transformation matrix between the camera coordinate system and the world coordinate system is denoted as... Then we have: ； Thus, the transformation relationship between the world coordinates of a point and the pixel coordinates after imaging through the pinhole model and the photosensitive element is obtained, where the camera's intrinsic parameter matrix is... It can be obtained in advance through calibration, therefore only the transformation matrix is ​​needed. It is unknown; In step S2, the transformation matrix is ​​determined with the aid of artificial landmarks. Substituting these values ​​into the above formula allows us to convert pixel coordinates to world coordinates. The conversion between the mobile robot coordinate system and the camera coordinate system can be obtained based on the camera's installation position, thereby enabling precise positioning of the mobile robot.

4. The method for global positioning of humans and mobile robots in an indoor environment according to claim 3, characterized in that, Step S2 specifically includes the following steps: S21. The first camera is a monocular RGB camera, which captures an RGB image of the AprilTag QR code pattern as an artificial landmark. S22. Convert the captured RGB image to grayscale using adaptive thresholding. S23. Extract image edge and contour information through gradient detection; S24. Extract the quadrilateral pattern of the AprilTag QR code by splicing adjacent edges; S25. Decode the content enclosed by the quadrilateral pattern and compare it with the AprilTag information stored in the database. If the ID matching is successful, output the pixel coordinates of the four corner points of the quadrilateral. S26. Using the pixel coordinates of the four corner points and the pre-calibrated world coordinates, calculate the transformation matrix using the reprojection iteration method. This enables precise positioning of mobile robots.

5. The method for global positioning of humans and mobile robots in an indoor environment according to claim 3, characterized in that, Step S3 specifically includes the following steps: S31. The second camera is an RGB-D camera to simultaneously acquire RGB images and depth images; S32. For an RGB image, obtain a mask covering the pedestrian through instance segmentation, and then estimate the centroid coordinates of the mask to obtain the pixel coordinates of the centroid. S33. Map the centroid coordinates to the depth image to obtain the depth value corresponding to the centroid of the mask, complete the depth estimation, and realize pedestrian localization.

6. The method for global positioning of humans and mobile robots in an indoor environment according to claim 5, characterized in that, In step S32, when estimating the centroid coordinates of the mask, specifically, the binary image of the mask covering the pedestrian is denoted as a matrix. , where m and n are the mask pixel resolution; In this binary image, pixels belonging to the target individual have a value of 1, while other pixels have a value of 0. The middle pixel coordinates are The pixel value is denoted as ,Right now: ； The expression for calculating the centroid coordinates of the mask can be written as: ； These are the pixel coordinates of the mask's centroid.

7. The method for global localization of humans and mobile robots in an indoor environment according to claim 6, characterized in that, In step S33, the depth value corresponding to the centroid of the mask is used Then, based on the coordinate system transformation relationship, the pedestrian's coordinates in the world coordinate system are obtained using the following formula: ； in, To obtain the pedestrian coordinates, K is the intrinsic parameter matrix of the second camera, which needs to be obtained through prior calibration, and T is the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

Images