Global localization methods for humans and mobile robots in indoor environments

Abstract

This invention provides a method for global positioning of humans and mobile robots in an indoor environment, characterized by the following steps: S1. Before operation, several artificial landmarks are pre-set on the ceiling of the indoor environment, and the coordinates of the artificial landmarks in the world coordinate system are accurately calibrated. A first camera with its optical axis pointing vertically upward and a second camera with its optical axis pointing forward are set on the mobile robot; S2. During operation, the mobile robot takes pictures of the artificial landmarks through the first camera, thereby achieving precise positioning of the mobile robot itself with the assistance of the artificial landmarks; S3. The mobile robot takes pictures of the front view through the second camera to determine the location of pedestrians. The method for global positioning of humans and mobile robots in an indoor environment provided by this invention achieves precise positioning of the mobile robot itself based on artificial landmarks and precise positioning of pedestrians based on the method of mask centroid estimation, thereby achieving precise global positioning of humans and mobile robots in an indoor environment.

Description

Technical Field

[0001] This invention relates to the field of global positioning technology, and in particular to a global positioning method for humans and mobile robots in an indoor environment. Background Technology

[0002] Currently, mobile robots are widely used in various locations to provide users with a variety of services. Positioning and navigation technology is a crucial prerequisite for indoor mobile robots to autonomously complete their tasks. Differences in positioning and navigation technology directly affect the user experience of mobile robots, and pedestrian detection in complex indoor environments is both a challenging and hot research topic in this field.

[0003] In existing technologies, indoor mobile robots primarily rely on inertial sensors, Wi-Fi, LiDAR, or image sensors for precise distance calculation to achieve localization and navigation. They also use camera calibration and training images to identify human targets. However, such solutions still suffer from insufficient accuracy, making it difficult to achieve precise global localization of both humans and mobile robots. Therefore, it is necessary to design a global localization method for humans and mobile robots in indoor environments. Summary of the Invention

[0004] The purpose of this invention is to provide a global positioning method for humans and mobile robots in an indoor environment, in order to solve the technical problems existing in the background art.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A method for global localization of humans and mobile robots in an indoor environment includes the following steps:

[0007] S1. Before starting work, set up several artificial landmarks on the ceiling in the indoor environment and accurately calibrate the coordinates of the artificial landmarks in the world coordinate system. Also, set up a first camera with the optical axis pointing vertically upward and a second camera with the optical axis pointing forward on the mobile robot.

[0008] S2. During operation, the mobile robot uses the first camera to photograph artificial landmarks, thereby achieving precise positioning of the mobile robot with the assistance of the artificial landmarks.

[0009] S3. The mobile robot uses a second camera to take pictures of the area in front of it to determine the location of pedestrians.

[0010] Furthermore, in step S1, the AprilTag QR code pattern is selected as an artificial landmark. The world coordinates and physical size of each QR code pattern are known information, and it is encoded with a unique ID.

[0011] Furthermore, prior to step S2, the transformation relationship between the camera's pixel coordinate system and the world coordinate system is derived using the following method:

[0012] The camera's pixel coordinate system has its origin at the top left corner of the image, while the image coordinate system has its origin at the center of the image. The pixel coordinates of the origin of the image coordinate system are denoted as... The transformation relationship between pixel coordinates and image coordinates can be written as:

[0013] ;

[0014] in, For pixel coordinates, For image coordinates, and This refers to the actual size of each pixel on the camera's imaging sensor.

[0015] In addition, the camera coordinate system has its origin at the projection focus. For a coordinate system where the optical axis coincides with the camera's optical axis, the coordinate transformation between the image coordinate system and the camera coordinate system, based on the pinhole model, is expressed as:

[0016] ;

[0017] in, The coordinates are in the camera coordinate system. The focal length for pinhole model imaging;

[0018] Let R and T be the rotation and translation matrices from the world coordinate system to the camera coordinate system, respectively. Then the transformation relationship between camera coordinates and world coordinates is:

[0019] ;

[0020] In the formula, Given the coordinates in the world coordinate system, combining the above formulas, we can obtain the transformation relationship between pixel coordinates and world coordinates as follows:

[0021] ;

[0022] in, , Let the camera intrinsic parameter matrix be denoted as The transformation matrix between the camera coordinate system and the world coordinate system is denoted as... Then we have:

[0023] ;

[0024] Thus, the transformation relationship between the world coordinates of a point and the pixel coordinates after imaging through the pinhole model and the photosensitive element is obtained, where the camera's intrinsic parameter matrix is... It can be obtained in advance through calibration, therefore only the transformation matrix is ​​needed. It is unknown;

[0025] In step S2, the transformation matrix is ​​determined with the aid of artificial landmarks. Substituting these values ​​into the above formula allows us to convert pixel coordinates to world coordinates. The conversion between the mobile robot coordinate system and the camera coordinate system can be obtained based on the camera's installation position, thereby enabling precise positioning of the mobile robot.

[0026] Furthermore, step S2 specifically includes the following steps:

[0027] S21. The first camera is a monocular RGB camera, which captures an RGB image of the AprilTag QR code pattern as an artificial landmark.

[0028] S22. Convert the captured RGB image to grayscale using adaptive thresholding.

[0029] S23. Extract image edge and contour information through gradient detection;

[0030] S24. Extract the quadrilateral pattern of the AprilTag QR code by splicing adjacent edges;

[0031] S25. Decode the content enclosed by the quadrilateral pattern and compare it with the AprilTag information stored in the database. If the ID matching is successful, output the pixel coordinates of the four corner points of the quadrilateral.

[0032] S26. Using the pixel coordinates of the four corner points and the pre-calibrated world coordinates, calculate the transformation matrix using the reprojection iteration method. This enables precise positioning of mobile robots.

[0033] Furthermore, step S3 specifically includes the following steps:

[0034] S31. The second camera is an RGB-D camera to simultaneously acquire RGB images and depth images;

[0035] S32. For an RGB image, obtain a mask covering the pedestrian through instance segmentation, and then estimate the centroid coordinates of the mask to obtain the pixel coordinates of the centroid.

[0036] S33. Map the centroid coordinates to the depth image to obtain the depth value corresponding to the centroid of the mask, complete the depth estimation, and realize pedestrian localization.

[0037] Further, in step S32, when estimating the centroid coordinates of the mask, specifically, the binary image of the mask covering the pedestrian is denoted as a matrix. , where m and n are the mask pixel resolution;

[0038] In this binary image, pixels belonging to the target individual have a value of 1, while other pixels have a value of 0. The middle pixel coordinates are The pixel value is denoted as ,Right now:

[0039] ;

[0040] The expression for calculating the centroid coordinates of the mask can be written as:

[0041] ;

[0042] These are the pixel coordinates of the mask's centroid.

[0043] Further, in step S33, the depth value corresponding to the centroid of the mask is used... Then, based on the coordinate system transformation relationship, the pedestrian's coordinates in the world coordinate system are obtained using the following formula:

[0044] ;

[0045] in, To obtain the pedestrian coordinates, K is the intrinsic parameter matrix of the second camera, which needs to be obtained through prior calibration, and T is the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0046] Compared with existing technologies, the global positioning method for people and mobile robots in indoor environments provided by this invention achieves accurate positioning of the mobile robot itself based on artificial landmarks and accurate positioning of pedestrians based on mask centroid estimation, thereby achieving accurate global positioning of people and mobile robots in indoor environments. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in this embodiment, the accompanying drawings used in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This is a schematic diagram illustrating the configuration of the global positioning method for humans and mobile robots in an indoor environment provided by the present invention in an indoor environment.

[0049] Figure 2 This is a schematic diagram illustrating the processing of a QR code RGB image in a specific embodiment;

[0050] Figure 3 This is a schematic diagram comparing the center of the rectangle and the centroid of the mask in a specific embodiment;

[0051] Figure 4 This is a schematic diagram illustrating the process of mask centroid estimation and depth estimation in a specific embodiment. Detailed Implementation

[0052] To make the technical means, creative features, objectives and effects of this invention easier to understand, the following description, in conjunction with the accompanying drawings and specific embodiments, further explains how this invention is implemented.

[0053] In one specific embodiment, the present invention provides a method for global localization of humans and mobile robots in an indoor environment, comprising the following steps:

[0054] In step S1, before starting work, refer to Figure 1 As shown, several artificial landmarks 2 are pre-set on the ceiling in the indoor environment, and the coordinates of the artificial landmarks 2 in the world coordinate system are accurately calibrated. A first camera 4 with the optical axis pointing vertically upward and a second camera 5 with the optical axis pointing forward are set on the mobile robot 1.

[0055] In this embodiment, the AprilTag QR code pattern is selected as the artificial landmark 2. The world coordinates and physical size of each QR code pattern are known information, and it is encoded with a unique ID. In addition, multiple artificial landmarks 2 can be evenly laid on the ceiling.

[0056] In step S2, during operation, the mobile robot 1 uses the first camera 4 to photograph the artificial landmark 2, thereby achieving precise positioning of the mobile robot 1 with the assistance of the artificial landmark 2.

[0057] Before step S2, the transformation relationship between the camera's pixel coordinate system and the world coordinate system is derived using the following method:

[0058] The camera's pixel coordinate system has its origin at the top left corner of the image, while the image coordinate system has its origin at the center of the image. The pixel coordinates of the origin of the image coordinate system are denoted as... The transformation relationship between pixel coordinates and image coordinates can be written as:

[0059] ;

[0060] in, For pixel coordinates, For image coordinates, and This refers to the actual size of each pixel on the camera's imaging sensor.

[0061] In addition, the camera coordinate system has its origin at the projection focus. For a coordinate system where the optical axis coincides with the camera's optical axis, the coordinate transformation between the image coordinate system and the camera coordinate system, based on the pinhole model, is expressed as:

[0062] ;

[0063] in, The coordinates are in the camera coordinate system. The focal length for pinhole model imaging;

[0064] Let R and T be the rotation and translation matrices from the world coordinate system to the camera coordinate system, respectively. Then the transformation relationship between camera coordinates and world coordinates is:

[0065] ;

[0066] In the formula, Given the coordinates in the world coordinate system, combining the above formulas, we can obtain the transformation relationship between pixel coordinates and world coordinates as follows:

[0067] ;

[0068] in, , Let the camera intrinsic parameter matrix be denoted as The transformation matrix between the camera coordinate system and the world coordinate system is denoted as... Then we have:

[0069] ;

[0070] Thus, the transformation relationship between the world coordinates of a point and the pixel coordinates after imaging through the pinhole model and the photosensitive element is obtained, where the camera's intrinsic parameter matrix is... It can be obtained in advance through calibration, therefore only the transformation matrix is ​​needed. It is unknown;

[0071] In step S2, the transformation matrix is ​​determined with the aid of artificial landmarks. Substituting these values ​​into the above formula allows us to convert pixel coordinates to world coordinates. The conversion between the mobile robot coordinate system and the camera coordinate system can be obtained based on the camera's installation position, thereby enabling precise positioning of the mobile robot.

[0072] Step S2 specifically includes the following steps:

[0073] S21. The first camera uses a monocular RGB camera to capture RGB images of the AprilTag QR code pattern as an artificial landmark.

[0074] S22. Convert the captured RGB image to grayscale using adaptive thresholding.

[0075] S23. Extract image edge and contour information through gradient detection.

[0076] S24. Extract the quadrilateral pattern of the AprilTag QR code by splicing adjacent edges.

[0077] S25. Decode the content enclosed by the quadrilateral pattern and compare it with the AprilTag information stored in the database. If the ID matching is successful, output the pixel coordinates of the four corner points of the quadrilateral.

[0078] S26. Using the pixel coordinates of the four corner points and the pre-calibrated world coordinates, calculate the transformation matrix using the reprojection iteration method. This enables precise positioning of mobile robots. The principle of the reprojection iterative method is to continuously iterate and optimize the model parameters to minimize the error between the calculated position of a 3D point projected onto the image plane and its actual observed position.

[0079] In this embodiment, the processing flow for RGB images is as follows: Figure 2 As shown. The AprilTag QR code used in this embodiment has a good encoding method and parsing speed, and can provide four coplanar corner points as feature points, thereby enabling the calculation of the transformation matrix. .

[0080] In step S3, the mobile robot 1 takes a picture of the front using the second camera 5 to determine the position of the pedestrian 3. Specifically, this includes the following steps:

[0081] S31, the second camera is an RGB-D camera, to acquire RGB images and depth images simultaneously.

[0082] Because of the lack of depth information, it is difficult to determine the location of pedestrians in space using only RGB images. Therefore, this invention uses an RGB-D camera to combine RGB images and depth images for pedestrian localization.

[0083] S32. For an RGB image, obtain a mask covering pedestrians through instance segmentation, and then estimate the centroid coordinates of the mask to obtain the pixel coordinates of the centroid.

[0084] Reference Figure 3 As shown, pedestrian areas can be detected from images using traditional object detection algorithms. The output of object detection algorithms is generally a rectangular bounding box. Traditional methods use the center of the rectangular bounding box output by the pedestrian detection model as the depth feature point. This may cause offset when the pedestrian's movements are large, leading to incorrect depth estimation and reduced positioning accuracy. Therefore, this invention adopts a pedestrian positioning method based on mask centroid estimation, which can adapt to scenarios where pedestrians have large limb movements.

[0085] When performing mask centroid coordinate estimation, specifically, the binary image of the mask covering the pedestrian is denoted as a matrix. , where m and n are the mask pixel resolution;

[0086] In this binary image, pixels belonging to the target individual have a value of 1, while other pixels have a value of 0. The middle pixel coordinates are The pixel value is denoted as ,Right now:

[0087] ;

[0088] The expression for calculating the centroid coordinates of the mask can be written as:

[0089] ;

[0090] These are the pixel coordinates of the mask's centroid.

[0091] S33. Reference Figure 4 As shown, the centroid coordinates are mapped to the depth image to obtain the depth value corresponding to the centroid of the mask, thus completing the depth estimation and realizing pedestrian localization.

[0092] The depth value corresponding to the centroid of the mask is used Then, based on the coordinate system transformation relationship, the pedestrian's coordinates in the world coordinate system are obtained using the following formula:

[0093] ;

[0094] in, To obtain the pedestrian coordinates, K is the intrinsic parameter matrix of the second camera, which needs to be obtained through prior calibration, and T is the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

[0095] In summary, the global positioning method for humans and mobile robots in indoor environments provided by this invention achieves accurate positioning of the mobile robot itself based on artificial landmarks and accurate positioning of pedestrians based on mask centroid estimation, thereby achieving accurate global positioning of humans and mobile robots in indoor environments.

[0096] Finally, it should be noted that the above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for global localization of humans and mobile robots in an indoor environment, characterized in that, Includes the following steps: S1. Before starting work, set up several artificial landmarks on the ceiling in the indoor environment and accurately calibrate the coordinates of the artificial landmarks in the world coordinate system. Also, set up a first camera with the optical axis pointing vertically upward and a second camera with the optical axis pointing forward on the mobile robot. S2. During operation, the mobile robot uses the first camera to photograph artificial landmarks, thereby achieving precise positioning of the mobile robot with the assistance of the artificial landmarks. S3. The mobile robot uses a second camera to capture an image of the area in front of it to determine the location of pedestrians; In step S1, AprilTag QR code patterns are selected as artificial landmarks. The world coordinates and physical dimensions of each QR code pattern are known information, and a unique ID is encoded. Before step S2, the transformation relationship between the camera's pixel coordinate system and the world coordinate system is derived using the following method: The camera's pixel coordinate system has its origin at the top left corner of the image, while the image coordinate system has its origin at the center of the image. The pixel coordinates of the origin of the image coordinate system are denoted as... The transformation relationship between pixel coordinates and image coordinates can be written as: ； in, For pixel coordinates, For image coordinates, and This refers to the actual size of each pixel on the camera's imaging sensor. In addition, the camera coordinate system is with the projection focus point as the origin, The coordinate system with the axis coinciding with the camera optical axis is according to the pinhole model, and the coordinate conversion between the image coordinate system and the camera coordinate system is represented as: ； wherein, is the coordinate in the camera coordinate system, is the focal length of the pinhole model imaging; Let R and T be the rotation and translation matrices from the world coordinate system to the camera coordinate system, respectively. Then the transformation relationship between camera coordinates and world coordinates is: ； In the formula, Given the coordinates in the world coordinate system, combining the above formulas, we can obtain the transformation relationship between pixel coordinates and world coordinates as follows: ； in, , Let the camera intrinsic parameter matrix be denoted as The transformation matrix between the camera coordinate system and the world coordinate system is denoted as... Then we have: ； Thus, the transformation relationship between the world coordinates of a point and the pixel coordinates after imaging through the pinhole model and the photosensitive element is obtained, where the camera's intrinsic parameter matrix is... It can be obtained in advance through calibration, therefore only the transformation matrix is ​​needed. It is unknown; In step S2, the transformation matrix is ​​determined with the aid of artificial landmarks. Substituting these values ​​into the above formula allows us to convert pixel coordinates to world coordinates. The conversion between the mobile robot coordinate system and the camera coordinate system can be obtained based on the camera's installation position, thereby enabling precise positioning of the mobile robot.

2. The method of claim 1, wherein, Step S2 specifically includes the following steps: S21. The first camera is a monocular RGB camera, which captures an RGB image of the AprilTag QR code pattern as an artificial landmark. S22. Convert the captured RGB image to grayscale using adaptive thresholding. S23. Extract image edge and contour information through gradient detection; S24. Extract the quadrilateral pattern of the AprilTag QR code by splicing adjacent edges; S25. Decode the content enclosed by the quadrilateral pattern and compare it with the AprilTag information stored in the database. If the ID matching is successful, output the pixel coordinates of the four corner points of the quadrilateral. S26, using the pixel coordinates of the four corner points and the pre-calibrated world coordinates, the conversion matrix is solved by re-projection iteration method , and thus the accurate positioning of the mobile robot is realized.

3. The method of claim 1, wherein, Step S3 specifically includes the following steps: S31. The second camera is an RGB-D camera to simultaneously acquire RGB images and depth images; S32. For an RGB image, obtain a mask covering the pedestrian through instance segmentation, and then estimate the centroid coordinates of the mask to obtain the pixel coordinates of the centroid. S33. Map the centroid coordinates to the depth image to obtain the depth value corresponding to the centroid of the mask, complete the depth estimation, and realize pedestrian localization.

4. The method of claim 3, wherein, In step S32, when estimating the centroid coordinates of the mask, specifically, the binary image of the mask covering the pedestrian is denoted as a matrix. , where m and n are the mask pixel resolution; In this binary image, pixels belonging to the target individual have a value of 1, while other pixels have a value of 0. The middle pixel coordinates are The pixel value is denoted as ,Right now: ； The expression for calculating the centroid coordinates of the mask can be written as: ； i.e. the pixel coordinates of the mask center.

5. The method of claim 4, wherein, In step S33, the depth value corresponding to the centroid of the mask is used Then, based on the coordinate system transformation relationship, the pedestrian's coordinates in the world coordinate system are obtained using the following formula: ； in, To obtain the pedestrian coordinates, K is the intrinsic parameter matrix of the second camera, which needs to be obtained through prior calibration, and T is the coordinate transformation matrix from the camera coordinate system to the world coordinate system.

Images