A method for indoor multi-mobile robot positioning based on camera splicing and area matching

Abstract

The present invention provides a method for indoor multi-mobile robot positioning with camera splicing and area matching, comprising the following steps: firstly, a coordinate conversion model is established, parameters are obtained through camera calibration, and the pixel coordinates of the mobile robot in the pixel coordinate system are converted into world coordinates in the world coordinate system; a label is set on the mobile robot, and the outline of the label in each frame image under the field of view angle is obtained through a joint algorithm of related image processing; finally, the minimum bounding box of each outline is obtained through function calculation, and the area of ​​the label on each mobile robot can be obtained, and the function of multi-target matching can be realized with different areas; at the same time, the center of mass position can be obtained, and finally the orientation and attitude angle of each mobile robot can be obtained. In the case of indoor positioning, the problem of small monocular camera screen is solved by fixing multiple cameras and using the screen splicing algorithm. The present invention solves the problem of multi-robot positioning difficulties, has high positioning accuracy, and reduces the complexity of visual positioning.

Description

Technical Field

[0001] The invention relates to a visual positioning method for multiple indoor mobile robots, belonging to the technical field of visual robots, and in particular to a positioning method for multiple indoor mobile robots involving visions of multiple fixed cameras. Background Art

[0002] Positioning technology is one of the key technologies in mobile robot research and plays a decisive role in the normal operation of robots. The positioning problem of robots is of great significance in the research of mobile robot intelligence and is the key to achieving autonomous navigation and completing complex intelligent tasks in a specific environment.

[0003] Mobile robot indoor positioning refers to the robot sensing the surrounding environment information through sensors and analyzing and processing it, and using specific positioning algorithms to solve its own position information and posture information. At present, visual positioning is more and more widely used in the field of robot positioning. With its advantages of rich information acquisition, strong understanding of the environment, long-term stability and low cost, it has become the most popular direction in the current research of mobile robot indoor positioning technology.

[0004] The existing indoor visual positioning methods still have defects in the actual positioning process and are easily limited by the field of view of a single camera; there is also the problem that the positioning method is relatively complex. For example, the CN210119230U patent discloses an indoor visual positioning system, which mainly locates the target through an image acquisition device and a monocular camera to achieve a visual positioning effect, but its positioning method uses a monochrome lighting module to obtain the image position to achieve target positioning, and the positioning method is relatively complex; the CN209280914U patent discloses an indoor visible light visual positioning system, which mainly achieves an indoor positioning effect through image acquisition of light signals, but its positioning method uses LED lamps to transmit signals for image acquisition to achieve target positioning, and the positioning method is relatively complex. Summary of the invention

[0005] In order to overcome the defects of the background technology, the present invention proposes a method for indoor multi-mobile robot positioning based on camera splicing and area matching.

[0006] The technical means adopted by the present invention are as follows:

[0007] A method for indoor multi-mobile robot positioning with camera splicing and area matching comprises the following steps:

[0008] Step 1, calibrate the monocular camera to obtain camera parameters; at the same time, calibrate the ground map to establish a one-to-one correspondence between the ground map area in the video image captured by the monocular camera and the real ground map area;

[0009] Step 2: Track multiple robots through positioning and tracking algorithms; fix markers on the mobile robots and use the markers as labels. The different areas of each label match the corresponding robots. Track multiple robots by tracking markers of different areas.

[0010] Step 3: Perform centroid positioning, and replace the robot position coordinates with the centroid coordinates of the marker; obtain the vertex coordinates and three side lengths of the marker through two-point positioning, and sort the three side lengths to obtain the shortest length of the three side lengths; then obtain the center coordinates of the bottom edge according to the coordinates of the two vertices of the bottom edge, and obtain the posture angle of the robot in combination with the centroid coordinates;

[0011] Step 4: By fixing multiple cameras and stitching multiple camera images, the activity trajectory image of the mobile robot is obtained for identification and positioning.

[0012] Compared with the prior art, the present invention has the following advantages:

[0013] The present invention uses labels of different areas to play a good matching role and distinguish different mobile robots; the method of the present invention can simultaneously locate multiple mobile robots, improves the positioning time and efficiency, and improves the problem of low robot positioning accuracy.

[0014] The bottom side of each label of the present invention is the shortest, and the head and tail of each mobile robot can be well determined, so that the attitude angle in any direction can be obtained.

[0015] The present invention uses a plurality of fixed cameras to splice the obtained multiple images, thereby effectively solving the problem that a single camera displays a small image. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0017] Figure 1 Flowchart of the overall algorithm of the present invention.

[0018] Figure 2 Schematic diagram of the filtering algorithm of the present invention.

[0019] Figure 3 It is a schematic diagram of the world coordinate system, camera coordinate system, image coordinate system and pixel coordinate system of the present invention.

[0020] Figure 4 This is a schematic diagram of fixing multiple cameras of the present invention (taking the experimental site covered by camera 1 as an example).

[0021] Figure 5 This is a diagram of the experimental location of the present invention. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0023] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0024] like Figure 1-5 As shown, the present invention provides a method for indoor multi-mobile robot positioning with camera splicing and area matching, comprising the following steps:

[0025] Step 1: calibrate the monocular camera to obtain the camera parameters; calibrate the ground map at the same time, establish a one-to-one correspondence between the ground map area in the video image captured by the monocular camera and the real ground map area; establish a geometric model of camera imaging to determine the relationship between the position of a spatial object and its corresponding point in the image; and establish a coordinate conversion relationship based on the camera imaging model. During the conversion of the coordinate system, the conversion relationship between the world coordinate system and the pixel coordinate system is:

[0026]

[0027] Among them, [X w Y w Z w 1] T Represents the homogeneous coordinates of the corresponding point in the world coordinate system; [X cY c Z c 1] T Represents the homogeneous coordinates of any point in the camera coordinate system; [uv 1] T Represents the corresponding point in the pixel coordinate system; They are respectively called the normalized focal length of the camera on the x-axis and y-axis, in pixels, f represents the focal length of the camera, in millimeters; d x d y Respectively represent the physical size of each pixel on the x-axis and y-axis, in millimeters; represents the camera internal parameters, represents the camera external parameters, R represents the rotation matrix, and t represents the translation vector.

[0028] Step 2: Track multiple robots through positioning and tracking algorithms; fix markers on the mobile robots and use the markers as labels. The different areas of each label match the corresponding robots, and track multiple robots by tracking markers of different areas; obtain the color space of each frame of the picture through the camera, convert it from RGB to LAB and grayscale it; then perform Gaussian filtering and image corrosion operations; then filter the red color in the LAB color space, find the outline of each label and output it; finally, after traversing each outline, use the function to obtain the area of ​​each label and put it into a list for sorting, so as to achieve matching tracking.

[0029] Step 3: Perform centroid positioning, and replace the robot position coordinates with the centroid coordinates of the marker; locate the posture through two points, the two points represent the centroid coordinates and the shortest side center coordinates of the marker respectively, obtain the vertex coordinates and three side lengths of the marker and sort the three side lengths to obtain the shortest length of the three side lengths; obtain the three vertex coordinates of the marker through the recognition algorithm, thereby obtaining the length of each side and sorting them, and the shortest side is the bottom side, and determine the two vertex coordinates of the bottom side; then obtain the center coordinates of the bottom side according to the two vertex coordinates of the bottom side, and obtain the posture angle of the robot in combination with the centroid coordinates;

[0030] Step 4: To expand the test site and improve the accuracy of recognition, multiple cameras are fixed around the site to obtain the activity trajectory image of the mobile robot in the form of multiple camera images splicing, and each frame of the video image is converted into the required color space and grayed out; Gaussian filtering and image erosion are then performed to make the label outline more obvious; then the outlines of all labels are traversed and their areas are calculated; finally, the method in step 3 is combined to perform recognition and positioning.

[0031] As a preferred implementation, in the present application, the step 1 further includes the following steps:

[0032] Step 1-1: Camera calibration and distortion removal; the camera calibration and distortion removal process, i.e. the process of converting from the world coordinate system to the pixel coordinate system, makes the position of the corresponding point on the surface of the object in space correspond to the position of each point in the image one by one; the camera is calibrated using the Zhang Zhengyou calibration method to obtain the camera's internal parameters and external reference

[0033] The distortion process includes radial distortion and tangential distortion. The radial distortion is the distortion distributed along the radius of the lens. The tangential distortion is caused by the non-parallelism between the lens itself and the camera sensor plane or the image plane, and has a small impact. Therefore, only radial distortion is considered, and the distortion parameter k is obtained by combining the camera internal and external parameters and the distortion model. 1 , k 2 , then the distortion model is:

[0034]

[0035] Among them, (u, v) represents the ideal undistorted pixel coordinates, Represents the pixel coordinates after distortion, in pixels. (x,y) represents the ideal undistorted continuous image coordinates, Represents the coordinates of the distorted continuous image in millimeters; k 1 , k 2 Represents the distortion parameter.

[0036] Step 1-2: Selection of initial calibration points and expected calibration points;

[0037] Step 1-3: Inverse perspective transformation and splicing of multiple images, obtaining a spliced ​​image after inverse perspective transformation of multiple images that is consistent with the shape of the ground map. In the process of inverse perspective transformation and splicing of multiple images, after each frame of each camera is dedistorted, the inverse perspective transformation method is used to stitch multiple images of the experimental site, and then the scaling relationship between the pixel coordinates after splicing and the real ground map is used to obtain the position coordinates of each robot under the real ground map; the formula for inverse perspective transformation of the i-th camera is as follows:

[0038]

[0039] Among them, [x i y i 1] T represents the coordinates of the pixel coordinate system of the original image of the i-th camera, [X i Y i Z i ] T Represents the pixel coordinates of the corresponding point after the i-th camera transformation, M irepresents the transformation matrix of the 3×3 i-th camera; since the pixel coordinate system is a two-dimensional coordinate system, and this transformation is a two-dimensional to three-dimensional conversion, the transformed pixel point coordinates are converted to the corresponding point coordinates [X i 'Y i ′Z i ′] T :

[0040]

[0041] From the above formula, we can get four pairs of coordinates to solve M i All unknowns in the matrix, where (l i ,h i )、(0,h i )、(0,0)、(l i ,0) represent the initial calibration points of the original image of the i-th camera, (L i ,H i )、(0,H i )、(0,0)、(L i ,0) represents the expected calibration point after the inverse perspective of the i-th camera, and the unit is pixel (piexl). Then the scaling ratio of the pixel coordinates after splicing and the length of the real ground map is: The scaling ratio of the pixel coordinates after stitching to the width of the real ground map: Among them, L and H represent the length and width of the real ground map respectively;

[0042] Therefore, after the n cameras are spliced ​​together, the position coordinates of each robot on the real ground map are: Among them, X represents the horizontal coordinate of each robot under the real ground map, and Y represents the vertical coordinate of each robot under the real ground map, and the units are both meters.

[0043] In order to verify the effectiveness of the indoor multi-mobile robot positioning method with camera splicing and area matching proposed in the present invention in actual engineering applications, the positioning effect was tested from the aspect of static positioning accuracy based on a self-built experimental platform.

[0044] In the positioning accuracy experiment, 9 positions as shown in the figure were found within the scene range, and the mobile robot was placed at 9 points for positioning. The statistical algorithm output the coordinate data and compared and analyzed it with the actual experimental measured data. The average static error obtained in the experiment was within 5mm, and the positioning accuracy was high.

[0045] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages and disadvantages of the embodiments. In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments. In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for indoor multi-mobile robot positioning based on camera splicing and area matching. It is characterized in that The following steps are involved: Step 1, calibrate the monocular camera to obtain camera parameters; at the same time, calibrate the ground map to establish a one-to-one correspondence between the ground map area in the video image captured by the monocular camera and the real ground map area; Step 2: Track multiple robots through positioning and tracking algorithms; fix markers on the mobile robots and use the markers as labels. The different areas of each label match the corresponding robots. Track multiple robots by tracking markers of different areas. Step 3: Perform centroid positioning, and replace the robot position coordinates with the centroid coordinates of the marker; locate the posture through two points, the two points represent the centroid coordinates and the shortest side center coordinates of the marker respectively, obtain the vertex coordinates and three side lengths of the marker and sort the three side lengths to obtain the shortest length of the three side lengths; obtain the three vertex coordinates of the marker through the recognition algorithm, thereby obtaining the length of each side and sorting them, and the shortest side is the bottom side, and determine the two vertex coordinates of the bottom side; then obtain the center coordinates of the bottom side according to the two vertex coordinates of the bottom side, and obtain the posture angle of the robot in combination with the centroid coordinates; Step 4: By fixing multiple cameras around the site, the activity trajectory image of the mobile robot is obtained in the form of multiple camera images stitching, and each frame of the video image is converted into the required color space and grayed out; Gaussian filtering and image erosion are then performed to make the outline of the label more obvious; then the outlines of all labels are traversed and their areas are calculated; Finally, the method in step 3 is combined to perform identification and positioning.

2. According to claim 1, a method for indoor multi-mobile robot positioning with camera splicing and area matching, It is characterized in that The step 1 also includes the following steps: Step 1-1: Camera calibration and distortion removal; Step 1-2: Selection of initial calibration points and expected calibration points; Step 1-3: Inverse perspective transformation and stitching of multiple images, obtaining a stitched image after inverse perspective transformation of multiple images that is consistent with the shape of the ground map.

3. According to claim 1, a method for indoor multi-mobile robot positioning with camera splicing and area matching, It is characterized in that In order to determine the one-to-one correspondence between the position of the spatial object and its corresponding point in the image mentioned in step 1, a camera imaging geometric model is established; based on the camera imaging model, a coordinate transformation relationship is established.

4. According to claim 3, a method for indoor multi-mobile robot positioning with camera splicing and area matching, It is characterized in that The camera imaging geometric model, during the conversion of the coordinate system, the conversion relationship between the world coordinate system and the pixel coordinate system is: Among them, [X w Y w Z w 1] T Represents the homogeneous coordinates of the corresponding point in the world coordinate system; [X c Y c Z c 1] T Represents the homogeneous coordinates of any point in the camera coordinate system; [uv 1] T Represents the corresponding point in the pixel coordinate system; They are respectively called the normalized focal length of the camera on the x-axis and y-axis, in pixels, f represents the focal length of the camera, in millimeters; d x ,d y Respectively represent the physical size of each pixel on the x-axis and y-axis, in millimeters; represents the camera internal parameters, represents the camera external parameters, R represents the rotation matrix, and t represents the translation vector.

5. According to the method for indoor multi-mobile robot positioning with camera splicing and area matching described in claim 2, the camera calibration and dedistortion process, that is, the process of converting from the world coordinate system to the pixel coordinate system, makes the position of the corresponding point on the surface of the object in space correspond to the position of each point in the image one by one; the camera is calibrated using the Zhang Zhengyou calibration method to obtain the camera's internal parameters and external reference The distortion process include: Radial distortion and tangential distortion; the radial distortion is the distortion distributed along the radius of the lens; the tangential distortion is caused by the non-parallelism of the lens itself and the camera sensor plane or the image plane, and has a small impact, so only radial distortion is considered, and the distortion parameter k is obtained by combining the camera internal and external parameters and the distortion model 1 , k 2 , then the distortion model is: Among them, (u, v) represents the ideal undistorted pixel coordinates, Represents the pixel coordinates after distortion, in pixels. (x,y) represents the ideal undistorted continuous image coordinates, Represents the coordinates of the distorted continuous image in millimeters; k 1 , k 2 Represents the distortion parameter.

6. According to claim 2, a method for indoor multi-mobile robot positioning with camera splicing and area matching, It is characterized in that In the multi-image inverse perspective transformation and stitching process, after each frame of each camera is dedistorted, the multi-image stitching of the experimental site is performed by the inverse perspective transformation method, and then the scaling relationship between the pixel coordinates after stitching and the real ground map is used to obtain the position coordinates of each robot under the real ground map; the inverse perspective transformation formula of the i-th camera is as follows: Among them, [x i y i 1] T represents the coordinates of the pixel coordinate system of the original image of the i-th camera, [X i Y i Z i ] T Represents the pixel coordinates of the corresponding point after the i-th camera transformation, M i represents the transformation matrix of the 3×3 i-th camera; since the pixel coordinate system is a two-dimensional coordinate system, and this transformation is a two-dimensional to three-dimensional conversion, the transformed pixel point coordinates are converted to the corresponding point coordinates [X i ′ Y i ′ Z i ′] T : From the above formula, we can get four pairs of coordinates to solve M i All unknowns in the matrix, where (l i ,h i )、(0,h i )、(0,0)、(l i ,0) represent the initial calibration points of the original image of the i-th camera, (L i ,H i )、(0,H i )、(0,0)、(L i ,0) represents the expected calibration point after the inverse perspective of the i-th camera, and the unit is pixel (piexl); then the scaling ratio of the pixel coordinates after splicing and the length of the real ground map is: The scaling ratio of the pixel coordinates after stitching to the width of the real ground map: Among them, L and H represent the length and width of the real ground map respectively; Therefore, after the n cameras are spliced ​​together, the position coordinates of each robot on the real ground map are: Among them, X represents the horizontal coordinate of each robot under the real ground map, and Y represents the vertical coordinate of each robot under the real ground map, and the units are both meters.

7. According to claim 1, a method for indoor multi-mobile robot positioning with camera splicing and area matching, It is characterized in that In the step 2, the color space of each frame of the image obtained by the camera is converted from RGB to LAB and grayed; then Gaussian filtering and image erosion operations are performed; then the red color is filtered in the LAB color space to find the outline of each label and output it; finally, after traversing each outline, the area of ​​each label is obtained by using a function and put into a list for sorting, thereby achieving matching tracking.

Images