Obstacle identification and synchronous positioning method, image acquisition device and robot

Abstract

The present invention discloses a method for obstacle recognition and simultaneous positioning, an image acquisition device, and a robot. The method comprises the following steps: S1: the robot activates a line laser and a camera to acquire an image of an environment containing the line laser; S2: based on the image of the environment containing the line laser, visual positioning and obstacle information recognition are performed, respectively; the obstacle information includes distance and orientation information. This application utilizes a single camera to achieve both obstacle avoidance and positioning functions, reducing system complexity and production costs.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robots, and in particular to a method for obstacle recognition and synchronous positioning, an image acquisition device and a robot. Background Art

[0002] Currently, there's a strong demand for both positioning and obstacle avoidance in indoor robots. Positioning allows for better robot planning, while obstacle avoidance prevents obstacles from hindering the robot's operation, making the robot appear more intelligent. Currently, products on the market implement these two sensors separately. Separate setups not only require more data to be acquired and processed, but also increase the production cost of the robot. Summary of the Invention

[0003] To address the above issues, the present invention provides a method for obstacle recognition and simultaneous positioning, an image acquisition device, and a robot. This application utilizes a single camera to achieve both obstacle avoidance and positioning, reducing system complexity and production costs. The specific technical solutions of the present invention are as follows:

[0004] A method for obstacle recognition and synchronous positioning, the method comprising the following steps: S1: a robot operates a line laser and a camera to obtain an environment image with the line laser; S2: based on the environment image with the line laser, visual positioning and obstacle information recognition are respectively realized; wherein the obstacle information includes distance information and orientation information.

[0005] Furthermore, in step S1, obtaining an environmental image with a line laser includes the following steps: the robot controls the line laser to work, so that the line laser projects a linear laser, and then controls the camera to work, obtaining several environmental images with the line laser.

[0006] Furthermore, in step S2, visual positioning is achieved based on the environment image with a line laser, including the following steps: acquiring two frames of images, then extracting feature points in the second frame of image, and obtaining the robot posture by obtaining the IMU data and odometer integral when the two images are acquired; acquiring the epipolar line of the feature points in the second frame of image on the first frame of image based on the feature points in the second frame of image and the robot posture; searching for points on the epipolar line that match the feature values ​​of the feature points in the second frame of image to obtain the corresponding feature points in the first frame of image; based on the matching feature points and robot posture between the two frames of image, the visual positioning posture between the two frames of image is obtained by minimizing the reprojection error calculation.

[0007] Furthermore, selecting feature points from the second frame image includes the following steps: setting a container for storing feature points, and then selecting corner points from positions in the second frame image where the change in pixel grayscale value is greater than a set threshold, using the selected corner points as identified feature points in the second frame image, and then storing the identified feature points in the container.

[0008] Furthermore, the calculation of minimizing the reprojection error includes the following steps: performing projection calculation through the feature points matched between the two frames of images and the robot posture to obtain the pixel values ​​of the feature points after projection; obtaining the difference between the pixel values ​​of the feature points matched between the two frames of images and the pixel values ​​of the feature points after projection; minimizing the sum of the differences to obtain the camera posture parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning posture, and realize visual positioning.

[0009] Furthermore, in step S2, the position information of the obstacle is obtained based on the environmental image with the line laser, including the following steps: the robot obtains the internal parameters of the camera through calibration, and then obtains the line laser plane relative to the light plane of the camera through the internal parameters of the camera; the robot tracks the line laser in the obtained image, and when it is found that the line laser is not located on the ground plane, then takes the end point of one end of the line laser as the starting point and selects several points as calculation points at a specific interval from the line laser; calculates the straight line equation of the straight line passing through the center point of the camera and the calculation point, and then obtains the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line coordinates and the light plane; and obtains the position information of the obstacle according to the three-dimensional coordinates of the intersection corresponding to the several calculation points.

[0010] Furthermore, the three-dimensional coordinates of the intersection of the straight line and the light plane are obtained based on the straight line equation and the light plane, including the following steps: converting the straight line equation into a parametric equation, then substituting the parametric equation into the equation of the light plane to obtain the parameters of the parametric equation, and then substituting the parameters into the parametric equation to obtain the three-dimensional coordinates of the intersection.

[0011] Furthermore, in step S2, based on the environment image with the line laser, the distance information of the obstacle is obtained, including the following steps: obtaining in advance the ratio of the position of the line laser on the image and the distance between the robot and the obstacle; obtaining the line laser in the environment image with the line laser; and obtaining the distance between the robot and the obstacle according to the position of the line laser on the image and the ratio, that is, obtaining the distance information of the obstacle.

[0012] An image acquisition device includes a camera and a line laser. The camera is set obliquely upward at a first preset angle or horizontally toward the front of a robot. The line laser is located at the upper end of the camera, and the central axis of the line laser is set obliquely downward at a second preset angle.

[0013] Furthermore, the camera is a monochrome camera or a color camera.

[0014] A robot is provided with the above-mentioned image acquisition device, and the robot executes the above-mentioned obstacle recognition and synchronous positioning method.

[0015] Compared with the existing technology, the technical solution of the present application uses a camera and a line laser to obtain an environmental image with a line laser, and then performs robot positioning and obstacle recognition based on the obtained environmental image, reducing system complexity and production costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flow chart of a method for obstacle identification and synchronous positioning according to an embodiment of the present invention;

[0017] Figure 2 A schematic structural diagram of a robot according to an embodiment of the present invention;

[0018] Figure 3 A schematic diagram of a straight line and a light plane according to an embodiment of the present invention. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described below are only used to explain the present invention and are not used to limit the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0020] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprising", etc. used herein indicate the presence of the features, operations and / or components, but do not exclude the presence or addition of one or more other features, operations or components. All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having meanings consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0021] like Figure 1As shown, a method for obstacle identification and simultaneous positioning is described. A robot uses a camera to capture images of a line laser for obstacle identification and positioning. The mobile robot also includes necessary sensors such as an IMU module and an odometer, as well as a lidar for building a clean map. The operating methods and data acquired by these sensor modules are similar to conventional methods. The method includes the following steps: S1: The robot activates a line laser and a camera to capture an image of the environment with the line laser. S2: Based on the image of the environment with the line laser, visual positioning and obstacle information recognition are performed, respectively. The obstacle information includes distance and orientation information. In step S1, acquiring the image of the environment with the line laser includes the following steps: the robot controls the line laser to project a linear laser, then controls the camera to capture several images of the environment with the line laser. These images are then used for visual positioning and obstacle identification. Obstacle identification involves computing point clouds of the obstacle and calculating the distance between the obstacle and the robot. These two methods are combined to generate obstacle information.

[0022] As one embodiment, in step S2, the robot realizes visual positioning based on an environmental image with a line laser, including the following steps: acquiring two frames of images, then extracting feature points in the second frame of image, and obtaining the robot posture by IMU data and odometer integral when acquiring the two frames of image, that is, determining the robot's normal function based on the robot's movement trajectory; obtaining the epipolar line of the feature points in the second frame of image on the first frame of image based on the feature points in the second frame of image and the robot posture, that is, according to the robot's movement trajectory and the acquisition time of the two frames of image, obtaining the relative position of the two frames of image in space and the corresponding shooting center of the camera when the image was acquired, and then obtaining the corresponding position of the epipolar line formed by the feature points of the second frame of image on the first frame of image based on 2D-2D epipolar geometry. Search for points on the epipolar line that match the feature values ​​of the feature points in the second frame of image to obtain the corresponding feature points in the first frame of image, that is, matching can be performed based on the similarity of the feature points in the two frames of image or the distance between the two, and selecting points on the epipolar line that correspond to the feature points in the second frame of image. Based on the matched feature points and robot pose between the two image frames, the visual localization pose between the two frames is calculated by minimizing the reprojection error. Feature point extraction from the second image includes the following steps: Setting up a container for storing feature points. A container is a special data structure. Detected corner points are typically stored, forming a KeyPoint-type container (vector). Corner or edge points are then used as feature points to set a detection template. A function is called to detect corner or edge points in the second image based on the detection template (corner or edge points can be selected from locations with significant pixel grayscale value changes in the image). The identified corner or edge points are then stored as feature points in the container (Fourteen Lectures on Visual SLAM: From Theory to Practice (2nd Edition) Lecture 7: Visual Odometry. Authors: Gao Xiang, Zhang Tao, et al. Publisher: Publishing House of Electronics Industry. Publication Year: August 2019). This is only one method of acquiring feature points; other methods are also possible. The calculation of minimizing the reprojection error includes the following steps: performing projection calculation through the feature points matched between the two frames of images and the robot posture to obtain the pixel values ​​of the feature points after projection; obtaining the difference between the pixel values ​​of the feature points matched between the two frames of images and the pixel values ​​of the feature points after projection; minimizing the sum of the differences to obtain the camera posture parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning posture, and realize visual positioning.In minimizing reprojection error, reprojection refers to the second projection: The first projection actually refers to the projection of a 3D point onto the image when the camera takes a picture. These images are then used to triangulate certain feature points, using geometric information (epipolar geometry) to construct triangles to determine the position of the 3D point. Finally, the calculated 3D point coordinates (note, not the real ones) and the calculated camera pose (which is also not real) are used for the second projection, or reprojection. Reprojection error refers to the difference between the projection of a real 3D point onto the image plane (i.e., a pixel on the image) and the reprojection (actually, a virtual pixel derived from the calculated value). For various reasons, the calculated value will not exactly match the actual value, meaning that this difference cannot be exactly zero. Therefore, the sum of these differences must be minimized to obtain the optimal camera pose parameters and 3D point coordinates. (Fourteen Lectures on Visual SLAM: From Theory to Practice (2nd Edition), Lecture 7: Visual Odometry. Authors: Gao Xiang, Zhang Tao, et al. Publisher: Publishing House of Electronics Industry. Publication Year: August 2019).

[0023] As one of the embodiments, in step S2, obstacle information is obtained based on an environmental image with a line laser, including the following steps: the robot obtains the internal parameters of the camera through calibration, and then obtains the line laser plane relative to the light plane of the camera through the internal parameters of the camera; the robot tracks the line laser in the obtained image, and when it is found that the line laser is not located on the ground plane, then the end point of one end of the line laser is used as the starting point, and several points are selected as calculation points according to a specific interval; if the line segment of the line laser is divided into two intersecting line segments, the longest line segment or the line segment with the smallest angle with the horizontal line on the image is taken for calculation. The straight line equation of the straight line passing through the center point of the camera and the calculation point is calculated, and then the three-dimensional coordinates of the intersection of the straight line and the light plane are obtained based on the straight line equation and the light plane; the orientation information of the obstacle is obtained according to the three-dimensional coordinates of the intersection corresponding to the several calculation points. As Figure 3As shown in the figure, 1 is the camera's center point, 2 is the obstacle, 3 is the calculated point, 4 is the line laser, 5 is the intersection of the line and the light plane, 6 is the light plane, and 7 is the intersection of the calculated point and the intersection of the line and the light plane. When the obstacle's surface is flat, the line laser actually projects onto the obstacle's surface as a straight line, and the line laser in the image is also a straight line. The line laser is located on the light plane. In this case, the calculated point on the line laser and the intersection of the line and the light plane coincide, i.e., point 7. The 3D coordinates of the calculated point on the line laser are also the 3D coordinates of the point on the obstacle. Based on these 3D coordinates, the obstacle's position on the robot can be determined. When the obstacle's surface is curved, the line laser actually projects onto the obstacle's surface as an arc, while the line laser in the image is a straight line. In this case, the calculated point 3 selected from the line laser is not a point on the obstacle. Instead, the intersection 5 of the line and the light plane is the point on the obstacle. Therefore, point 5 is calculated from point 3 to obtain the actual 3D coordinates of the obstacle's point, thus determining the obstacle's position on the robot. The intersection of several calculated points corresponds to a point on the obstacle. Based on the three-dimensional coordinates of these points, the robot's position on the obstacle can be determined. After obtaining the three-dimensional coordinates of the intersections corresponding to the calculated points, these intersections can be filtered to improve the accuracy of the calculation results. Because the calculated points are selected at specific intervals on the same laser line segment, the three-dimensional coordinates of the intersections corresponding to the calculated points can be deleted if the three-dimensional coordinates vary significantly before the calculation is performed. Obtaining the three-dimensional coordinates of the intersection of a line and a light plane based on the line equation and the light plane includes the following steps: converting the line equation into a parametric equation, substituting the parametric equation into the light plane equation to obtain the parameters of the parametric equation, and then substituting the parameters into the parametric equation to obtain the three-dimensional coordinates of the intersection. Given the line equation L: (xa) / m=(xb) / n=(zc) / p and the light plane equation π: Ax+By+Cz+D=0, find the coordinates of the intersection of line L and plane π. Rewrite the equation of the line into parametric form: Let (xa) / m=(xb) / n=(zc) / p=t; then x=mt+a; y=nt+b; z=pt+c; substituting into the equation of plane π yields: A(mt+a)+B(nt+b)+C(pt+c)+D=0; thus, solving for t=-(Aa+Bb+Cc+D) / (Am+Bn+Cp); substituting into the parametric equation yields the coordinates (x, y, z) of the intersection point. In step S2, obstacle position information is obtained based on the environmental image with the line laser, including the following steps: obtaining in advance the ratio of the position of the line laser on the image to the distance between the robot and the obstacle; obtaining the line laser in the environmental image with the line laser; and obtaining the distance between the robot and the obstacle based on the position of the line laser on the image and the ratio, that is, obtaining the distance information of the obstacle.A line laser projects a laser line segment diagonally downward onto an obstacle. When capturing an environment with both an obstacle and a line laser, the farther the obstacle is from the robot, the lower the line laser appears in the image. This ratio can be calculated in advance through calibration and varies depending on the installation angle of the line laser. Calculation can also be performed based on the length of the line laser in the image, but this is difficult to do if the obstacle is too small. However, the length or height of the obstacle can be calculated based on the length of the line laser in the image, facilitating obstacle avoidance.

[0024] An image acquisition device includes a camera and a line laser. The camera is positioned obliquely upward at a first preset angle or horizontally facing directly in front of a robot. The line laser is located above the camera and emits light obliquely downward at a second preset angle relative to the horizontal. The camera is a monochrome camera without a color filter or an infrared filter.

[0025] like Figure 2 A robot is shown, equipped with the aforementioned image acquisition device. In the figure, 101 represents the ground, 102 represents the robot body, 103 represents a camera, 104 represents a line laser, 105 represents a horizontal line, 106 represents the camera's central axis, 107 represents the camera's upper viewing angle, 108 represents the camera's lower viewing angle, and 109 represents the line laser projection line. The robot uses the image acquisition device to perform the aforementioned obstacle recognition and synchronous positioning method.

[0026] Compared with the existing technology, the technical solution of the present application uses a camera and a line laser to obtain an environmental image with a line laser, and then performs robot positioning and obstacle recognition based on the obtained environmental image, reducing system complexity and production costs.

[0027] Obviously, the above-mentioned embodiments are only some embodiments of the present invention, rather than all embodiments, and the technical solutions between the various embodiments can be combined with each other. In addition, if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like appear in the embodiments, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. If the terms "first", "second", "third" and the like appear in the embodiments, it is to facilitate the distinction between related features and cannot be understood as indicating or implying their relative importance, order or number of technical features.

[0028] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. These programs can be stored in a computer-readable storage medium (e.g., ROM, RAM, magnetic disk, optical disk, or other medium capable of storing program code). When executed, the program performs the steps of the above-described method embodiments.

[0029] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for obstacle identification and synchronous positioning, characterized in that: The method comprises the following steps: S1: The robot operates the line laser and the camera to obtain an image of the environment with the line laser; S2: Based on the environmental image with line laser, visual positioning and obstacle information recognition are realized respectively; Wherein, the obstacle information includes distance information and orientation information; In step S2, visual positioning is achieved based on the environment image with line laser, including the following steps: Acquire two frames of images, extract the feature points in the second frame, and obtain the robot pose by integrating the IMU data and odometer data when acquiring the two images; Obtaining epipolar lines of the feature points in the second frame image on the first frame image based on the feature points in the second frame image and the robot posture; Searching for a point on the epipolar line that matches a feature value of a feature point in the second frame image, and obtaining a corresponding feature point in the first frame image; Based on the matched feature points and robot pose between the two frames, the visual positioning pose between the two frames is obtained by minimizing the reprojection error.

2. The method for obstacle identification and synchronous positioning according to claim 1, characterized in that: In step S1, an environmental image with a line laser is obtained, including the following steps: The robot controls the line laser to project a linear laser, and then controls the camera to obtain a number of environment images with the line laser.

3. The method for obstacle identification and synchronous positioning according to claim 1, characterized in that: Selecting feature points from the second frame image includes the following steps: Set a container for storing feature points, and then select corner points from the second frame image, where the pixel grayscale value change is greater than the set threshold, and use the selected corner points as the identified feature points in the second frame image, and then store the identified feature points in the container.

4. The method for obstacle identification and synchronous positioning according to claim 1, characterized in that: The calculation of minimizing the reprojection error includes the following steps: Projection calculation is performed by matching the feature points between the two frames of images and the robot posture to obtain the pixel values ​​of the projected feature points; Obtain the difference between the pixel value of the feature point matched between the two frames of image and the pixel value of the feature point after projection; The sum of the differences is minimized to obtain the camera pose parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning pose, and achieve visual positioning.

5. The method for obstacle identification and synchronous positioning according to claim 1, characterized in that: In step S2, the position information of the obstacle is obtained based on the environment image with the line laser, including the following steps: The robot obtains the intrinsic parameters of the camera through calibration, and then uses the intrinsic parameters of the camera to obtain the line laser plane relative to the light plane of the camera; The robot tracks the line laser in the acquired image. If it finds that the line laser is not on the ground plane, it selects several points on the line laser at a specific interval as calculation points, starting from the end point of one end of the line laser. Calculating a straight line equation of a straight line passing through the center point of the camera and the calculation point, and then obtaining the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line equation and the light plane; The orientation information of the obstacle is obtained according to the three-dimensional coordinates of the intersection points corresponding to several calculation points.

6. The method for obstacle identification and synchronous positioning according to claim 5, characterized in that: Obtaining the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line equation and the light plane includes the following steps: The straight line equation is converted into a parametric equation, and then the parametric equation is substituted into the equation of the light plane to obtain the parameters of the parametric equation, and then the parameters are substituted into the parametric equation to obtain the three-dimensional coordinates of the intersection point.

7. The method for obstacle identification and synchronous positioning according to claim 1, characterized in that: In step S2, obtaining distance information of obstacles based on the environment image with the line laser includes the following steps: Obtain in advance the ratio of the position of the line laser on the image to the distance between the robot and the obstacle; Acquire a line laser in an environment image having a line laser; The distance between the robot and the obstacle is obtained according to the position of the line laser on the image and the ratio, thereby obtaining the distance information of the obstacle.

8. A robot, characterized in that: The robot is provided with an image acquisition device, and the robot performs the method for obstacle recognition and synchronous positioning according to any one of claims 1 to 7; The image acquisition device includes a camera and a line laser. The camera is set obliquely upward at a first preset angle or horizontally toward the front of the robot. The line laser is located at the upper end of the camera, and the central axis of the line laser is set obliquely downward at a second preset angle.

9. The robot according to claim 8, characterized in that The camera is a monochrome camera or a color camera.

Images