Image acquisition and processing method, image acquisition device and robot

Abstract

The present invention discloses an image acquisition and processing method, an image acquisition device, and a robot. The method comprises the following steps: S1: emitting line lasers of varying brightness from a line laser, and using a camera to acquire environmental images of the line lasers of varying brightness; S2: based on the environmental images of the line lasers, visual positioning and obstacle information recognition are performed, respectively; the obstacle information includes distance and orientation information. By employing a bright and dark line laser control method, the present application improves camera detection performance and reduces interference from ambient light on positioning and obstacle avoidance detection functions.

Description

Technical Field

[0001] The present invention relates to the field of intelligent robots, and in particular to an image acquisition and processing method, an image acquisition device and a robot. Background Art

[0002] Currently, there is a strong demand for indoor robot positioning and obstacle avoidance. Positioning is used to better plan the robot, and obstacle avoidance is used to prevent obstacles from hindering the robot's operation, making the robot look more intelligent. Currently, there are robots on the market that obtain environmental information through line lasers and cameras. However, when using existing line lasers for detection, the line lasers are all used to emit line lasers of the same brightness. When the robot is in a dark environment, the same brightness line laser is used for detection. The automatic exposure of the camera will increase the exposure time, resulting in serious overexposure of the laser line, making it difficult to track, and increasing the impact of ambient light on detection. The existing method uses the inter-frame difference method, which uses a one-frame-on-one-frame-off method, but this will reduce the detection frame rate by half, reducing the robot's detection efficiency. Summary of the Invention

[0003] To address the above issues, the present invention provides an image acquisition and processing method, an image acquisition device, and a robot. By employing bright and dark line laser control, the present invention improves camera detection performance and reduces the interference of ambient light on positioning and obstacle avoidance detection functions. The specific technical solutions of the present invention are as follows:

[0004] A method for image acquisition and processing includes the following steps: S1: causing a line laser to emit line lasers of different brightnesses, and acquiring an environmental image of the line lasers of different brightnesses through a camera; S2: based on the environmental image with the line lasers, respectively realizing visual positioning and obstacle information recognition; wherein the obstacle information includes distance information and orientation information.

[0005] Furthermore, in step S1, the line laser is caused to emit line lasers of different brightness, and an environmental image with line lasers of different brightness is obtained through a camera, including the following steps: the robot switches the power of the line laser during the blanking period of each frame of the camera obtaining an image, so that the brightness of the line laser in each frame of the environmental image obtained by the camera is different.

[0006] Furthermore, in step S2, visual positioning is achieved based on the environment image with a line laser, including the following steps: acquiring two frames of images, then extracting feature points in the second frame of image, and obtaining the robot posture by obtaining the IMU data and odometer integral when the two images are acquired; acquiring the epipolar line of the feature points in the second frame of image on the first frame of image based on the feature points in the second frame of image and the robot posture; searching for points on the epipolar line that match the feature values ​​of the feature points in the second frame of image to obtain the corresponding feature points in the first frame of image; based on the matching feature points and robot posture between the two frames of image, the visual positioning posture between the two frames of image is obtained by minimizing the reprojection error calculation.

[0007] Furthermore, selecting feature points from the second frame image includes the following steps: setting a container for storing feature points, and then selecting corner points from positions in the second frame image where the change in pixel grayscale value is greater than a set threshold, using the selected corner points as identified feature points in the second frame image, and then storing the identified feature points in the container.

[0008] Furthermore, the calculation of minimizing the reprojection error includes the following steps: performing projection calculation through the feature points matched between the two frames of images and the robot posture to obtain the pixel values ​​of the feature points after projection; obtaining the difference between the pixel values ​​of the feature points matched between the two frames of images and the pixel values ​​of the feature points after projection; minimizing the sum of the differences to obtain the camera posture parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning posture, and realize visual positioning.

[0009] Furthermore, in step S2, the position information of the obstacle is obtained based on the environmental image with the line laser, including the following steps: the robot obtains the internal parameters of the camera through calibration, and then obtains the line laser plane relative to the light plane of the camera through the internal parameters of the camera; the robot tracks the line laser in the obtained image, and when it is found that the line laser is not located on the ground plane, then takes the end point of one end of the line laser as the starting point and selects several points as calculation points at a specific interval from the line laser; calculates the straight line equation of the straight line passing through the center point of the camera and the calculation point, and then obtains the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line coordinates and the light plane; and obtains the position information of the obstacle according to the three-dimensional coordinates of the intersection corresponding to the several calculation points.

[0010] Furthermore, the three-dimensional coordinates of the intersection of the straight line and the light plane are obtained based on the straight line equation and the light plane, including the following steps: converting the straight line equation into a parametric equation, then substituting the parametric equation into the equation of the light plane to obtain the parameters of the parametric equation, and then substituting the parameters into the parametric equation to obtain the three-dimensional coordinates of the intersection.

[0011] Furthermore, in step S2, based on the environment image with the line laser, the distance information of the obstacle is obtained, including the following steps: obtaining in advance the ratio of the position of the line laser on the image and the distance between the robot and the obstacle; obtaining the line laser in the environment image with the line laser; and obtaining the distance between the robot and the obstacle according to the position of the line laser on the image and the ratio, that is, obtaining the distance information of the obstacle.

[0012] An image acquisition device includes a camera and a line laser module. The camera is set obliquely upward at a first preset angle or horizontally facing the front of a robot. The line laser module is located at the upper end of the camera, and the central axis of the line laser module is set obliquely downward at a second preset angle.

[0013] Furthermore, the line laser module includes a line laser and N signal control switches, the signal control switches are arranged in parallel, the line laser and the signal control switches are arranged in series, and the signal control switches are turned on or off according to the received signals to switch the power of the line laser; wherein N is a natural number greater than or equal to 2.

[0014] A robot is provided with the above-mentioned image acquisition device, and the robot executes the above-mentioned image acquisition and processing method.

[0015] Compared with existing technologies, the technical solution of this application uses a single camera to simultaneously realize the robot's positioning function and obstacle avoidance function, reducing the robot production cost while reducing the complexity of the robot system; it uses bright and dark line laser control to improve the camera detection effect and reduce the interference of ambient light on the positioning and obstacle avoidance detection functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 A flowchart of an image acquisition and processing method according to an embodiment of the present invention;

[0017] Figure 2 A schematic diagram of the connection between a straight line and a light plane according to an embodiment of the present invention;

[0018] Figure 3 This is a circuit diagram of a line laser module according to an embodiment of the present invention;

[0019] Figure 4 A schematic structural diagram of a robot according to an embodiment of the present invention. DETAILED DESCRIPTION

[0020] The technical solutions in the embodiments of the present invention will be described in detail below in conjunction with the accompanying drawings in the embodiments of the present invention. It should be understood that the specific embodiments described below are only used to explain the present invention and are not used to limit the present invention. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit this application. Based on the embodiments provided in this application, all other embodiments obtained by those of ordinary skill in the art without making creative work are within the scope of protection of this application.

[0021] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "include", "comprising", etc. used herein indicate the presence of the features, operations and / or components, but do not exclude the presence or addition of one or more other features, operations or components. All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used here should be interpreted as having meanings consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0022] like Figure 1 As shown, an image acquisition and processing method primarily involves a robot using a camera to capture images of a line laser for obstacle identification and positioning. The mobile robot also includes necessary sensors such as an IMU module and an odometer, as well as a lidar for building a cleaning map. These sensor modules operate and acquire data in the same manner as conventional sensors. The method includes the following steps: S1: The robot causes a line laser to emit laser beams of varying brightness and uses a camera to capture images of the environment containing the laser beams of varying brightness. S2: Based on the images of the environment containing the line laser, the robot performs visual positioning and obstacle identification, respectively. The obstacle information includes distance and orientation information.

[0023] In one embodiment, in step S1, the robot causes a line laser to emit line lasers of varying brightness, and uses a camera to capture an environmental image with the line lasers of varying brightness. This includes the following steps: the robot switches the power of the line laser during the blanking period of each frame captured by the camera, so that the brightness of the line laser varies in each frame of the environmental image captured by the camera. When a camera renders realistic graphics, depth information is lost due to projection transformation, often leading to graphic ambiguity. To eliminate this ambiguity, it is necessary to remove obscured, invisible lines or surfaces during rendering. This is commonly referred to as removing hidden lines and hidden surfaces, or simply blanking. The time the camera performs the blanking operation while capturing an image is called the blanking period. During the blanking period of each frame captured by the camera, the robot sends a control signal to a signal control switch, turning the signal control switch on or off. This affects the power of the line laser, causing the brightness of the line laser emitted by the line laser to change, and the camera then captures an environmental image with the line laser. The system uses a combination of bright and dark emission to emit line lasers, which allows the camera to automatically expose the object without increasing the exposure time. This allows for a simple, environmental image that can easily track the line laser. Acquiring an environmental image with the line laser involves the following steps: the robot controls the line laser to project a linear laser, then controls the camera to capture several images of the environment with the line laser. These images are then used for visual positioning and obstacle identification. Obstacle identification involves computing the obstacle's point cloud and calculating the distance between the obstacle and the robot. These two processes are combined to generate obstacle information.

[0024] As one embodiment, in step S2, the robot performs visual positioning based on an image of the environment with a line laser, including the following steps: acquiring two frames of images, then extracting feature points from the second frame, and obtaining the robot pose using IMU data and odometry integrals when the two frames are acquired, i.e., determining the robot's normal function based on the robot's movement trajectory; obtaining the epipolar line of the feature points in the second frame on the first frame based on the feature points in the second frame and the robot's pose, i.e., obtaining the relative spatial position of the two frames and the corresponding camera shooting center when the images were acquired based on the robot's movement trajectory and the acquisition time of the two frames, and then obtaining the corresponding position of the epipolar line formed by the feature points of the second frame on the first frame based on 2D-2D epipolar geometry. Searching for points on the epipolar line that match the feature values ​​of the feature points in the second frame to obtain the corresponding feature points in the first frame, i.e., matching can be performed based on the similarity of the feature points in the two frames or the distance between them, and selecting points on the epipolar line that correspond to the feature points in the second frame. Based on the matching feature points and the robot pose between the two images, the visual positioning pose between the two images is obtained by minimizing the reprojection error. Selecting feature points from the second image frame includes the following steps: setting up a container for storing feature points. A container is a special data structure. Detected corner points are typically stored, forming a KeyPoint-type container (vector). Corner points or edge points are then used as feature points to set a detection template. A function is called to detect corner points or edge points in the second image frame based on the detection template (corner points or edge points can be selected from locations in the image where pixel grayscale values ​​vary significantly). The identified corner points or edge points are then stored as feature points in the container (Fourteen Lectures on Visual SLAM: From Theory to Practice (2nd Edition) Lecture 7: Visual Odometry. Authors: Gao Xiang, Zhang Tao, et al. Publisher: Publishing House of Electronics Industry. Publication Year: August 2019). This is only one method of acquiring feature points; other methods are also possible. The calculation of minimizing the reprojection error includes the following steps: performing projection calculation through the feature points matched between the two frames of images and the robot posture to obtain the pixel values ​​of the feature points after projection; obtaining the difference between the pixel values ​​of the feature points matched between the two frames of images and the pixel values ​​of the feature points after projection; minimizing the sum of the differences to obtain the camera posture parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning posture, and realize visual positioning.In minimizing reprojection error, reprojection refers to the second projection: The first projection actually refers to the projection of a 3D point onto the image when the camera takes a picture. These images are then used to triangulate certain feature points, using geometric information (epipolar geometry) to construct triangles to determine the position of the 3D point. Finally, the calculated 3D point coordinates (note, not the real ones) and the calculated camera pose (which is also not real) are used for the second projection, or reprojection. Reprojection error refers to the difference between the projection of a real 3D point onto the image plane (i.e., a pixel on the image) and the reprojection (actually, a virtual pixel derived from the calculated value). For various reasons, the calculated value will not exactly match the actual value, meaning that this difference cannot be exactly zero. Therefore, the sum of these differences must be minimized to obtain the optimal camera pose parameters and 3D point coordinates. (Fourteen Lectures on Visual SLAM: From Theory to Practice (2nd Edition), Lecture 7: Visual Odometry. Authors: Gao Xiang, Zhang Tao, et al. Publisher: Press of Electronics Industry. Publication Year: August 2019).

[0025] As one of the embodiments, in step S2, obstacle information is obtained based on an environmental image with a line laser, including the following steps: the robot obtains the internal parameters of the camera through calibration, and then obtains the line laser plane relative to the light plane of the camera through the internal parameters of the camera; the robot tracks the line laser in the obtained image, and when it is found that the line laser is not located on the ground plane, then the end point of one end of the line laser is used as the starting point, and several points are selected as calculation points according to a specific interval; if the line segment of the line laser is divided into two intersecting line segments, the longest line segment or the line segment with the smallest angle with the horizontal line on the image is taken for calculation. The straight line equation of the straight line passing through the center point of the camera and the calculation point is calculated, and then the three-dimensional coordinates of the intersection of the straight line and the light plane are obtained based on the straight line equation and the light plane; the orientation information of the obstacle is obtained according to the three-dimensional coordinates of the intersection corresponding to the several calculation points. As Figure 2As shown in the figure, 1 is the camera's center point, 2 is the obstacle, 3 is the calculated point, 4 is the line laser, 5 is the intersection of the line and the light plane, 6 is the light plane, and 7 is the intersection of the calculated point and the intersection of the line and the light plane. When the obstacle's surface is flat, the line laser actually projects onto the obstacle's surface as a straight line, and the line laser in the image is also a straight line. The line laser is located on the light plane. In this case, the calculated point on the line laser and the intersection of the line and the light plane coincide, i.e., point 7. The 3D coordinates of the calculated point on the line laser are also the 3D coordinates of the point on the obstacle. Based on these 3D coordinates, the obstacle's position on the robot can be determined. When the obstacle's surface is curved, the line laser actually projects onto the obstacle's surface as an arc, while the line laser in the image is a straight line. In this case, the calculated point 3 selected from the line laser is not a point on the obstacle. Instead, the intersection 5 of the line and the light plane is the point on the obstacle. Therefore, point 5 is calculated from point 3 to obtain the actual 3D coordinates of the obstacle's point, thus determining the obstacle's position on the robot. The intersection of several calculated points corresponds to a point on the obstacle. Based on the three-dimensional coordinates of these points, the robot's position on the obstacle can be determined. After obtaining the three-dimensional coordinates of the intersections corresponding to the calculated points, these intersections can be filtered to improve the accuracy of the calculation results. Because the calculated points are selected at specific intervals on the same laser line segment, the three-dimensional coordinates of the intersections corresponding to the calculated points can be deleted if the three-dimensional coordinates vary significantly before the calculation is performed. Obtaining the three-dimensional coordinates of the intersection of a line and a light plane based on the line equation and the light plane includes the following steps: converting the line equation into a parametric equation, substituting the parametric equation into the light plane equation to obtain the parameters of the parametric equation, and then substituting the parameters into the parametric equation to obtain the three-dimensional coordinates of the intersection. Given the line equation L: (xa) / m=(xb) / n=(zc) / p and the light plane equation π: Ax+By+Cz+D=0, find the coordinates of the intersection of line L and plane π. Rewrite the equation of the line into parametric form: Let (xa) / m=(xb) / n=(zc) / p=t; then x=mt+a; y=nt+b; z=pt+c; substituting into the equation of plane π yields: A(mt+a)+B(nt+b)+C(pt+c)+D=0; thus, solving for t=-(Aa+Bb+Cc+D) / (Am+Bn+Cp); substituting into the parametric equation yields the coordinates (x, y, z) of the intersection point. In step S2, obstacle position information is obtained based on the environmental image with the line laser, including the following steps: obtaining in advance the ratio of the position of the line laser on the image to the distance between the robot and the obstacle; obtaining the line laser in the environmental image with the line laser; and obtaining the distance between the robot and the obstacle based on the position of the line laser on the image and the ratio, that is, obtaining the distance information of the obstacle.A line laser projects a laser line segment diagonally downward onto an obstacle. When capturing an environment with both an obstacle and a line laser, the farther the obstacle is from the robot, the lower the line laser appears in the image. This ratio can be calculated in advance through calibration and varies depending on the installation angle of the line laser. Calculation can also be performed based on the length of the line laser in the image, but this is difficult to do if the obstacle is too small. However, the length or height of the obstacle can be calculated based on the length of the line laser in the image, facilitating obstacle avoidance.

[0026] An image acquisition device, comprising a camera and a line laser module, wherein the camera is arranged obliquely upward at a first preset angle or horizontally facing the front of the robot, and the line laser module is located at the upper end of the camera, and the laser emission direction of the line laser module is obliquely downward and at a second preset angle to the horizontal line. The camera is a monochrome camera without a color filter or an infrared filter. Figure 3 As shown, the line laser module includes a line laser and N signal control switches. The signal control switches are arranged in parallel, and the line laser and the signal control switches are arranged in series. The signal control switches are turned on or off according to the received signals, so that the power of the line laser is switched; wherein N is a natural number greater than or equal to 2. In the figure, one end of the line laser is connected to the power supply terminal VCC, and the other end is connected to the signal control switches I1 and I2 respectively. The signal control switches I1 and I2 can be transistors or MOS transistors. One end of the signal transmission end of the signal control switches I1 and I2 is connected to the line laser laser, and the other end is connected to the ground end. The signal receiving end of the signal control switches I1 and I2 is used to receive external signals to turn the signal transmission end on or off. The turning on and off of the signal control switches I1 and I2 affects the power change of the line laser, so that the brightness of the line laser emitted by the line laser varies each time.

[0027] like Figure 4 A robot is shown, equipped with the aforementioned image acquisition device. In the figure, 101 represents the ground, 102 represents the robot body, 103 represents a camera, 104 represents a line laser, 105 represents a horizontal line, 106 represents the camera's central axis, 107 represents the camera's upper viewing angle, 108 represents the camera's lower viewing angle, and 109 represents the line laser projection line. The robot executes the aforementioned image acquisition and processing method via the image acquisition device.

[0028] Compared with existing technologies, the technical solution of this application uses a single camera to simultaneously realize the robot's positioning function and obstacle avoidance function, reducing the robot production cost while reducing the complexity of the robot system; it uses bright and dark line laser control to improve the camera detection effect and reduce the interference of ambient light on the positioning and obstacle avoidance detection functions.

[0029] Obviously, the above-mentioned embodiments are only some embodiments of the present invention, rather than all embodiments, and the technical solutions between the various embodiments can be combined with each other. In addition, if the terms "center", "up", "down", "left", "right", "vertical", "horizontal", "inside", "outside" and the like appear in the embodiments, the orientation or position relationship indicated is based on the orientation or position relationship shown in the accompanying drawings, which is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. If the terms "first", "second", "third" and the like appear in the embodiments, it is to facilitate the distinction between related features and cannot be understood as indicating or implying their relative importance, order or number of technical features.

[0030] Those skilled in the art will appreciate that all or part of the steps in the above-described method embodiments can be implemented using hardware associated with program instructions. These programs can be stored in a computer-readable storage medium (e.g., ROM, RAM, magnetic disk, optical disk, or other medium capable of storing program code). When executed, the program performs the steps of the above-described method embodiments.

[0031] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for acquiring and processing an image, characterized in that: The method comprises the following steps: S1: The line laser emits line lasers of different brightness, and the camera obtains environmental images of the line lasers of different brightness; S2: Based on the environmental image with line laser, visual positioning and obstacle information recognition are realized respectively; Wherein, the obstacle information includes distance information and orientation information; In step S1, the line laser is caused to emit line lasers of different brightness, and an environmental image of the line lasers of different brightness is acquired through a camera, which includes the following steps: The robot switches the power of the line laser during the blanking period when the camera acquires each frame of image, so that the brightness of the line laser in each frame of the environmental image acquired by the camera is different.

2. The image acquisition and processing method according to claim 1, characterized in that: In step S2, visual positioning is achieved based on the environment image with line laser, including the following steps: Acquire two frames of images, extract the feature points in the second frame, and obtain the robot pose by integrating the IMU data and odometer data when acquiring the two images; Obtaining epipolar lines of the feature points in the second frame image on the first frame image based on the feature points in the second frame image and the robot posture; Searching for a point on the epipolar line that matches a feature value of a feature point in the second frame image, and obtaining a corresponding feature point in the first frame image; Based on the matched feature points and robot pose between the two frames, the visual positioning pose between the two frames is obtained by minimizing the reprojection error.

3. The image acquisition and processing method according to claim 2, characterized in that: Selecting feature points from the second frame image includes the following steps: Set a container for storing feature points, and then select corner points from the second frame image, where the pixel grayscale value change is greater than the set threshold, and use the selected corner points as the identified feature points in the second frame image, and then store the identified feature points in the container.

4. The image acquisition and processing method according to claim 2, characterized in that: The calculation of minimizing the reprojection error includes the following steps: Projection calculation is performed by matching the feature points between the two frames of images and the robot posture to obtain the pixel values ​​of the projected feature points; Obtain the difference between the pixel value of the feature point matched between the two frames of image and the pixel value of the feature point after projection; The sum of the differences is minimized to obtain the camera pose parameters and the coordinates of the three-dimensional space points of the feature points, determine the visual positioning pose, and achieve visual positioning.

5. The image acquisition and processing method according to claim 1, characterized in that: In step S2, the position information of the obstacle is obtained based on the environment image with the line laser, including the following steps: The robot obtains the intrinsic parameters of the camera through calibration, and then uses the intrinsic parameters of the camera to obtain the line laser plane relative to the light plane of the camera; The robot tracks the line laser in the acquired image. If it finds that the line laser is not on the ground plane, it selects several points on the line laser at a specific interval as calculation points, starting from the end point of one end of the line laser. Calculating a straight line equation of a straight line passing through the center point of the camera and the calculation point, and then obtaining the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line equation and the light plane; The orientation information of the obstacle is obtained according to the three-dimensional coordinates of the intersection points corresponding to several calculation points.

6. The image acquisition and processing method according to claim 5, characterized in that: Obtaining the three-dimensional coordinates of the intersection of the straight line and the light plane based on the straight line equation and the light plane includes the following steps: The straight line equation is converted into a parametric equation, and then the parametric equation is substituted into the equation of the light plane to obtain the parameters of the parametric equation, and then the parameters are substituted into the parametric equation to obtain the three-dimensional coordinates of the intersection point.

7. The image acquisition and processing method according to claim 1, characterized in that: In step S2, obtaining distance information of obstacles based on the environment image with the line laser includes the following steps: Obtain in advance the ratio of the position of the line laser on the image to the distance between the robot and the obstacle; Acquire a line laser in an environment image having a line laser; The distance between the robot and the obstacle is obtained according to the position of the line laser on the image and the ratio, thereby obtaining the distance information of the obstacle.

8. An image acquisition device, characterized in that: Used to execute the image acquisition and processing method according to any one of claims 1 to 7, the device includes a camera and a line laser module, the camera is set obliquely upward at a first preset angle or horizontally facing directly in front of the robot, the line laser module is located at the upper end of the camera, and the central axis of the line laser module is set obliquely downward at a second preset angle.

9. The image acquisition device according to claim 8, characterized in that The line laser module includes a line laser and N signal control switches, the signal control switches are arranged in parallel, the line laser and the signal control switches are arranged in series, and the signal control switches are turned on or off according to the received signals to switch the power of the line laser; wherein N is a natural number greater than or equal to 2.

10. A robot, characterized in that: The robot is provided with the image acquisition device according to any one of claims 8 to 9, and the robot executes the image acquisition and processing method according to any one of claims 1 to 7.

Images