Laser and smudginess joint calibration method and device and smudginess detection equipment

Abstract

The invention discloses a laser and smudginess joint calibration method and device and smudginess detection equipment, and the method comprises the steps: determining the smudginess resolution, arranging a grid corresponding to the smudginess resolution at a ground position, placing the smudginess detection equipment with a single camera or a plurality of cameras on the ground provided with the grid, and carrying out the calibration of the smudginess detection equipment with the single camera or the plurality of cameras. Determining grid positions of the one or more smudginess points in the acquired image, and constructing a first corresponding relation between the one or more smudginess points and grids; acquiring a light plane formed by laser rays emitted by a single line laser or a light plane formed by crossed laser rays emitted by at least two line lasers in a plurality of line lasers, projecting the light plane to the grid map, determining the grid position of the light plane projection, and constructing a second corresponding relationship between the light plane projection and the grid; a map is drawn according to the first corresponding relation and the second corresponding relation, and laser and smudginess joint calibration is achieved. Through a map drawn after joint calibration, obstacle information and smudginess information can be determined at the same time, and effective fusion of a robot obstacle avoidance function and a smudginess detection function is realized.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence, and more specifically, to a laser-based combined calibration method, apparatus, and dirt detection equipment. Background Technology

[0002] With advancements in navigation and obstacle avoidance technologies, robots have achieved remarkable success in addressing issues such as collisions with walls, getting stuck, and falls. Navigation and obstacle avoidance capabilities are considered core standards for measuring a robot's intelligence level. Currently, mainstream navigation and obstacle avoidance technologies primarily include binocular vision, LiDAR, and line structured light.

[0003] Line structured light obstacle avoidance detects obstacles by relying on changes in the distance or depth between the emitted and reflected light beams. The emitted light beams are typically in the form of a three-dimensional dot matrix or stripe array. When the light beam hits an object, the time delay or intensity of the reflected light increases. These changes in light intensity and phase can be processed and analyzed using computer algorithms to determine the position and distance of the object or obstacle.

[0004] Dirt detection technology is a technique used to detect dirt and pollutants on object surfaces or in the environment. It is widely used in manufacturing, service industries, healthcare, and environmental protection, aiming to protect public health and maintain environmental cleanliness.

[0005] With ever-increasing user demands, the integration of dirt detection capabilities into robots, in addition to basic functions such as obstacle avoidance, is becoming an inevitable development trend. However, a technical solution for the joint calibration of lasers and dirt detection is currently lacking. Summary of the Invention

[0006] The main objective of this invention is to disclose a method, apparatus, and dirt detection device for joint calibration of laser and dirt, so as to at least solve the problem that there is currently a lack of relevant technical solutions for how to achieve joint calibration of laser and dirt in related technologies.

[0007] According to one aspect of the present invention, a method for joint calibration of laser and contamination is provided.

[0008] The laser and dirt joint calibration method according to the present invention includes: determining the dirt resolution; setting a grid corresponding to the dirt resolution on the ground; placing a dirt detection device with a single camera or multiple cameras on the ground with the grid; determining the grid position of one or more dirt points in the acquired image; and constructing a first correspondence between the one or more dirt points and the grid; acquiring a light plane formed by a laser line emitted by a single line laser or a light plane formed by intersecting laser lines emitted by at least two line lasers; projecting it onto the grid map; determining the grid position of the light plane projection; and constructing a second correspondence between the light plane projection and the grid; and drawing a map based on the first and second correspondences to achieve laser and dirt joint calibration.

[0009] According to another aspect of the present invention, a laser and dirt combined calibration device is provided.

[0010] The laser and dirt joint calibration device according to the present invention includes: a first construction module, used to determine the dirt resolution, set a grid map corresponding to the dirt resolution on the ground, place a single camera or one of multiple cameras on the ground with the grid map, determine the grid position of the one or more dirt points in the acquired image, and construct a first correspondence between the one or more dirt points and the grid; a second construction module, used to project a light plane formed by a laser line emitted by a single line laser or a light plane formed by intersecting laser lines emitted by at least two line lasers into the grid map, determine the grid position of the light plane projection, and construct a second correspondence between the light plane projection and the grid; and a joint calibration module, used to draw a map according to the first correspondence and the second correspondence to achieve joint calibration of laser and dirt.

[0011] According to another aspect of the present invention, a dirt detection device is provided.

[0012] The dirt detection device according to the present invention comprises: one or more line lasers for emitting laser lines to achieve obstacle detection, wherein, when the dirt detection device comprises multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect; one or more cameras with supplementary lighting modules, and / or one or more cameras without supplementary lighting modules and one or more supplementary lighting devices, wherein the one or more cameras with supplementary lighting modules and / or the one or more cameras without supplementary lighting modules are used to time-division multiplex the acquired image information when the processor performs obstacle detection and dirt detection, and the supplementary lighting modules and / or the supplementary lighting devices are used to provide supplementary lighting to the cameras acquiring image information; a processor is used to perform obstacle detection and dirt detection based on the time-division multiplexing of the acquired image information from the one or more cameras with supplementary lighting modules and / or the one or more cameras without supplementary lighting modules, wherein the dirt points detected by the dirt detection and the light plane formed by the laser lines emitted by the one or more line lasers are used to implement the laser and dirt joint calibration method as described in any one of claims 1 to 6.

[0013] According to the present invention, a method for joint calibration of laser and dirt is provided. This method determines the grid position of one or more dirt points in an acquired image, establishes a first correspondence between the dirt points and the grid, determines the grid position of the light plane projection, establishes a second correspondence between the light plane projection and the grid, and draws a map based on the first and second correspondences, thereby achieving joint calibration of laser and dirt. The map drawn after joint calibration can simultaneously determine obstacle information and dirt information, effectively integrating robot obstacle avoidance and dirt detection functions. Attached Figure Description

[0014] Figure 1 This is a flowchart of the laser and contamination joint calibration method according to an embodiment of the present invention;

[0015] Figure 2 This is a schematic diagram illustrating the correspondence between dirt spots and grids according to a preferred embodiment of the present invention;

[0016] Figure 3 This is a structural block diagram of the laser and contamination joint calibration device according to an embodiment of the present invention;

[0017] Figure 4 This is a structural block diagram of a laser and contamination combined calibration device according to a preferred embodiment of the present invention;

[0018] Figure 5 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention. Detailed Implementation

[0019] The specific implementation of the present invention will now be described in detail with reference to the accompanying drawings.

[0020] According to an embodiment of the present invention, a method for joint calibration of laser and dirt is provided.

[0021] Figure 1 This is a flowchart of a laser and contamination joint calibration method according to an embodiment of the present invention. Figure 1 As shown, the laser and dirt joint calibration method includes:

[0022] Step S101: Determine the dirt resolution, set the grid corresponding to the dirt resolution on the ground, place the dirt detection device with a single camera or multiple cameras on the ground with the grid, determine the grid position of one or more dirt points in the acquired image, and establish the first correspondence between the one or more dirt points and the grid.

[0023] Step S103: Project the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers into the above grid map, determine the grid position of the light plane projection, and construct a second correspondence between the light plane projection and the grid.

[0024] Step S105: Draw a map based on the first correspondence and the second correspondence mentioned above to achieve joint calibration of laser and dirt.

[0025] In related technologies, there is currently a lack of technical solutions for the joint calibration of laser and contamination. Figure 1 The provided laser and dirt joint calibration method determines the grid position of one or more dirt points in the acquired image, establishes a first correspondence between the dirt points and the grid, determines the grid position of the light plane projection, establishes a second correspondence between the light plane projection and the grid, and draws a map based on the first and second correspondences, thus achieving joint calibration of laser and dirt. The map drawn after joint calibration can simultaneously determine obstacle information and dirt information, effectively integrating the robot's obstacle avoidance and dirt detection functions.

[0026] In step S101, various methods from existing technologies can be used to detect dirt spots. For example, cameras and image processing techniques (edge ​​detection algorithms, etc.) can be used to detect dirt on the surface of an object. Alternatively, spectral techniques (including infrared spectroscopy, ultraviolet-visible spectroscopy, etc.) can be used to determine the degree of dirt contamination based on the brightness value of the dirt in the image. Furthermore, deep learning technology can be used to pre-train a model on a large dataset, and the trained target detection model can then be used to achieve dirt detection.

[0027] Preferably, before projecting the light plane formed by the laser line emitted by a single line laser or the light plane formed by intersecting laser lines emitted by at least two line lasers from multiple line lasers onto the grid image in step S103, the following processing may be included: setting up marker plates at different locations, and capturing multiple images of the marker plates using a single camera or multiple cameras, wherein the multiple images include: the projection of the laser line emitted by a single line laser onto the marker plate, or the projection of intersecting laser lines emitted by at least two line lasers from multiple line lasers onto the marker plate; extracting the projection center lines of the laser lines onto the marker plates from the multiple images, wherein when the extracted projection center lines of intersecting laser lines are obtained, according to the above... The spatial relationship between intersecting laser lines is used to determine the optical plane corresponding to the projection center line of the extracted laser lines. Based on each extracted projection center line, discrete points are calculated on the intersection line of the optical plane and the marking plate plane. These discrete points are fitted to obtain equations for multiple intersection lines. Based on these equations, the normal vectors of all optical planes are obtained, and the optimal solution for the normal vector is determined from all the normal vectors. In the equations of the multiple intersection lines, lines whose perpendicularity deviation from the optimal solution for the normal vector exceeds a predetermined deviation threshold are removed, resulting in the direction vectors corresponding to the remaining lines. Equations are constructed using the direction vectors corresponding to the remaining lines to calculate the normal vectors of the optical plane formed by the laser lines emitted by the single line laser or the optical plane formed by the intersecting laser lines.

[0028] In a preferred embodiment, multiple images can be captured at a predetermined distance (e.g., 20 cm) from one or more cameras, and a line laser can be activated to emit a line laser, or multiple line lasers can be activated simultaneously to emit line lasers. When multiple line lasers are activated to emit line lasers, at least two of the multiple line lasers emit laser lines that intersect. For example, two line lasers located on both sides can be activated simultaneously to emit line lasers, and the line lasers emitted by these two line lasers intersect.

[0029] The cross-line laser solution uses at least two intersecting laser lines and the robot's movement to create a 3D map, accurately measuring height and distance, thus providing more precise 3D obstacle avoidance information. This cross-line laser obstacle avoidance technology offers advantages such as millimeter-level high precision, low cost, high stability, and strong resistance to ambient light interference.

[0030] It should be noted that when extracting the projection center line of the intersecting laser lines from the above multiple images, the light plane corresponding to the projection center line of the extracted laser line can be determined according to the spatial positional relationship of the intersecting laser lines. That is, since the intersection line of the two light planes is located in front of the marking plate and parallel to the imaging plane, according to the geometric spatial relationship, it can be known that the intersection lines of these two light planes and the calibration plate are on both sides of the center of the image and have no intersection.

[0031] Preferably, extracting the projection center line of the laser line on the marking plate from the multiple images may further include: normalizing the pixel values ​​of each of the multiple images; performing contrast enhancement and filtering on the image; and extracting the projection center line of the laser line on the marking plate from the processed image.

[0032] Image pixel normalization is a preprocessing method for image data, aiming to adjust the pixel values ​​in the image to a reasonable range. This processing is crucial for subsequent image processing tasks because it ensures that all pixel values ​​are on a uniform scale, thus avoiding the impact of excessive differences in pixel value ranges on image processing results. This application employs image pixel normalization to reduce the influence of lighting conditions. Next, gamma transform can be used to enhance image contrast. Gamma transform is a non-linear transformation method used in image processing, primarily for adjusting image contrast. Through gamma transform, the grayscale values ​​of darker areas in the image can be enhanced, while the grayscale values ​​of overly bright areas can be reduced, thereby improving the overall image contrast. This application uses gamma transform to enhance image contrast, making the laser line stand out more in the image. Next, Gaussian filtering is applied to make the image conform to a Gaussian distribution, thereby reducing noise interference. Finally, the Steger method can be used to extract the center line of the laser line's projection on the calibration board, ensuring that the extracted line is accurate and continuous. The aforementioned Steger algorithm is an image edge detection algorithm used to extract center lines or edge information in an image. Its theoretical assumption is that the brightness of the fringes follows a Gaussian distribution, meaning it is brighter at the center and gradually darkens towards the sides. By calculating the Hessian matrix at each point, the normal direction of the fringes can be located, thus enabling the extraction of the center line of the light stripes.

[0033] Preferably, based on each extracted projection center line, the discrete points on the intersection line of the light plane and the marking plate plane are calculated, including: the intrinsic parameters of the calibration camera (e.g., focal length f, camera principal point coordinates (c...)). x c yThe extrinsic parameters of the aforementioned marker plate (including rotation matrix R and translation variable t, etc.) and the distortion coefficients of the camera (including radial distortion coefficient and tangential distortion coefficient) are used to perform distortion correction on the extracted projection centerline. The intrinsic parameters of the camera are used to transform the distortion-corrected projection centerline into a ray equation in the camera coordinate system. Based on the ray equation and the extrinsic parameters of the aforementioned marker plate, the discrete points on the intersection line of the light plane and the marker plate plane are calculated.

[0034] Preferably, obtaining the normal vectors of all light planes based on the equations of the multiple intersection lines, and determining the optimal solution for the normal vector from the normal vectors of all light planes, may further include: randomly selecting two equations from the equations of the multiple intersection lines, obtaining the light planes corresponding to the equations of the two intersection lines, calculating the normal vector of the light plane, performing a dot product calculation on the normal vector and the multiple intersection lines respectively to obtain multiple dot product values, taking the absolute value of the multiple dot product values ​​and summing them to obtain the accumulated dot product value corresponding to the normal vector, repeating this step until the accumulated dot product value of the normal vectors of all light planes is obtained, and taking the normal vector corresponding to the smallest accumulated dot product value among the obtained accumulated dot product values ​​as the optimal solution for the normal vector.

[0035] Preferably, the calculation of the normal vector of the optical plane formed by the laser line emitted by the single line laser or the optical plane formed by the intersecting laser lines can further include: constructing the following equation using the direction vector corresponding to the remaining line: Where D = [d1, d2, ..., dm] T d1, d2, ..., dm are the direction vectors of the m remaining lines mentioned above; solve the constructed equations above. in, The normal vector of the optical plane formed by the laser lines emitted by the single line laser or the optical plane formed by the intersecting laser lines.

[0036] The preferred embodiment of simultaneously exciting two line lasers positioned on both sides to emit line lasers, forming a cross-line laser, is described below. It mainly includes the following steps:

[0037] Step 1: Take multiple images;

[0038] For example, 3D structured light technology typically employs a cross-line laser scheme, where two line lasers positioned on opposite sides are simultaneously excited, emitting line lasers that intersect. A marker board is placed 20 centimeters away from the camera, and multiple images are captured. It's important to note that the distance between the marker board and the camera should be greater than the distance between the intersection of the two laser lines and the camera. The marker board can be a flat checkerboard, a circular board, or an Apritag board, etc. It is crucial to ensure that the laser line projections on the marker board can be extracted from the captured images.

[0039] The laser emitted by the aforementioned line laser can be visible light or non-visible light; preferably, it can be infrared light.

[0040] Step 2: Calibrate the parameters of the camera and marker board;

[0041] By utilizing the corner points on the calibration board and employing existing calibration algorithms, such as the Zhang Zhengyou calibration algorithm, the intrinsic parameters of the camera (focal length f, camera principal point coordinates (cx, cy)), the extrinsic parameters of the calibration board (rotation matrix R and translation variable t), and the camera's distortion coefficients (including radial and tangential distortion coefficients) can be calibrated. This step helps correct image distortion in the camera and obtain accurate imaging parameters. The aforementioned Zhang Zhengyou calibration algorithm is an effective method for multi-camera calibration; it uses a marker board with internal reference points to detect at least six independent 2D-3D registrations, thereby determining the relationship between each camera.

[0042] Step 3: Laser line extraction;

[0043] For example, the center line of the laser line's projection onto the marking board is extracted from multiple captured images. Specifically, the pixel values ​​of the images are normalized to reduce the impact of changes in lighting conditions; then, gamma transform is used to enhance the image contrast, making the laser line more prominent; next, Gaussian filtering is applied to ensure the image conforms to a Gaussian distribution, thereby reducing noise interference. Finally, the Steger method is used to extract the center line of the laser line's projection onto the calibration board, ensuring that the extracted line is accurate and continuous.

[0044] Step 4: Distinguishing between the left and right light planes;

[0045] Since the intersection of the two light planes is located in front of the calibration plate and parallel to the imaging plane, according to the geometric spatial relationship, the intersection of these two light planes with the calibration plate is on both sides of the image center and does not intersect. That is, based on the spatial relationship of the intersecting laser lines, the light plane corresponding to the projection center line of the extracted laser line can be determined. The intersection line on the left side of the image center corresponds to the light plane formed by the line laser emitted by the right-side line laser, and the intersection line on the right side of the image center corresponds to the light plane formed by the line laser emitted by the left-side line laser.

[0046] Step 5: Calibration of the optical plane;

[0047] The accuracy of 3D structured light technology primarily relies on the precise calibration of the camera and the light plane. Camera calibration determines its intrinsic and extrinsic parameters to ensure that the captured image accurately reflects the geometric spatial features of the real scene. Light plane calibration involves the precise position and orientation of the laser line in three-dimensional space to ensure that the laser is correctly projected onto the target object. Specifically, it includes the following steps:

[0048] (1) The distortion coefficient of the camera is used to perform distortion correction on the extracted laser center line.

[0049] The distortion coefficients of a camera include radial distortion coefficients and tangential distortion coefficients, used to describe and correct radial and tangential distortions in an image. Radial distortion mainly manifests as stretching or compression near the image center, while tangential distortion causes image tilting or stretching. By using a polynomial model to describe the distortion and obtaining distortion correction parameters through methods such as calibration plates or calibrating the camera, image quality and accuracy can be effectively improved.

[0050] (2) Using the camera's intrinsic parameters (focal length f, camera principal point coordinates (cx, cy)), the distortion-corrected laser centerline is converted into a ray emitted from the camera. Specifically:

[0051] For each point (x, y) on the laser centerline after distortion removal, it can be converted into normalized camera coordinates using the camera's pinhole model:

[0052] X=(xc x ) / f

[0053] Y = (yc y ) / f

[0054] Where f is the focal length of the camera, (c x c y () represents the principal point coordinates of the camera;

[0055] Then, convert each of the above points into rays in the camera coordinate system:

[0056] r(t) = t·[X,Y,1] T

[0057] Where t is a parameter.

[0058] (3) Solve the equations of the ray and the external parameters of the marker plate simultaneously, and calculate the discrete points on the intersection line of the light plane and the marker plate plane. Details are as follows:

[0059] First, transform the ray from the camera coordinate system to the world coordinate system: r w (t) = R·r(t) + t, where the extrinsic parameters of the calibration plate include the rotation matrix R and the translation vector t.

[0060] Then, the plane equation of the calibration plate is n T ·p+d=0, where n is the plane normal vector, d is the distance to the origin, and p is a point on the calibration plate; the ray equation r w Substituting (t)=R·r(t)+t into the above equation for the plane of the calibration plate, solve for t;

[0061] Finally, after obtaining t, substitute it into r. w (t) can be used to calculate discrete points on the intersection line between the light plane and the calibration plate plane.

[0062] (4) To avoid cumulative error, RANSAC is used to fit a straight line to the discrete points mentioned above, and the equation of the intersection line is calculated. Each calibration plate plane has one and only one intersection line with a certain optical plane.

[0063] RANSAC (Random Sample Consensus) is an iterative method used to estimate statistical parameters or geometric models from a dataset. It identifies inliers (data points that fit the model) and outliers (data points that do not fit the model) and selects the model with the most inliers as the final result. Using RANSAC for line fitting mainly involves the following steps: randomly selecting a set of points from the dataset as samples; fitting a linear model using the selected sample points; calculating the distance from the remaining data points to the fitted line, and considering points with a distance less than a certain threshold as inliers; if the number of inliers exceeds a predetermined threshold and the quality of the fitted model meets the requirements, the model is considered acceptable; repeating the above steps multiple times, with each iteration potentially selecting different sample points for model fitting, and finally selecting the model with the most inliers as the best fit result.

[0064] (5) Based on the equations of the intersection lines above, for example, the equations of k straight lines were obtained on the light plane. Each straight line l i This can be represented in vector form as:

[0065] l i =ai +td i

[0066] Among them, a i It is the i-th point on the straight line, d i It is the direction vector of the line, and t is the parameter.

[0067] The normal vector n of the light plane is perpendicular to the direction vector d of all lines. i That is, satisfying:

[0068] n·d i =0

[0069] (6) First, use RANSAC to filter the normal vectors calculated from any two different direction vectors and obtain the optimal solution.

[0070] Specifically, from the obtained k equations, any two line equations can be selected. Based on these two line equations, a light plane can be determined. The normal vector of this light plane can be calculated using the different direction vectors of these two lines; that is, the cross product of the direction vectors of the two lines yields the normal vector of the light plane. This normal vector is then multiplied by the dot product of all the lines in the k equations. Based on the result of the dot product calculation, the line with the best perpendicularity to the k lines is found as the optimal solution for the normal vector. For example, for a certain normal vector n... i The normal vector n i Perform a dot product with the direction vector of each of the k lines (if the two vectors being multiplied are perpendicular, the dot product is 0), obtaining multiple dot product values. Take the absolute value of each dot product value and sum them to obtain the cumulative value SUM. i Repeat the above steps until the accumulated values ​​corresponding to the normal vectors of all light planes are calculated. Select the smallest accumulated value from all accumulated values ​​and take the normal vector corresponding to the smallest accumulated value as the optimal solution for the normal vector. Using the above scheme can further reduce the error.

[0071] (7) Among the k lines, the lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold (e.g., 10%) are removed.

[0072] (8) Based on the direction vectors corresponding to the remaining m lines out of the k lines, solve the simultaneous equations to obtain the final normal vector of the light plane.

[0073]

[0074] Where D = [d1, d2, ..., dm] T d1, d2, ..., dm are the direction vectors of the m remaining lines.

[0075] In existing technologies, each laser in a dual-laser system emitting intersecting laser lines is calibrated individually. This means each laser requires separate setup, measurement, and calculation. This method is not only time-consuming but also requires additional equipment to ensure the stability of the calibration process. This cumbersome calibration process places high demands on the precise installation and repeatability of the equipment, increasing the operational difficulty. In the multi-line structured light obstacle avoidance module calibration process provided in this application, when extracting the projection center line of the intersecting laser lines, the light plane corresponding to the extracted laser line's projection center line can be determined based on the spatial relationship of the intersecting laser lines. Therefore, for multiple lasers emitting intersecting laser lines, simultaneous calibration of multiple lasers can be achieved.

[0076] Furthermore, existing calibration methods are sensitive to noise and easily affected by factors such as changes in ambient light, equipment vibration, and measurement errors, leading to inaccurate calibration results. Even small deviations can accumulate into significant errors in the final system application, impacting overall system performance. In this application, RANSAC is used to filter the normal vectors calculated from any two different direction vectors and obtain the optimal solution. Among the multiple intersection equations obtained from the extracted projection centerline, lines whose perpendicularity deviation from the optimal normal vector solution exceeds a predetermined deviation threshold are discarded. The final light plane normal vector is obtained by simultaneously solving the equations of the direction vectors of the remaining lines that meet the predetermined conditions after the discarding. This error processing mechanism provides intelligent analysis and correction of various errors occurring during the calibration process, further improving the system's accuracy and reliability.

[0077] Step 6: Achieve joint calibration of laser and dirt.

[0078] Specifically, dirt detection equipment is used to perform dirt detection operations. For example, the dirt detection equipment uses cameras and image processing technology (edge ​​detection algorithms, etc.) to detect dirt information on the ground. Figure 2 As shown, the detected dirt information is projected onto a local grid in the camera coordinate system to confirm the actual detection range of dirt. Specifically, first, the dirt resolution is confirmed. If the dirt resolution is 5cm, a 5cm resolution grid image (e.g., a checkerboard grid image) is prepared in advance. Then, the grid image is laid on a flat ground. After placing the camera on the flat checkerboard ground, the dirty grid interval of each pixel is confirmed through the image. Then, a dirty pixel-grid hash table is constructed to query the grid where the dirty pixel is located.

[0079] The left and right light planes are projected onto a grid map (e.g., a checkerboard grid) on the ground to determine the grid interval where the projection lines are located. Then, a laser information-grid hash table is constructed to query the grid where the left and right light planes are projected.

[0080] A map is drawn based on the dirty pixel-grid hash table and the laser information-grid hash table. The laser information and dirty information are jointly labeled into the map. Using this map, obstacle information and dirty information can be determined simultaneously, realizing the effective integration of robot obstacle avoidance function and dirty detection function.

[0081] According to an embodiment of the present invention, a laser and dirt combined calibration device is also provided.

[0082] Figure 3 This is a structural block diagram of a laser and contamination combined calibration device according to an embodiment of the present invention. Figure 3 As shown, the laser and dirt joint calibration device includes: a first construction module 30, used to determine the dirt resolution, set the grid map corresponding to the dirt resolution on the ground, place a single camera or one of multiple cameras on the ground with the grid map, determine the grid position of one or more dirt points in the acquired image, and construct a first correspondence between the one or more dirt points and the grid; a second construction module 32, used to project the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers into the grid map, determine the grid position of the light plane projection, and construct a second correspondence between the light plane projection and the grid; and a joint calibration module 34, used to draw a map based on the first and second correspondences to achieve joint calibration of laser and dirt.

[0083] use Figure 3 The provided laser and dirt joint calibration device comprises a first construction module 30 that determines the grid position of one or more dirt points in the acquired image and establishes a first correspondence between the dirt points and the grid. A second construction module 32 that determines the grid position of the light plane projection and establishes a second correspondence between the light plane projection and the grid. A map is drawn based on the first and second correspondences, thus achieving joint calibration of laser and dirt. The map drawn after joint calibration can simultaneously determine obstacle information and dirt information, effectively integrating the robot's obstacle avoidance and dirt detection functions.

[0084] Preferably, such as Figure 4As shown, the second construction module 32 further includes: an image acquisition unit 320, used to set up marker boards at different locations and use a single camera or multiple cameras to capture multiple images of the marker boards, wherein the multiple images include: the projection of a laser line emitted by a single line laser on the marker board, or the projection of intersecting laser lines emitted by at least two line lasers on the marker board; an extraction unit 322, used to extract the projection center lines of the laser lines on the marker boards from the multiple images, wherein when the extracted projection center lines of intersecting laser lines are the center lines of the intersecting laser lines, the light plane corresponding to the extracted projection center lines of the laser lines is determined according to the spatial positional relationship of the intersecting laser lines; and a first calculation unit 324, used to... Based on each extracted projection center line, discrete points on the intersection line of the light plane and the marking plate plane are calculated. The discrete points are fitted to obtain multiple equations of the intersection lines. The determination unit 326 is used to obtain the normal vector of all light planes based on the equations of the multiple intersection lines, and determine the optimal solution of the normal vector from the normal vectors of all light planes. The elimination unit 328 is used to eliminate lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold from the equations of the multiple intersection lines, and obtain the direction vectors corresponding to the remaining lines. The second calculation unit 330 is used to construct equations using the direction vectors corresponding to the remaining lines, and calculate the normal vector of the light plane formed by the laser line emitted by the single line laser or the light plane formed by the intersecting laser lines.

[0085] It should be noted that specific details of the aforementioned laser and contamination combined calibration device can be found in the relevant references. Figures 1 to 2 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0086] According to an embodiment of the present invention, a dirt detection device is also provided.

[0087] Figure 5 This is a structural block diagram of a dirt detection device according to an embodiment of the present invention. Figure 5 As shown, the dirt detection device 5 includes: one or more line lasers 50 ( Figure 5 The diagram shows two line lasers 50_1 and 50_2 for emitting laser lines to achieve an obstacle detection method. When the aforementioned dirt detection device includes multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect. One or more cameras 52 with supplementary lighting modules, and / or one or more cameras 54 without supplementary lighting modules and one or more supplementary lighting devices 56 (…). Figure 5The diagram shows a camera 54 without a supplementary lighting module and a supplementary lighting device 56. The one or more cameras 52 with supplementary lighting modules and / or the one or more cameras 54 without supplementary lighting modules are used to time-division multiplex the acquired image information when the processor executes the obstacle detection method and the dirt detection method. The supplementary lighting module and / or the supplementary lighting device 56 are used to provide supplementary lighting to the camera acquiring the image information (typically, supplementary lighting is provided to the camera acquiring the image when executing the dirt detection method). The processor 58 is used to execute the obstacle detection method and the dirt detection method based on the time-division multiplexing of the acquired image information from the one or more cameras 52 with supplementary lighting modules and / or the one or more cameras 54 without supplementary lighting modules. The dirt points detected by the dirt detection method and the light plane formed by the laser lines emitted by the one or more line lasers are used to implement the laser and dirt joint calibration method as described above.

[0088] It should be noted that the aforementioned dirt detection device may simultaneously include one or more cameras 52 with supplementary lighting modules and one or more cameras 54 without supplementary lighting modules. Alternatively, the aforementioned dirt detection device may include only one or more cameras 52 with supplementary lighting modules, or only one or more cameras 54 without supplementary lighting modules. All of the above embodiments are within the protection scope of this invention. For the aforementioned cameras 52 with supplementary lighting modules, there is no need to provide a separate supplementary lighting device 56 to illuminate the camera 52. For the aforementioned cameras 54 without supplementary lighting modules, since the camera 54 does not have a self-illuminating function, a separate supplementary lighting device 56 is required to illuminate the camera 54.

[0089] Figure 5 An example is shown that includes only a camera 54 without a fill light module, a fill light device 56, and two line lasers 50_1 and 50_2.

[0090] The processor 58 can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0091] When the aforementioned processor 58 is working, it executes as follows: Figures 1 to 2The laser and dirt joint calibration method in the illustrated embodiment.

[0092] It should be noted that specific details regarding the aforementioned dirt detection equipment can be found in the relevant references. Figures 1 to 2 The relevant descriptions and effects in the illustrated embodiments are for understanding purposes only and will not be repeated here.

[0093] In summary, by utilizing the above-described embodiments provided by this invention, the grid positions of one or more dirt spots in the acquired image are determined, a first correspondence between the one or more dirt spots and the grid is constructed, the grid positions of the light plane projection are determined, a second correspondence between the light plane projection and the grid is constructed, and a map is drawn based on the first and second correspondences, achieving joint calibration of laser and dirt. The map drawn after joint calibration can simultaneously determine obstacle information and dirt information, effectively integrating the robot's obstacle avoidance function and dirt detection function. Furthermore, in the calibration process of the multi-line structured light obstacle avoidance module provided in this application, multiple lasers (e.g., dual lasers on the left and right sides) can be calibrated simultaneously, and the error handling scheme differs from existing technologies. By simultaneously exciting and calibrating multiple lasers, this application can perform accurate measurements within a unified framework, effectively avoiding the cumulative errors caused by individual calibration in traditional methods. In addition, the innovative error handling mechanism can intelligently analyze and correct various errors occurring during the calibration process, further improving the system's accuracy and reliability.

[0094] The above-disclosed embodiments are merely a few specific examples of the present invention. However, the present invention is not limited thereto, and any variations that can be conceived by those skilled in the art should fall within the protection scope of the present invention.

Claims

1. A method for combined laser and dirt calibration, characterized in that, include: Determine the dirt resolution, set the grid corresponding to the dirt resolution on the ground, place the dirt detection device with a single camera or multiple cameras on the ground with the grid, determine the grid position of the one or more dirt points in the acquired image, and establish a first correspondence between the one or more dirt points and the grid. Obtain the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers from multiple line lasers, project it onto the grid map, determine the grid position where the light plane projection is located, and construct a second correspondence between the light plane projection and the grid. Maps are drawn based on the first and second correspondences to achieve joint calibration of laser and dirt.

2. The method according to claim 1, characterized in that, Obtaining the optical plane formed by a laser line emitted from a single line laser, or the optical plane formed by intersecting laser lines emitted from at least two line lasers among multiple line lasers, includes: Marking boards are set up at different locations, and multiple images of the marking boards are captured using a single camera or multiple cameras. The multiple images include: the projection of a laser line emitted by a single line laser on the marking board, or the projection of intersecting laser lines emitted by at least two line lasers on the marking board. The projection center lines of the laser lines on the marking plate are extracted from the multiple images respectively. When the extracted projection center lines are of intersecting laser lines, the light plane corresponding to the extracted projection center lines of the laser lines is determined according to the spatial position relationship of the intersecting laser lines. Based on each extracted projection center line, discrete points on the intersection line of the light plane and the marking plate plane are calculated, and the discrete points are fitted to obtain the equations of multiple intersection lines. Based on the equations of the multiple intersection lines, obtain the normal vectors of all light planes, and determine the optimal solution of the normal vectors from the normal vectors of all light planes; In the equations of the multiple intersection lines, lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold are removed to obtain the direction vectors corresponding to the remaining lines; An equation is constructed using the direction vectors corresponding to the remaining straight lines, and the normal vector of the optical plane formed by the laser line emitted by the single line laser or the optical plane formed by the intersecting laser lines is calculated.

3. The method according to claim 2, characterized in that, Extracting the projection center line of the laser line on the marking plate from the multiple images includes: For each of the multiple images, the pixel values ​​of that image are normalized. Perform contrast enhancement and filtering processing on the image; The center line of the laser line projection on the marking plate is extracted from the processed image.

4. The method according to claim 2, characterized in that, Based on each extracted projection center line, the discrete points on the intersection line of the light plane and the marking plate plane are calculated, including: The intrinsic parameters of the camera, the extrinsic parameters of the marker plate, and the distortion coefficient of the camera are calibrated. The distortion coefficient of the camera is used to perform distortion correction on the extracted projection center line; The distortion-corrected projection centerline is transformed into a ray equation in the camera coordinate system using the camera's intrinsic parameters. Based on the ray equation and the extrinsic parameters of the marker plate, calculate the discrete points on the intersection line of the light plane and the marker plate plane.

5. The method according to claim 2, characterized in that, Based on the equations of the multiple intersection lines, the normal vectors of all light planes are obtained. Determining the optimal solution for the normal vectors from the normal vectors of all light planes includes: From the equations of the multiple intersection lines, randomly select two equations to obtain the light plane corresponding to the equations of the two intersection lines. Calculate the normal vector of the light plane. Perform a dot product calculation on the normal vector and the multiple intersection lines respectively to obtain multiple dot product values. After taking the absolute value of the multiple dot product values, sum them up to obtain the dot product sum value corresponding to the normal vector. Repeat this step until the dot product sum values ​​of the normal vectors of all light planes are obtained. The normal vector corresponding to the minimum dot product sum value among the obtained dot product sum values ​​is taken as the optimal solution of the normal vector.

6. The method according to claim 2, characterized in that, Using the direction vectors corresponding to the remaining straight lines to construct equations, the normal vectors of the optical plane formed by the laser lines emitted by the single line laser or the optical plane formed by the intersecting laser lines are calculated, including: Construct the following equations using the direction vectors corresponding to the remaining lines: Where, D=[d1,d 2, ...,dm] T d1, d2, ..., dm are the direction vectors of the m remaining straight lines; Solve the constructed equation in, The normal vector of the optical plane formed by the laser lines emitted by the single line laser or the optical plane formed by the intersecting laser lines.

7. A laser and dirt combined calibration device, characterized in that, The device includes: The first construction module is used to determine the dirt resolution, set the grid image corresponding to the dirt resolution on the ground, place a single camera or one of multiple cameras on the ground with the grid image set, determine the grid position of the one or more dirt points in the captured image, and construct a first correspondence between the one or more dirt points and the grid. The second construction module is used to project the light plane formed by the laser line emitted by a single line laser or the light plane formed by the intersecting laser lines emitted by at least two line lasers from multiple line lasers onto the grid map, determine the grid position where the light plane projection is located, and construct a second correspondence between the light plane projection and the grid. The joint calibration module is used to draw a map based on the first correspondence and the second correspondence to achieve joint calibration of laser and dirt.

8. The apparatus according to claim 7, characterized in that, The second building module further includes: An image acquisition unit is used to set up a marker board at different locations and use a single camera or multiple cameras to capture multiple images of the marker board. The multiple images include: the projection of a laser line emitted by a single line laser on the marker board, or the projection of intersecting laser lines emitted by at least two line lasers on the marker board. An extraction unit is used to extract the projection center lines of laser lines on the marking plate from the multiple images respectively. When the extracted projection center lines are intersecting laser lines, the light plane corresponding to the extracted projection center lines is determined according to the spatial positional relationship of the intersecting laser lines. The first calculation unit is used to calculate discrete points on the intersection line of the light plane and the marking plate plane based on each extracted projection center line, and to fit the discrete points to obtain the equations of multiple intersection lines. The determining unit is used to obtain the normal vectors of all light planes based on the equations of the multiple intersection lines, and to determine the optimal solution of the normal vectors from the normal vectors of all light planes. The elimination unit is used to eliminate lines in the equations of the plurality of intersection lines whose perpendicularity deviation from the optimal solution of the normal vector exceeds a predetermined deviation threshold, so as to obtain the direction vectors corresponding to the remaining lines. The second calculation unit is used to construct equations using the direction vectors corresponding to the remaining straight lines, and calculate the normal vector of the optical plane formed by the laser lines emitted by the single line laser or the optical plane formed by the intersecting laser lines.

9. The apparatus according to claim 8, characterized in that, The determining unit is further configured to: From the equations of the multiple intersection lines, randomly select two equations to obtain the light plane corresponding to the equations of the two intersection lines. Calculate the normal vector of the light plane. Perform a dot product calculation on the normal vector and the multiple intersection lines respectively to obtain multiple dot product values. After taking the absolute value of the multiple dot product values, sum them up to obtain the dot product sum value corresponding to the normal vector. Repeat this step until the dot product sum values ​​of the normal vectors of all light planes are obtained. The normal vector corresponding to the minimum dot product sum value among the obtained dot product sum values ​​is taken as the optimal solution of the normal vector.

10. A dirt detection device, characterized in that, include: One or more line lasers are used to emit laser lines to achieve an obstacle detection method, wherein when the dirt detection device includes multiple line lasers, at least two of the multiple line lasers emit laser lines that intersect. One or more cameras with a supplementary lighting module, and / or one or more cameras without a supplementary lighting module and one or more supplementary lighting devices, wherein the one or more cameras with a supplementary lighting module and / or the one or more cameras without a supplementary lighting module are used to time-division multiplex the acquired image information when the processor executes the obstacle detection method and the dirt detection method, and the supplementary lighting module and / or the supplementary lighting device are used to provide supplementary lighting to the cameras that acquire image information; The processor is configured to execute an obstacle detection method and a dirt detection method based on image information acquired by time-division multiplexing from one or more cameras with supplementary lighting modules and / or one or more cameras without supplementary lighting modules, wherein the dirt points detected by the dirt detection method and the light plane formed by the laser lines emitted by the one or more line lasers are used to implement the laser and dirt joint calibration method as described in any one of claims 1 to 6.

Images