Laser ray extraction and visualization method based on dual-camera complementation

Abstract

The invention relates to a laser ray extraction and visualization method based on dual-camera complementation, and belongs to the technical field of image processing. According to the method, accurate extraction and visualization of laser rays are realized by complementation of a camera with an optical filter and a camera without an optical filter. According to the invention, accurate extraction and visualization of laser rays can be realized.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a method for laser line extraction and visualization based on dual-camera complementarity. Background Technology

[0002] In the field of image processing technology, laser line extraction and visualization are crucial for various applications, including 3D reconstruction, virtual reality, and industrial measurement. Traditional laser line extraction methods are primarily based on a single camera. Single-line structured light technology projects a light beam or band onto a target surface and reconstructs the scene in 3D by observing the surface deformation or reflection. However, due to differences in the photosensitive elements and lens characteristics of different cameras, as well as the influence of environmental factors, these methods often struggle to simultaneously achieve high accuracy and robustness. In special environments and complex scenes, traditional structured light methods are often constrained by factors such as lighting and texture, making it difficult to achieve high-precision laser line extraction and visualization.

[0003] Current structured light technology often uses a single camera for laser line extraction, but this method has some limitations. When acquiring laser line information, some cameras use filters to block non-specific wavelengths of light to improve the contrast of the laser line. However, such cameras may lose crucial background and environmental information when extracting laser lines from grayscale images; furthermore, viewers cannot see the background and environment when viewing the image, thus negating the purpose of surveillance. Conversely, color cameras without filters, while possessing rich texture information, are significantly affected by changes in lighting and other environmental factors, resulting in inaccurate laser line extraction. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of a single camera and provide a novel laser line extraction and visualization method based on dual-camera complementarity, which comprehensively utilizes both a filter camera and a non-filter camera to achieve accurate extraction and visualization of laser lines.

[0005] To achieve the above objectives, the technical solution of the present invention is: a laser line extraction and visualization method based on dual-camera complementarity, which utilizes a camera with a filter and a camera without a filter to achieve accurate laser line extraction and visualization.

[0006] In one embodiment of the present invention, the method is implemented as follows:

[0007] (1) Camera parameter calibration with and without filter camera B;

[0008] (2) Extraction of laser points from camera B image;

[0009] (3) Project the laser points extracted in step (2) into three-dimensional space to obtain three-dimensional points;

[0010] (4) Project the three-dimensional points obtained in step (3) onto the image captured by camera A to obtain the projected laser points;

[0011] (5) Extract laser points from the image captured by camera A based on the color characteristics of the laser, and fuse them with the projected laser points obtained in step (4) to obtain the final laser points.

[0012] In one embodiment of the present invention, step (1) is specifically implemented as follows:

[0013] The camera parameter model is a pinhole model. A calibration board is placed in the common field of view of cameras A and B. The pose of the calibration board is changed, and cameras A and B simultaneously capture N sets of images. Using a dual-camera calibration method, the transformation relationship from camera B to camera A, i.e., rotation R and displacement t, as well as the intrinsic parameter K of camera A, are calculated. A and camera B internal reference K B Place the calibration plate in the area illuminated by the laser line, and calibrate the parameters L = {n} of the laser plane in the camera B coordinate system. a ,n b N c ,D}, where n a ,n b ,n c Let D be the laser plane normal vector and D be the intercept. After calibration, maintain the relative poses of camera A, camera B, and the laser generator.

[0014] In one embodiment of the present invention, step (2) is specifically implemented as follows:

[0015] A grayscale image containing the laser line is captured by camera B. The laser line region in the image is obtained through image thresholding. The center point of the laser line is calculated using the grayscale centroid method. These points are denoted as p. B =[u B v B ] T That is, the extracted laser point, where u B ,v B These are the pixel coordinates of the image.

[0016] In one embodiment of the present invention, step (3) is specifically implemented as follows:

[0017] Using the camera parameter matrix K calibrated in step (1) B Extracted laser point p B Projected into 3D space; 3D point P on the laser line in the camera's B coordinate system. B =[x B y Bz B ] T The laser beam lies on the ray connecting the optical center of the camera and the laser point in the image, and also on the laser plane L. Based on the geometric constraints of the three-dimensional points on the laser line, the equations are as follows.

[0018]

[0019] Substitute p B By solving for the parameters of the laser plane L, the three-dimensional point P is obtained. B .

[0020] In one embodiment of the present invention, step (4) is specifically implemented as follows:

[0021] Using the camera parameters calibrated in step (1), the extracted laser point P is... B Projecting the image onto the camera A; first, projecting the 3D point P in the coordinate system of camera B. B Transform to camera A coordinate system:

[0022] P A =RP B +t

[0023] R and displacement t are the rotation and displacement from camera B to camera A, respectively;

[0024] Using the intrinsic parameters of camera A, the 3D point P is... A In the image projected onto camera A, z A For a three-dimensional point P A Depth:

[0025] z A p A =K A P A

[0026] The laser point p extracted from the image of camera B is obtained. B Projected laser point p in the image of camera A A =[u A v A ] T , where u A ,v A These are the pixel coordinates of the image.

[0027] In one embodiment of the present invention, step (5) is specifically implemented as follows:

[0028] Calculate the projected laser point p in camera A according to step (4). A Then, the laser point p' is extracted from the image of camera A based on the color characteristics of the laser. A Project laser point p AThe extracted laser points are used as the initial state values ​​for observation. Where u represents the observation noise, which follows a Gaussian distribution with zero mean and variance U. The prediction of the laser point state is as follows: The prediction of the state covariance matrix is ​​P. - =P, where P is the initial state covariance matrix, and the state estimate of the laser point after fusion is: Where the gain E = P - (P - +U) -1 .

[0029] In one embodiment of the present invention, assuming the laser line is a red laser line, the laser point p' is extracted from the image of camera A based on the color characteristics of the laser. A The method is as follows:

[0030] Extracting the red feature from the image and denoting it as I R , Where R, G, and B are the three channels of the RGB color space image, image thresholding is used to extract the red feature map I. R The laser point in the middle.

[0031] The present invention also provides a laser line extraction and visualization system based on dual-camera complementarity, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps described above.

[0032] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0033] Compared with the prior art, the present invention has the following beneficial effects: The method of the present invention utilizes the complementary use of cameras with and without filters to achieve accurate extraction and visualization of laser lines, thus enabling accurate extraction and visualization of laser lines. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the system of the present invention.

[0035] Figure 2 This is a schematic diagram of dual-target positioning.

[0036] Figure 3 This is a schematic diagram of laser calibration.

[0037] Figure 4 The laser line was captured by a camera with a filter.

[0038] Figure 5 The laser line captured by the camera with a filter is projected onto the camera without a filter and then merged.

[0039] Figure 6 This is a flowchart of the method of the present invention. Detailed Implementation

[0040] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0041] It should be noted that the following detailed descriptions are exemplary and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0042] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this application. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0043] like Figure 6 As shown, this invention provides a laser line extraction and visualization method based on dual-camera complementarity, which utilizes a camera with and without filters to achieve accurate laser line extraction and visualization. The implementation steps of this method are as follows:

[0044] (1) Camera parameter calibration with and without filter camera B;

[0045] (2) Extraction of laser points from camera B image;

[0046] (3) Project the laser points extracted in step (2) into three-dimensional space to obtain three-dimensional points;

[0047] (4) Project the three-dimensional points obtained in step (3) onto the image captured by camera A to obtain the projected laser points;

[0048] (5) Extract laser points from the image captured by camera A based on the color characteristics of the laser, and fuse them with the projected laser points obtained in step (4) to obtain the final laser points.

[0049] The following are specific implementation examples of the present invention.

[0050] For example, the installation of each device in this invention Figure 1As shown, the system includes a camera without a filter, a camera with a filter, and a laser generator. The laser generator emits laser light of a specific frequency, illuminating the outline of an object's surface. Both camera images share a common viewing area, allowing the laser line illuminating the object's surface to be observed. First, calibration is performed: the extrinsic parameters between camera A (without a filter) and camera B (with a filter) are calibrated, as are the intrinsic parameters of cameras A and B. The plane equation of the laser plane in camera B's coordinate system is also calibrated. Then, the laser line is extracted from camera B's image, and the coordinates of points on the laser line in the three-dimensional space of camera B's coordinate system are calculated using the laser plane equation. Next, the 3D laser points in camera B's coordinate system are projected onto camera A's coordinate system using the extrinsic parameters between cameras A and B, and then reprojected onto the image using camera A's intrinsic parameters to obtain the projected laser points. Finally, the laser points are refined by merging the color image from camera A, and the points are plotted in the color image for visualization.

[0051] The specific implementation steps of the laser line extraction and visualization method based on dual-camera complementarity of the present invention are as follows:

[0052] (1) Calibration:

[0053] After installing and fixing camera A (without filter) and camera B (with filter), the camera parameters are first calibrated. A pinhole model is selected as the camera parameter model. The calibration board is placed in the shared field of view of cameras A and B. The pose of the calibration board is changed, and cameras A and B simultaneously capture N sets of images, such as... Figure 2 As shown. Using the dual-target calibration method, the transformation relationship from camera B to camera A is calculated, namely the rotation R and displacement t, as well as the intrinsic parameter K of camera A. A and camera B internal reference K B Place the calibration board in the area illuminated by the laser line, and camera B captures an image of the calibration board as shown below. Figure 3 As shown, the parameters L = {A, B, C, D} of the laser plane in the coordinate system of camera B are calibrated. After calibration, the calibration parameters will not change as long as the relative poses of camera A, camera B, and the laser remain unchanged.

[0054] (2) Extraction of laser points from camera B image:

[0055] In the grayscale image captured by camera B, which is equipped with a filter, the laser line is the most prominent part of the image. This is because the filter is a bandpass filter, which filters out most of the light frequencies other than the laser frequency, such as... Figure 4 As shown. The laser line region in the image is obtained through image thresholding segmentation, and then the center point of the laser line is calculated using the gray-scale centroid method. These points are denoted as p. B =[u B v B ] T .

[0056] (3) Projecting laser points to three dimensions:

[0057] Using the parameters calibrated in step (1), p B Projected into 3D space. A 3D point P on the laser line in the camera's B coordinate system. B =[x B y B z B ] T The laser beam lies on the ray connecting the optical center of the camera and the laser point in the image, and also on the laser plane L. Based on the geometric constraints of the three-dimensional points on the laser line, the equations are as follows.

[0058]

[0059] Substitute p B Given the laser plane parameter L, the three-dimensional point P can be obtained. B .

[0060] (4) Reproject the image onto camera A:

[0061] Using the parameters calibrated in step (1), P B Project it onto the image captured by camera A. First, project the 3D point P in the coordinate system of camera B. B Transform to the coordinate system of camera A.

[0062] P A =RP B +t.

[0063] Using the intrinsic parameters of camera A, the 3D point P is... A The image projected onto camera A,

[0064] z A p A =K A P A

[0065] The laser point p in the image from camera B is obtained. B Projection p in the image from camera A A =[u A v A ] T .

[0066] (5) Refine laser points by fusing color images:

[0067] Extracting laser lines solely from the grayscale image captured by camera A without a filter and the color image captured by camera B with a filter results in noise. To leverage the advantages of both images, information from the two images is fused to extract more accurate laser points. First, the projected laser point p in camera A is calculated according to step (4). A Then, the laser point p' is extracted from the image of camera A based on the color characteristics of the laser. A For example, if the laser line is a red laser line, the red feature extracted from the image is denoted as I. R , Where R, G, and B are the three channels of the RGB color space image, image thresholding is used to extract the red feature map I. R The projected laser points and the extracted laser points are then fused to obtain the final laser points. The projected laser points are used as the initial state values, and the extracted laser points are used as the observed laser points. The prediction of the laser point state is as follows The prediction of the state covariance matrix is ​​P. - =P, where P is the initial state covariance matrix. The state estimate of the laser point after fusion is: Where the gain E = P - (P - +U) -1 The result is as follows Figure 5 As shown.

[0068] The present invention also provides a laser line extraction and visualization system based on dual-camera complementarity, including a memory, a processor, and computer program instructions stored in the memory and executable by the processor. When the processor executes the computer program instructions, it can implement the steps described above.

[0069] The present invention also provides a computer-readable storage medium having stored thereon computer program instructions that can be executed by a processor, wherein when the processor executes the computer program instructions, it can implement the steps of the method described above.

[0070] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0071] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0072] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0073] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0074] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other way. Any person skilled in the art may make changes or modifications to the above-disclosed technical content to create equivalent embodiments. However, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the protection scope of the present invention.

Claims

1. A method for laser line extraction and visualization based on dual-camera complementarity, characterized in that, The laser line is accurately extracted and visualized by complementing the camera with and without the filter.

2. The method of claim 1, wherein, The method comprises the following steps: (1) Camera parameter calibration of camera B with filter and camera A without filter; (2) Laser point extraction in the image of camera B; (3) Projection of the laser points extracted in step (2) to the three-dimensional space to obtain three-dimensional points; (4) Projection of the three-dimensional points obtained in step (3) to the image taken by camera A to obtain projected laser points; (5) Laser point extraction from the image taken by camera A according to the color characteristics of the laser and fusion of the projected laser points obtained in step (4) to obtain the final laser points.

3. The method of claim 2, wherein, Step (1) is implemented as follows: The pinhole model is selected as the camera parameter model, the calibration board is placed in the common view area of camera A and camera B, the pose of the calibration board is transformed, and N groups of images are taken by camera A and camera B at the same time; Using the double target calibration method, the transformation relationship of camera B to camera A, i.e. rotation R and displacement t, and the camera A intrinsic K A and camera B intrinsic K B are calculated; the calibration board is placed in the area irradiated by the laser line, and the parameters L = {n a ,n b ,n c ,D} of the calibration laser plane in the camera B coordinate system are determined, wherein n a ,n b ,n c is the normal vector of the laser plane, and D is the intercept; after calibration, the relative poses of camera A, camera B and the laser generator are maintained.

4. The method of claim 2, wherein, Step (2) is implemented as follows: Based on camera B acquires gray image containing laser line, through image threshold segmentation to obtain laser line region in image, using gray gravity method algorithm to calculate laser line center point, record these points as p B = [u B v B ] T That is, the extracted laser point, wherein u B , v B It is image pixel coordinate.

5. The method of claim 2, wherein, Step (3) is implemented as follows: Using the camera parameter matrix K calibrated in step (1) B The extracted laser point p B is projected to the three-dimensional space; the three-dimensional point P on the laser line in the camera B coordinate system B = [x B y B z B ] T falls on the ray of the line connecting the camera optical center and the image laser point, and falls on the laser plane L; according to the geometric constraint of the three-dimensional point on the laser line, the equation is listed as follows, Bring in p B and laser plane L parameters, three-dimensional point P is solved B .

6. The method of claim 2, wherein, Step (4) is implemented as follows: The extracted laser points P B are projected into the image taken by camera A using the camera parameters calibrated in step (1); first the three-dimensional point P B is transformed into the camera A coordinate system: P A = RP B + t R and t are the rotation and displacement of camera B to camera A, respectively; Using the intrinsic parameters of camera A, the three-dimensional point P A is projected into the image of camera A, where z A is the depth of the three-dimensional point P A . z A p A = K A P A p is the laser point extracted in the camera B image B p is the projected laser point in the camera A image A = [u A v A ] T where u A , v A are image pixel coordinates.

7. The method of claim 2, wherein the method is based on complementary dual-camera laser line extraction and visualization. Step (5) is implemented as follows: According to step (4) calculate the projected laser point p in camera A A Then extract the laser point p' from camera A image according to the color characteristics of the laser A , project the laser point p A As the initial value of the state, the extracted laser point is taken as the observation Where u is the observation noise subject to a Gaussian distribution with mean 0 and variance U; the prediction of the laser point state is The prediction of the state covariance matrix is P - = P, where P is the initial state covariance matrix, and the state estimation of the laser point after fusion is Where the gain E = P - (P - + U) -1 .

8. The method of claim 7, wherein, Assuming the laser line is a red laser line, the laser point p' is extracted from the camera A image according to the color characteristics of the laser A in the following way: Extracting the red feature from the image and denoting it as I R , Where R, G, and B are the three channels of the RGB color space image, image thresholding is used to extract the red feature map I. R The laser point in the middle.

9. A dual camera complementary based laser line extraction and visualization system, comprising: The computer readable storage medium stores computer program instructions capable of being executed by the processor, and when the processor executes the computer program instructions, the method steps of any one of claims 1-8 can be implemented.

10. A computer readable storage medium, which stores computer program instructions capable of being executed by a processor, and when the processor executes the computer program instructions, the method steps of any one of claims 1-8 can be implemented.

Images