Using method of 3D visual sensor

Abstract

The invention relates to a use method of a 3D visual sensor, and the 3D visual sensor comprises a housing, and the 3D visual sensor also comprises a first transmission assembly which is rotatably disposed at the inner side of the housing and is used for transmitting line-shaped light to the surface of an object to be detected; an angle is formed between the receiving assembly and the first transmission assembly, and the receiving assembly is used for receiving the reflected light and imaging; and the second imaging assembly is arranged on the inner side of the shell in a micro-swinging manner, is obliquely positioned on one side of the first transmission assembly, and is used for imaging the other end of the object to be detected. The second imaging assembly and the receiving assembly are adopted to enable the receiving assembly to complete imaging, 2D imaging is carried out on the corresponding fuzzy part through the first transmission assembly, then fusion is carried out to construct a clear three-dimensional model, it is ensured that the scene of the to-be-measured object can be completely and clearly imaged, and it is ensured that clear and accurate imaging can be carried out to facilitate subsequent use.

Description

Technical Field

[0001] The present invention relates to the field of 3D imaging technology, and in particular to a method for using a 3D vision sensor. Background Art

[0002] With the rapid development of machine vision and intelligent manufacturing, non-contact 3D measurement and 3D reproduction technology have become hot issues in the research of major colleges and enterprises in recent years. By scanning the workpiece with a 3D vision sensor to form precise 3D point cloud data, the depth information of the object can be accurately detected. However, when there is an angle in the imaging process of the existing 3D vision sensor, the imaging angle is distorted, the detection is blurred, and the monitoring effect is affected. Summary of the invention

[0003] The present invention provides a method for using a 3D vision sensor, so as to at least solve the problem in the prior art that when an angle exists during the imaging process of the 3D vision sensor, one end of the imaging angle is distorted, the detection is blurred, and the monitoring effect is affected.

[0004] To implement the above technical solution, the technical solution of the present invention is as follows: a method for using a 3D vision sensor, the 3D vision sensor includes a housing, and the 3D vision sensor also includes:

[0005] A first transmission component, rotatably disposed inside the housing, for transmitting linear light to the surface of the object to be measured;

[0006] A receiving component, arranged at an angle with the first transmitting component, for receiving the reflected light and forming an image; and

[0007] A second imaging component, which is slightly swingable and disposed inside the housing and is tilted on one side of the first transmission component, and is used to image the other end of the object to be measured;

[0008] Wherein: the image formed by the second imaging component is fused with the image formed by the receiving component through the controller to form a panoramic three-dimensional depth-of-field image of the object to be measured.

[0009] Furthermore, the first transmission component includes a light source emitter horizontally arranged on the inner side of the shell; a rotating mechanism is rotatably provided along the optical path of the light transmitted by the light source emitter, which is used to change the refraction of the light to the surface of the object to be tested and move from the leftmost end to the rightmost end to complete the scanning.

[0010] Furthermore, the light source emitter is a laser emitter; and the rotating mechanism is one or more of a MEMS galvanometer or a rotatable plate-shaped refraction mirror.

[0011] Furthermore, the rotating mechanism is a MEMS galvanometer, and the light generated by the laser transmitter is transmitted to the surface of the object to be measured through the MEMS galvanometer for scanning.

[0012] Furthermore, the 3D vision sensor further comprises a controller; the controller is respectively connected to the first transmission component 、 Receiving Components 、 The second imaging component is electrically connected.

[0013] Furthermore, the 3D vision sensor also includes a first light source assembly and a second light source assembly; the first light source assembly can be adjusted to be arranged on the inner side of the shell, and is arranged along the receiving light path direction of the receiving assembly; the second light source assembly is arranged on the inner side of the shell, and is arranged along the shooting light path of the second imaging assembly.

[0014] Furthermore, the second imaging component is a 2D vision camera; a fine-tuning component is provided on one side of the 2D vision camera, and the fine-tuning component can drive the 2D vision camera to swing back and forth.

[0015] The present invention also discloses: a method for using a 3D vision sensor, wherein the first transmission component scans the light in a straight line on the surface of the object to be measured in the moving direction, and the light is received by the receiving component to form a first image, and at the same time, the second imaging component forms a second image at the tail end of the moving direction to be measured, and finally the controller merges the first image and the second image into a complete three-dimensional image.

[0016] Compared with the prior art, the present invention has the following beneficial effects: by adopting the second imaging component 、 The receiving component enables the receiving component to complete the imaging, and the corresponding blurred part is imaged in 2D through the first transmission component, and then fused to construct a clear three-dimensional model, ensuring that the image of the object to be measured can be completely and clearly imaged, ensuring clear and accurate imaging for subsequent use. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] To further illustrate the various embodiments, the present invention provides drawings. These drawings are part of the disclosure of the present invention, which are mainly used to illustrate the embodiments and can be used in conjunction with the relevant descriptions in the specification to explain the operating principles of the embodiments. With reference to these contents, a person of ordinary skill in the art should be able to understand other possible implementations and advantages of the present invention. The components in the figures are not drawn to scale, and similar component symbols are generally used to represent similar components.

[0018] Figure 1 This is a schematic diagram of the structure of the 3D vision sensor of the present invention;

[0019] Figure 2 Schematic diagram of sub-pixel center point. DETAILED DESCRIPTION

[0020] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0021] In order to enable those skilled in the art to better understand the scheme of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0022] Please see attached Figure 1 As shown: The embodiment of the present invention discloses a 3D vision sensor, which includes a detachable and hollow shell 1. The stamp 3D vision sensor also includes: a first transmission component 2, which can be rotatably arranged on the inner side of the shell 1, and is used to transmit linear light to the surface of the object to be measured; a receiving component 3, which is arranged at an angle with the first transmission component 2, and is used to receive the reflected light and perform imaging; and a second imaging component 4, which can be slightly swung and arranged on the inner side of the shell 1, and is tilted on one side of the first transmission component 2, and is used to perform imaging on the other end of the object to be measured; specifically, the image formed by the second imaging component 4 is fused with the image formed by the receiving component 3 through a controller to form a panoramic three-dimensional depth of field image of the object to be measured. In this embodiment, by using the second imaging component 、 The receiving component enables the receiving component to complete the imaging, and the corresponding blurred part is imaged in 2D through the first transmission component, and then fused to construct a clear three-dimensional model, ensuring that the image of the object to be measured can be completely and clearly imaged, ensuring clear and accurate imaging for subsequent use.

[0023] On the basis of the above embodiment, the first transmission component 2 includes a light source transmitter 21 horizontally arranged inside the housing 1, used for emitting light; a rotating mechanism 22 is rotatably arranged along the optical path of the light transmitted by the light source transmitter 21, used for changing the light refracted to the surface of the object to be tested and moving from the leftmost end to the rightmost end to complete the scanning. That is, when the light emitted by the light source transmitter 21 is projected onto the rotating mechanism 22, a light beam is formed and transmitted longitudinally to the surface of the object to be tested, and scans from one end of the object to be tested to the other end, and finally is received by the receiving component 3 for imaging.

[0024] Based on the above embodiment, the light source emitter 21 is a laser emitter; the rotating mechanism 22 is one or more of a MEMS galvanometer or a rotatable plate-shaped refraction mirror.

[0025] Based on the above embodiment, the rotating mechanism 22 is a MEMS galvanometer, and the light generated by the laser emitter is transmitted through the MEMS galvanometer to the surface of the object to be measured for scanning, that is, the light emitted by the laser emitter is projected onto the MEMS galvanometer, and the MEMS galvanometer can enable the laser to scan the object to be measured within an angle range of -38° to 38° at a specified rate.

[0026] Based on the above embodiment, the 3D vision sensor further includes a controller; the controller is respectively connected to the first transmission component 2 、 Receiving component 3 、 The second imaging component 4 is electrically connected.

[0027] Based on the above embodiment, the 3D vision sensor also includes a first light source component 5 and a second light source component 6; the first light source component 5 can be adjusted to be arranged on the inner side of the shell 1, and is arranged along the receiving light path direction of the receiving component 3; the second light source component 6 is arranged on the inner side of the shell 1, and is arranged along the shooting light path of the second imaging component 4.

[0028] On the basis of the above embodiment, the second imaging component 4 is a 2D vision camera; a fine-tuning component is provided on one side of the 2D vision camera, and the fine-tuning component can drive the 2D vision camera to swing back and forth; the fine-tuning component includes a connecting base and a mounting pendulum seat that are inserted into each other and can rotate, and a fine-tuning deformation source that can deform along the circumference direction is provided between adjacent connecting bases and mounting pendulum seats; the fine-tuning deformation source is expanded and contracted along the circumference direction under the action of an electric field, and then drives the mounting pendulum seat to rotate around the axis of the connecting base to achieve automatic adjustment, and the fine-tuning deformation source is an arc-shaped piezoelectric ceramic. When the shooting angle is not clear, the arc-shaped piezoelectric ceramic is connected to the circuit, and the connecting base and the mounting pendulum seat are driven to rotate relative to each other under the action of the electric field, thereby achieving adjustment of the 2D vision camera to ensure that the 2D vision camera is always at an optimal shooting angle, further improving the accuracy of 3D imaging.

[0029] The present invention also discloses a method for using the 3D vision sensor as described above, characterized in that the first transmission component 2 scans the light in a straight line on the surface of the object to be measured in the moving direction, and then the light is received by the receiving component 3 to form a first image. At the same time, the second imaging component 4 forms a second image at the tail end of the moving direction to be measured. Finally, the controller fuses the first image and the second image into a complete three-dimensional image.

[0030] Based on the above embodiment, the first and second images are processed by the following algorithm: When performing accurate 3D reconstruction tasks, we use a synchronous excitation method to trigger the laser projection and the data acquisition of the binocular camera at the same time. The initial step of this process involves the calibration of the binocular camera, which is a process of determining the internal and external parameters of the camera through a series of complex algorithms to ensure that the camera can accurately capture and interpret images of the real world. Based on the calibration, we use the epipolar constraint method, which is a common technique in 3D reconstruction to find matching points in different views by maintaining the consistency of the epipolar lines between the point pairs in the image.

[0031] This process first requires the acquisition of laser image data from the left and right lenses of the binocular camera. Subsequently, we use the laser line center extraction algorithm, which can identify and extract the center position of the laser in the image with high precision and convert it into sub-pixel coordinates, thereby greatly improving the accuracy of subsequent processing. Next, we use the image matching method based on epipolar constraints to find matching points in the left and right images according to the parameters calibrated by the binocular camera. This step is the key to the entire 3D reconstruction process, because only by correctly matching the points in the image can the subsequent 3D reconstruction work be accurate. Finally, with the help of these precisely matched image points, we can use the triangulation principle and other geometric transformation methods to complete the 3D reconstruction of the target object. This process not only relies on the high precision of the camera and laser projector, but also requires complex algorithm support to ensure that the final 3D model is both accurate and detailed. In this way, we can obtain high-quality 3D models without sacrificing details, which has broad application prospects in computer vision, robot navigation, and industrial manufacturing.

[0032] In order to accurately locate the center position of the structured light, it is necessary to obtain the normal of the initially extracted center line, and then perform the grayscale centroid method on the effective pixel point set in the normal direction.

[0033] In actual shooting, the obtained structured light images often contain noise, which greatly affects the structured light stripe extraction effect. To address this problem, this paper uses two-dimensional Gaussian convolution to calculate the extracted structured light image, which can effectively reduce the negative impact of noise on the extraction of stripe centers. The calculation method is as follows:

[0034]

[0035]

[0036] In the formula, f(x, y) represents the original image, g(x, y) is the Gaussian convolution kernel, δ represents the mean square error of the Gaussian function, and its size satisfies D is the stripe width.

[0037] After two-dimensional Gaussian convolution, the gradient distribution map of the stripe image can be obtained. The pixel-level center point obtained by the geometric center method is selected as the main point set for calculating the normal direction, and the deformation of the laser stripe can be quickly obtained. Assuming that (Gx, Gy) represents the gradient vector, the covariance matrix C of the gradient vector is formula (3):

[0038]

[0039] Where Gx and Gy represent the gradient vectors of the image, which are expressed as formula (4):

[0040]

[0041] The covariance matrix is ​​decomposed by principal component analysis, and the eigenvector corresponding to the maximum eigenvalue is the normal direction. The eigenvalue and the corresponding eigenvector are expressed as (5) to (8).

[0042]

[0043]

[0044]

[0045]

[0046] In formulas (5) to (8), the eigenvalues ​​λ1 and λ2 correspond to eigenvectors v1 and v2, respectively. From the expression, we can see that λ1>λ2, so the corresponding eigenvector v1 is the normal direction of the stripe. Normalizing the normal vector can obtain a unit vector n=(nx, ny) that is consistent with the normal direction.

[0047] Finally, the grayscale centroid method is performed using equations (9) and (10).

[0048]

[0049]

[0050] In the formula, Gx(x, y) represents the pixel set in the x direction of the center point, and Gy(x, y) represents the pixel set in the y direction of the center point. xi and yi represent the final extracted structured light center point. By traversing all the center points obtained by the geometric center method, the center point is found to obtain the effective pixel point set of the sub-pixel point, and finally the sub-pixel center point can be obtained by using the grayscale centroid method. Figure 2 shown.

[0051] According to the constraint principle of the epipolar geometry of the binocular camera, a point X is set in space, and the projection points on the left and right cameras are m and m', which are called the matching points of the image and are recorded as Then m′ is located on the epipolar line l′ corresponding to m m On the left, m is located on the polar line lm' corresponding to m', that is, m'∈l' m ,m∈lm', as shown in Figure 7. The relationship between the image points of the binocular camera and its epipolar lines can be expressed using the basic matrix. Let the two camera matrices be p, p', and the phase planes of the two cameras be I, I' respectively, then Backprojection epipolar The parametric equation is:

[0052] X(s)=p + m+sC,s∈(-∞,+∞)

[0053] where p + is the generalized inverse of P, that is, PP + =I, C is the optical center of the first camera, that is, PC = 0, so we can get:

[0054] l′ m =e′m′=(P′C)(P′X(s))=(P′C)((P′Pm+sP′C))

[0055] =(P′C)((P′Pm)=[e′] + P'P + m

[0056] That is l′ m =[e′] + P'P + m, so the fundamental matrix between the two cameras F = [e′] × P′P +

[0057] Assume that the intrinsic matrix of the left camera is K, the memory matrix of the right camera is K', and the transformation relationship from the right camera to the left camera is [R, t]. Therefore, the coordinate transformation relationship between the right camera and the left camera is X' = RX + t, where R is the rotation matrix and t is the translation vector. The projection matrix of the left and right cameras can be expressed as:

[0058] P=K(I,0),P′=K′(R,t) (3)

[0059] according to so:

[0060]

[0061] Therefore, according to formula (4), we only need to know the intrinsic and extrinsic parameters of the two cameras to calculate the basic matrix, thereby determining the epipolar constraint relationship between the two cameras.

[0062] Based on the spatial geometric relationship, the three-dimensional coordinates of the spatial point P{x, y, z} can be derived. Among them, (x, y, z) is the quantity to be determined, that is, the three-dimensional coordinates of the spatial point.

[0063]

[0064] The above formula represents the rotation and translation matrix between the two cameras, (f l ,f r ) represents the focal length of the left and right cameras, (X l ,Y l ),(X r ,Y r ) represent the coordinates of the spatial point on the left and right camera image planes, (u l ,v l ) represents the pixel coordinates of the spatial point on the acquired image, and (u0,v0) represents the coordinates of the principal point.

[0065]

[0066] x=zX l / f l ,y=zY l / f l

[0067] The above is only a preferred embodiment of the present invention, and does not limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment, it is not used to limit the present invention. Any person skilled in the art should be able to use the technical content disclosed above to make equivalent embodiments that are equivalent changes by slight changes or modifications without departing from the scope of the technical solution of the present invention. However, any brief modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.

Claims

1. A method for using a 3D vision sensor, the 3D vision sensor comprising a housing (1), characterized in that: The 3D vision sensor also includes: A first transmission component (2) is rotatably arranged inside the housing (1) and is used to transmit linear light to the surface of the object to be measured; a receiving component (3), arranged at an angle with the first transmitting component (2), and used for receiving the reflected light and forming an image; and A second imaging component (4) is arranged on the inner side of the housing (1) in a manner that allows for slight swinging, and is obliquely located on one side of the first transmission component (2), and is used for imaging the other end of the object to be measured; Wherein: the image formed by the second imaging component (4) is fused with the image formed by the receiving component (3) through a controller to form a panoramic three-dimensional depth-of-field image of the object to be measured.

2. The method for using a 3D vision sensor as claimed in claim 1, characterized in that: The first transmission component (2) comprises a light source emitter (21) arranged horizontally on the inner side of the housing (1); a rotating mechanism (22) is rotatably provided along the optical path of the light transmitted by the light source emitter (21) for changing the refraction of the light to the surface of the object to be tested and moving from the leftmost end to the rightmost end to complete scanning.

3. The method for using a 3D vision sensor as claimed in claim 2, characterized in that: The light source emitter (21) is a laser emitter; the rotating mechanism (22) is one or more of a MEMS oscillating mirror and a rotatable plate-shaped refraction mirror.

4. The method for using a 3D vision sensor as claimed in claim 3, characterized in that: The rotating mechanism (22) is a MEMS oscillating mirror, and the light generated by the laser transmitter is transmitted to the surface of the object to be measured through the MEMS oscillating mirror for scanning.

5. A method for using a 3D vision sensor as claimed in any one of claims 1 to 4, characterized in that: The 3D vision sensor further comprises a controller; the controller is respectively connected to the first transmission component (2) 、 Receiving Components(3) 、 The second imaging component (4) is electrically connected.

6. The method for using a 3D vision sensor as claimed in claim 5, characterized in that: The 3D vision sensor further comprises a first light source assembly (5) and a second light source assembly (6); the first light source assembly (5) is adjustably arranged on the inner side of the housing (1) and is arranged along the receiving light path direction of the receiving assembly (3); the second light source assembly (6) is arranged on the inner side of the housing (1) and is arranged along the shooting light path of the second imaging assembly (4).

7. The method for using a 3D vision sensor according to claim 6, characterized in that: The second imaging component (4) is a 2D vision camera; a fine-tuning component is provided on one side of the 2D vision camera, and the fine-tuning component can drive the 2D vision camera to swing back and forth.

8. A method for using the 3D vision sensor as claimed in claim 1, characterized in that: The first transmission component (2) scans the light in a straight line on the surface of the object to be measured in the moving direction, and then the light is received by the receiving component (3) to form a first image. At the same time, the second imaging component (4) forms a second image at the tail end of the moving direction to be measured. Finally, the controller fuses the first image and the second image into a complete three-dimensional image.

Images