3D camera-based point cloud acquisition method and system

Abstract

The invention provides a point cloud acquisition method and system based on a 3D camera, the method relates to the 3D camera provided with a camera, a laser and a movable reflector, the method comprises the following steps: after determining an imaging mode, controlling the laser to emit laser to the reflector according to the imaging mode, the imaging mode comprises a fixed line mode, a swing line scanning mode and an area array structure light projection mode; and calling the camera to collect reflection laser of the to-be-measured object to obtain reflection data, and calculating the reflection data to obtain three-dimensional point cloud data. The laser and the galvanometer can be controlled to work cooperatively according to different modes, so that the processing frequency can be reduced, and the processing efficiency can be improved; noise of reflection data can be reduced, data precision is improved, cameras of different 3D imaging principles are fused in the same set of equipment, and imaging flexibility is further improved.

Description

Technical Field

[0001] This application relates to the field of 3D cameras, and particularly to a method and system for obtaining point clouds based on a 3D camera. Background Art

[0002] A 3D camera, also known as a depth camera, is different from a 2D camera. A 3D camera can obtain the depth distance of the shooting space. Through the data obtained by the 3D camera, we can accurately know the distance of each point in the image from the camera. Adding the (x, y) coordinates of the point in the 2D image, we can obtain the three-dimensional space coordinates of each point in the image. By restoring the real scene through the three-dimensional coordinates, applications such as scene modeling, product volume detection, defect detection, and railway detection can be realized.

[0003] Common 3D cameras generally only carry one type of light source device (a light source device that can project lines or planes). By driving the light source device to emit light towards the object to be measured, and then collecting the reflected light through the camera for three-dimensional imaging, three-dimensional point cloud data of lines or planes can be obtained.

[0004] However, the current common methods have the following technical problems: Scanning with one type of structured light source results in single reflected light source data, small capacity of the reflected light source data, and a lot of noise, making it difficult to meet the imaging requirements. When it is necessary to obtain three-dimensional point cloud data of a plane or a solid, multiple scans or an increase in the light source device are often required to achieve plane three-dimensional imaging. Multiple scans have a slow processing speed and low processing efficiency; while increasing the light source device will increase the processing cost and requires continuous manual adjustment, further reducing the processing efficiency and having low imaging flexibility. Summary of the Invention

[0005] In view of the above problems, this application is proposed to provide a method and system for obtaining point clouds based on a 3D camera that overcomes the above problems or at least partially solves the above problems, including:

[0006] A method for obtaining point clouds based on a 3D camera, the method involving a 3D camera provided with a camera, a laser, and a movable mirror, the method including:

[0007] After determining the imaging mode, control the laser to emit laser towards the mirror according to the imaging mode, so that the mirror projects the laser onto the object to be measured, and the imaging mode includes: fixed line mode, swinging line scan mode, and area array structured light projection mode;

[0008] Call the camera to collect the reflected laser of the object to be measured to obtain reflected data, and calculate the reflected data to obtain three-dimensional point cloud data.

[0009] In a possible implementation, controlling the laser to emit laser light towards the mirror according to the imaging mode includes:

[0010] If the imaging mode is the fixed line mode, determine a fixed angle value;

[0011] After controlling the mirror to rotate to the position of the fixed angle value, control the laser to emit laser light towards the mirror.

[0012] In a possible implementation, the reflection data is a single-line laser image collected by the camera;

[0013] Calculating the reflection data to obtain three-dimensional point cloud data includes:

[0014] After undistorting and epipolar rectifying the sub-pixel center line coordinates of the laser line projected by the mirror by the camera, search for corresponding matching points according to the center line of the sub-pixel center line coordinates;

[0015] After determining the coordinates of the matching points in the normalized coordinate system, construct a normalized coordinate point equation with the single-line laser image and the coordinates of the matching points and solve it to obtain three-dimensional point cloud data.

[0016] In a possible implementation, controlling the laser to emit laser light towards the mirror according to the imaging mode includes:

[0017] If the imaging mode is the swinging line scan mode, determine the projection angle value of the rotation of the mirror within a unit time;

[0018] Control the rotation of the mirror according to the projection angle value and control the laser to emit laser light towards the mirror.

[0019] In a possible implementation, the reflection data is a moving image of continuous laser lines projected during the scanning of the mirror collected by the camera;

[0020] Calculating the reflection data to obtain three-dimensional point cloud data includes:

[0021] Obtain the single-line point cloud data of the moving image at each scanning position, where the single-line point cloud data is point cloud data obtained by constructing an equation with the point cloud data of the moving image and solving it;

[0022] Merge the single-line point clouds at each scanning position to obtain three-dimensional point cloud data.

[0023] In a possible implementation, controlling the laser to emit laser light towards the mirror according to the imaging mode includes:

[0024] If the imaging mode is the area array structured light projection mode, control the swing of the reflector according to a preset field of view range;

[0025] Control the laser to intermittently emit laser light to the reflector.

[0026] In a possible implementation manner, the reflection data is multiple area array structured light images collected by a camera;

[0027] Calculating the three-dimensional point cloud data from the reflection data includes:

[0028] Calculating the wrapped phase of the phase shift fringes of the area array structured light image and unwrapping the wrapped phase into an absolute phase image;

[0029] After de-distorting and epipolar correcting the phase map corresponding to the absolute phase using the internal and external parameters of the camera, search for phase matching pairs;

[0030] Construct a normalized coordinate point equation using the coordinates of the phase matching pairs and solve it to obtain three-dimensional point cloud data.

[0031] A point cloud acquisition system based on a 3D camera, the system includes: a 3D camera and a control unit applicable to the point cloud acquisition method based on a 3D camera as described above;

[0032] The control unit is used to control the fixation or rotation of the reflector of the 3D camera; control the laser of the 3D camera to emit laser light so that the reflector is sent to the object to be measured; and is used to control the camera of the 3D camera to collect the reflection data of the object to be measured reflecting the laser light, so as to calculate the point cloud data through the reflection data.

[0033] In a possible implementation manner, the 3D camera includes: a base and at least one image collector, the base is provided with a laser and a movable reflector, and the image collector is arranged on the side of the base.

[0034] In a possible implementation manner, the laser is a line laser, and the line laser coincides with the galvanometer mirror of the reflector on the rotation axis in the horizontal direction.

[0035] A device, including a processor, a memory, and a computer program stored on the memory and capable of running on the processor, the computer program realizes the steps of the point cloud acquisition based on a 3D camera as described above when executed by the processor.

[0036] A computer-readable storage medium, on which a computer program is stored, the computer program realizes the steps of the point cloud acquisition based on a 3D camera as described above when executed by a processor.

[0037] The present application has the following advantages:

[0038] In an embodiment of the present application, the present application can control a laser to emit laser light towards a mirror according to different imaging modes, so that the mirror projects the laser light onto an object to be measured; call a camera to collect the reflected laser light of the object to be measured to obtain reflection data, and calculate the reflection data to obtain three-dimensional point cloud data. The present application can control the laser and the galvanometer to work together according to different modes, without the need for multiple scanning processes, which can reduce the number of processes and improve the processing speed, thereby improving the processing efficiency; moreover, the present application uses a laser to emit laser light, and the laser has strong anti-high reflection, anti-multiple reflection, and anti-strong light capabilities, which can reduce the noise of the reflection data and improve the data accuracy; in addition, no additional light source device is required throughout the process, which can reduce the processing cost and does not require manual adjustment. Different 3D imaging principle cameras can be integrated into the same set of equipment, thereby further improving the flexibility of imaging. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the present application, the drawings required for the description of the present application will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0040] Figure 1 is a flowchart of the steps of a method for obtaining point cloud based on a 3D camera provided by an embodiment of the present application;

[0041] Figure 2 is a structural block diagram of a device for obtaining point cloud based on a 3D camera provided by an embodiment of the present application;

[0042] Figure 3 is a structural block diagram of a system for obtaining point cloud based on a 3D camera provided by an embodiment of the present application;

[0043] Figure 4 is a schematic structural diagram of a 3D camera provided by an embodiment of the present application;

[0044] Figure 5 is a schematic structural diagram of a computer device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0045] To make the objectives, features, and advantages of this application more apparent and understandable, the following provides a more detailed description of this application in conjunction with the accompanying drawings and specific embodiments. Obviously, the described embodiments are part of this application, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments in this application without creative efforts belong to the scope protected by this application.

[0046] A 3D camera, also known as a depth camera, is different from a 2D camera. A 3D camera can obtain the depth distance of the shooting space. From the data obtained by the 3D camera, we can accurately know the distance of each point in the image from the camera. By adding the (x, y) coordinates of this point in the 2D image, we can obtain the three-dimensional space coordinates of each point in the image, and restore the real scene through the three-dimensional coordinates to achieve applications such as scene modeling, product volume detection, defect detection, and railway detection.

[0047] Common 3D cameras generally only carry one light source device. By driving the light source device to emit light towards the object to be measured, and then collecting the reflected light source through the camera for three-dimensional imaging to obtain three-dimensional point cloud data of lines or surfaces.

[0048] For example, a 3D camera is equipped with a laser or a structured light device. When line imaging is required, the 3D camera uses a fixed laser to emit laser light, uses a monocular or binocular camera to collect the laser line, and uses the principle of triangulation to obtain 3D point information on the fixed laser line to obtain three-dimensional point cloud data of the line. When surface imaging is required, a motor is used to drive the laser, or a laser rotating mirror system is adopted to make the laser line scan left and right. A monocular or binocular camera is used to collect the laser line to generate three-dimensional point cloud data of the surface of the production scene. Or a structured light projection device (such as DLP projection, galvanometer projection, MEMS projection, etc.) is used to quickly project a surface structured light pattern onto the scene. The camera is used to collect the structured light pattern, and through surface structured light matching and calculation, three-dimensional point cloud data of the surface of the scene is generated.

[0049] However, the current common methods have the following technical problems: By scanning with one type of structured light source (for example, scanning alone with a laser that emits a laser line or scanning alone with a structured light device that emits a surface light source), since the laser can only emit one beam of laser each time, the reflected light source data has only a single line of data, and the capacity of the reflected light source data is small, making it difficult to meet the imaging requirements; while the structured light device that emits a surface array light source has poor anti-high reflection, anti-multiple reflection, and anti-strong light capabilities, with a lot of noise in the reflected data and large data deviation, and it is also difficult to meet the imaging requirements.

[0050] When it is necessary to obtain the three-dimensional point cloud data of a surface or a solid, it is often necessary to perform multiple scans, add an external moving guide rail or other moving devices, or add a light source device to achieve three-dimensional surface imaging. However, multiple scans have a slow processing speed and low processing efficiency. Adding external devices will increase the processing cost, and manual adjustment is required continuously, further reducing the processing efficiency. The flexibility of imaging is low, and the positioning and tracking functions that can only be achieved by line scanning cannot be realized, making it difficult to meet the existing usage requirements.

[0051] For example, in application scenarios such as welding and cutting, a fixed line scan or area array camera is often used simultaneously. The imaging quality of a line scan camera is relatively high, but the efficiency is low. It often needs to move to multiple perspectives for shooting in order to use multiple lines for positioning to determine the workpiece pose. A swinging line scan can also image a surface, but the efficiency is not very high. An area array camera is generally only used for rough positioning with high speed but low accuracy, and it is difficult to stably image workpieces under conditions of high reflectivity, multiple reflections, and strong light. This results in users often having to choose one of the three types of cameras, or use multiple cameras simultaneously, causing the imaging system to be large in volume, complex in structure, high in cost, and poor in stability.

[0052] Referring to Figure 1 , a flowchart showing the steps of a method for obtaining point cloud based on a 3D camera provided by an embodiment of the present application is shown;

[0053] In one embodiment, the method for obtaining point cloud based on a 3D camera is applicable to the control unit of the 3D camera. The control unit can communicate with the 3D camera to control the 3D camera to emit a light source to the object to be measured and collect the light source reflected by the object to be measured, and then calculate the three-dimensional point cloud data according to the light source reflected by the object to be measured.

[0054] In one embodiment, the 3D camera is provided with a camera, a laser, and a movable mirror. The mirror can be a structure with similar rotational reflection such as a galvanometer scanner, a rotating mirror, a MEMS, etc.; the laser can be a line laser.

[0055] Among them, as an example, the method for obtaining point cloud based on a 3D camera may include:

[0056] S11. After determining the imaging mode, control the laser to emit laser to the mirror according to the imaging mode, so that the mirror projects the laser to the object to be measured. The imaging mode includes: fixed line mode, swinging line scan mode, and area array structured light projection mode.

[0057] In one embodiment, the imaging mode input by the user can be obtained. The imaging mode includes: fixed line mode, swinging line scan mode, and area array structured light projection mode.

[0058] Among them, the fixed-line mode can be to fixedly emit a single-line laser so that the laser is reflected into the scene to form a laser line. Then, calculate the 3D points on this laser line.

[0059] The swinging line scan mode can be to emit a single-line laser that swings back and forth, so that the laser can scan the object and form a laser line that sweeps across the target scene. Then, calculate the 3D points on the laser line in this frame of image.

[0060] The area array structured light projection mode can be a mode in which various fringe structured light patterns are projected by a laser, and the three-dimensional imaging result of the surface is calculated by obtaining the complete structured light pattern.

[0061] Since different modes require different lasers to be projected, the mirror and the laser can be controlled to work together respectively, so that different lasers can be emitted.

[0062] In this application, the control unit can control the 3D camera to work in different modes, and can emit different lasers according to actual needs to obtain point cloud data and perform subsequent imaging processing. There is no need to perform multiple scanning processes, which can reduce the number of processes, improve the processing speed, and thus improve the processing efficiency. And this application uses a laser to emit laser light, and the laser has strong anti-high reflection, anti-multiple reflection, and anti-strong light capabilities, which can reduce the noise of the reflected data and improve the data accuracy.

[0063] In addition, the entire process is controlled by the control unit, without the need to additionally increase light source equipment, which can reduce the processing cost and does not require manual adjustment, further improving the flexibility of imaging.

[0064] In order to enable the laser to emit laser light to the mirror at a fixed position, among them, by way of example, controlling the laser to emit laser light to the mirror according to the imaging mode may include the following sub-steps:

[0065] S21. If the imaging mode is the fixed-line mode, determine the fixed angle value.

[0066] S22. After controlling the mirror to rotate to the position of the fixed angle value, control the laser to emit laser light to the mirror.

[0067] In the fixed line scanning mode, the single-line laser is turned on, and the angular value corresponding to the fixed position of the mirror can be determined to obtain the fixed angular value. Then, the mirror can be controlled to rotate to a fixed angle according to the fixed angular value. This angle can be customized and only needs to be within the field of view of the 3D camera of the monocular camera or the binocular camera. The single-line laser is reflected by the galvanometer lens of the mirror into the scene to form a laser line. Subsequently, the 3D camera of the monocular camera or the binocular camera is used to collect this laser line to calculate the 3D points on this laser line.

[0068] In order to enable the laser to emit laser light to the mirrors at different positions, for example, the controlling the laser to emit laser light to the mirror according to the imaging mode may include the following sub-steps:

[0069] S31. If the imaging mode is the swinging line scanning mode, determine the projection angular value of the rotation of the mirror within a unit time.

[0070] S32. Control the rotation of the mirror according to the projection angular value, and control the laser to emit laser light to the mirror.

[0071] For example, if the imaging mode is the swinging line scanning mode, the projection angular value of the rotation of the mirror within a unit time can be determined. Suppose the required projection range angle set by the user is Δθ. The rotation range of the mirror is equal to this projection range angle Δθ, and the set scanning rotation time is Δt. Starting from time t 0 , the mirror starts to rotate from the starting angle θ 0 ; at time t 0 +Δt, the mirror rotates to the ending angle θ 0 +Δθ.

[0072] It should be noted that in the swinging line scanning mode, within the entire Δt scanning time, the laser line can remain constantly on, so that the 3D camera can continuously and rapidly collect the projection position images of the laser line.

[0073] In order to enable the laser to emit laser light to the mirror to form a complete view of a plane, for example, the controlling the laser to emit laser light to the mirror according to the imaging mode may include the following sub-steps:

[0074] S41. If the imaging mode is the area array structured light projection mode, control the swinging of the mirror according to the preset field of view range.

[0075] S42. Control the laser to intermittently emit laser light to the mirror.

[0076] In one operation mode, when using the area array structured light mode, by controlling the high-speed swing of the mirror within the field of view and synchronously controlling the laser to turn on or off, various stripe structured light patterns can be projected. The start exposure time of the 3D camera with a monocular camera or a binocular camera is also synchronized with the projection. When the laser reflected by the mirror starts to rotate into the field of view, the camera starts to expose. When the laser reflected by the mirror moves out of the field of view, the laser camera ends the exposure. This can ensure that the 3D camera with a monocular camera or a binocular camera can collect a complete area array structured light pattern projected in each frame.

[0077] For example, if it is necessary to project and collect multiple area array structured light stripes, the scanning and projection time for each pattern is Δt. Using the stripe pattern projection method of the above area array structured light mode, it is only necessary to synchronously start the camera exposure at the moment t n when the projection of the nth pattern starts. The exposure time is the same as the scanning and projection, both being Δt, and the exposure ends at the moment t n + Δt. And so on, after nΔt time, the acquisition images of all projected area array structured light patterns can be obtained.

[0078] In addition, when using the area array structured light mode, according to the different structured light patterns, the laser will determine the on or off time according to the bright and dark positions of the pattern. The structured light pattern mentioned here only has bright and dark changes in the scanning direction, but in the direction perpendicular to the scanning direction, that is, the direction parallel to the laser line, the brightness will remain consistent. For example, if it is necessary to project an area array structured light pattern with n periods of stripes (half bright and half dark within the stripe period), the projection will be controlled as follows. At the moment t 0 , the galvanometer starts to rotate from the starting angle θ 0 , and the laser remains on, and a pattern area that is half bright within the first stripe period (i = 1, i ≤ n) can be projected; at moment, the galvanometer rotates to half of the angle of the first stripe period and the laser is turned off, and the remaining half of the stripe period is dark. And so on, as i changes from 1 to n, an area array structured light pattern with n periods of equally spaced bright and dark changes can be projected.

[0079] In the fixed line scan mode, the mirror does not need to swing, and only needs to be controlled to maintain a certain fixed angle, such as θ x , and the laser remains on.

[0080] By coordinately controlling the mirror and the laser to work synchronously, cameras with different 3D imaging principles can be integrated in the same set of equipment, and at the same time, the camera can flexibly switch between the above three modes, improving the flexibility of operation.

[0081] S12. Call the camera to collect the reflected laser of the object to be measured to obtain reflection data, and calculate the reflection data to obtain three-dimensional point cloud data.

[0082] In one embodiment, after emitting the laser, the camera of the 3D camera can be called to collect the data of the reflected laser of the object to be measured to obtain the reflection data. Subsequently, according to the reflection data, resolution, matching, and three-dimensional imaging calculations are performed to obtain the three-dimensional point cloud result of the line or surface.

[0083] For a 3D camera with a monocular camera or a binocular camera, the control unit can use a hardware trigger signal to maintain high-precision synchronization of projection and collection for both projection and collection.

[0084] Before calling the camera for data collection, calibration can be performed; for a 3D camera with a monocular camera or a binocular camera, the internal parameters of the camera need to be calibrated first. The Zhang Zhengyou calibration method can be used, and a checkerboard or circular array calibration plate is used to sample the calibration plate at various positions and poses, and the internal parameter matrix K of the camera is calculated. i and the distortion coefficient D i . If it is a binocular system, it is also necessary to calculate the binocular external parameter relationship [R i T i using the same calibration plate position.

[0085] For monocular structured light, the mirror projection system can be calibrated. The calibration plate can be placed at different poses in the imaging space, and then the mirror is used to project the phase-shifted structured light fringe pattern onto the calibration plate. By solving the phase and the internal parameter matrix K of the monocular i and the distortion coefficient D i [k 1 ,k 2 ,k 3 ,p 1 ,p 2 , the mapping relationship between the phase value and the spatial depth can be calibrated.

[0086] In one embodiment, the reflection data is the single-line laser image collected by the camera. Specifically, if it is a fixed-line mode, the mirror can be controlled to rotate to a certain fixed angle θ x , and the laser is kept on. Let the laser line be in the field of view of the 3D camera, and the 3D camera continuously collects the single-line laser images of the scene. Each single-line laser image can be immediately used for subsequent three-dimensional point cloud resolution.

[0087] Among them, as an example, the calculation of the reflection data to obtain three-dimensional point cloud data may include the following sub-steps:

[0088] S51, after the camera performs dedistortion and epipolar correction on the sub-pixel centerline coordinates of the laser line projected by the reflector, searching for corresponding matching points according to the centerline of the sub-pixel centerline coordinates.

[0089] S52, after determining the coordinates of the matching point in the normalized coordinate system, construct a normalized coordinate point equation with the single-line laser image and the coordinates of the matching point and solve the equation to obtain three-dimensional point cloud data.

[0090] In one operation mode, the 3D camera can be a 3D camera of a binocular camera. In fixed line scanning mode, the 3D camera of the binocular camera collects a fixed position laser line projected by a reflector. How to use a sub-pixel center line extraction algorithm to extract the sub-pixel center line coordinates of the laser line in the left and right cameras. Then, according to the internal and external parameters of the 3D camera of the binocular camera, the extracted center line sub-pixel coordinates are dedistorted and epipolar corrected.

[0091] Among them, the distortion formula can be shown as follows:

[0092]

[0093] in [x distorted ,y distorted ] represents the normalized coordinates after distortion,

[0094] [x cn ,y cn ] represents the normalized coordinate without distortion. After the epipolar correction, for each centerline point of the left camera, according to the principle of epipolar alignment, a search is performed on the centerline obtained by the right camera to find the closest matching point of the centerline of the right camera.

[0095] After determining the sub-pixel matching relationship of the center lines of the left and right cameras, the coordinates of the matching point pairs in the normalized coordinate system can be obtained. Combine the 3D point of the single-line laser image of the 3D camera of the binocular camera to the normalized coordinate point equation:

[0096]

[0097] in Represents the rotation matrix R i The j-row and k-column element of Represents the translation vector T i The jth element of . By solving the least squares solution of the equation system, the corresponding 3D point coordinates P are obtained. w (X w ,Y w ,Z w ). After traversing the sub-pixel centerline coordinates of the left camera, the fixed single-line point cloud can be obtained.

[0098] In one embodiment, the reflection data is a moving image of continuous laser lines projected during the scanning of the mirror by the camera.

[0099] When using the swinging line scan mode, the mirror can be controlled to swing within the range of Δθ angle while keeping the laser constantly on. The swinging laser line range is placed within the field of view of a monocular camera or a 3D camera of a binocular camera. The 3D camera continuously captures images of the swinging laser line in the scene to obtain a moving image.

[0100] For example, within the time of the entire scanning period Δt, the camera is controlled to take pictures at a speed of n frames per second. Starting from time t 0 moment, every interval of seconds, the camera will capture and collect the scanned laser line once. After the entire scanning process, at time t 0 +Δt, Δt×n laser maps covering the entire scene can be obtained, and multiple moving images can be obtained.

[0101] Among them, as an example, the calculation of the reflection data to obtain three-dimensional point cloud data may include the following sub-steps:

[0102] S61. Obtain the single-line point cloud data of the moving image at each scanning position, where the single-line point cloud data is obtained by constructing an equation using the point cloud data of the moving image and solving the equation.

[0103] S62. Merge the single-line point clouds at each scanning position to obtain three-dimensional point cloud data.

[0104] In one operation mode, the 3D camera can be a 3D camera of a binocular camera. In the swinging line scan mode, the 3D camera of the binocular camera captures a moving image of continuous laser lines projected from left to right by the mirror. Since the projection and acquisition system ensures high-precision synchronization, when the laser line sweeps to a certain position, the corresponding left and right two images captured by the binocular camera can be determined.

[0105] Then, the point cloud data of each moving image can be calculated respectively according to the calculation method of the above fixed line mode to obtain the single-line point cloud data at this position. After the single-line point cloud data of all the scanned images is completed, all the single-line point cloud data is merged together to obtain the complete three-dimensional point cloud of the entire scanning scene.

[0106] In one embodiment, the reflected data is multiple frame structured light images captured by a camera. When using the frame structured light mode, by controlling the galvanometer to swing at high speed within the field of view and synchronously controlling the laser to turn on or off, various stripe structured light patterns can be projected. The start exposure times of the monocular and binocular cameras are also synchronized with the projection. When the laser reflected by the galvanometer starts to rotate into the field of view, the camera starts to expose. When the laser reflected by the galvanometer moves out of the field of view, the camera ends the exposure. This can ensure that the camera can capture the complete frame structured light pattern projected each time.

[0107] For example, if it is necessary to project and collect multiple frame structured light stripes, the scanning and projection time for each pattern is Δt. Using the above stripe pattern projection method of the frame structured light mode, it is only necessary to synchronously start the camera exposure at the moment t n when the projection of the nth pattern starts. The exposure time is the same as the scanning and projection, both being Δt, and the exposure ends at the moment t n +Δt. By analogy, after nΔt time, all the images of the projected frame structured light patterns can be obtained.

[0108] Among them, as an example, the calculation of the reflected data to obtain three-dimensional point cloud data may include the following sub-steps:

[0109] S71. Calculate the wrapped phase of the phase-shifted stripes of the frame structured light image and unwrap the wrapped phase into the absolute phase.

[0110] S72. After de-distorting and epipolar rectifying the phase map corresponding to the absolute phase using the internal and external parameters of the camera, search for phase matching pairs.

[0111] S73. Construct a normalized coordinate point equation using the coordinates of the phase matching pairs and solve it to obtain three-dimensional point cloud data.

[0112] In one operation mode, the 3D camera can be a 3D camera of a binocular camera. In the frame structured light mode, the binocular camera captures multiple frame structured light patterns projected by the mirror, which can include but are not limited to structured light patterns such as Gray code + phase-shifted stripes, complementary Gray code + phase-shifted stripes, XOR Gray code + phase-shifted stripes, multi-frequency heterodyne stripes, etc. After the left camera and the right camera capture the frame structured light patterns, the wrapped phase φ of the frame structured light image can be calculated first using the phase-shifted stripes, and then the wrapped phase can be unwrapped into the absolute phase □ using decoding patterns such as Gray code, so as to obtain a complete phase unwrapping result.

[0113] Among them, the calculation formula for the wrapped phase is:

[0114]

[0115] where N represents N-step phase shift, In It represents the gray value of the phase-shifted pixels at the nth step. After obtaining the order k by using a decoding pattern such as a Gray code, the absolute phase can be expressed as: □ = φ + 2kπ.

[0116] After obtaining the phase unwrapping results of the left camera and the right camera, the internal and external parameters of the binocular camera are used to undistort and rectify the epipolar lines of the phase maps of the left camera and the right camera. Then, the epipolar line rectified phase results of the phase map of the left camera are traversed, and for each pixel in the epipolar line rectified phase of the phase map of the right camera, the nearest phase search on the epipolar line is performed to find the pixel position in the right image with the closest phase, forming a phase matching pair.

[0117] Then, the 3D point corresponding to this phase matching pair can be solved by using the simultaneous equations of the binocular camera (the simultaneous equations are the same as those of the above fixed line pattern). After traversing all pixel matching pairs, a complete area array structured light point cloud can be obtained.

[0118] In actual operation, the control unit may include the following steps:

[0119] In the first step, the mirror is precisely controlled to achieve fixed line, swinging line scan, and area array structured light projection.

[0120] In the second step, a monocular camera or a binocular camera is called to cooperate with the mirror projection to obtain the structured light projection image of the line or surface of the scene.

[0121] In the third step, the structured light pattern of the line or surface is solved, matched, and three-dimensional imaging calculation is performed to obtain the three-dimensional point cloud results of the line or surface.

[0122] This application can achieve three-dimensional imaging based on the principles of fixed line scan, swinging line scan, and area array structured light, and can switch modes at any time, realizing three modes in one machine. It solves the problem that in the using process, a single system cannot flexibly obtain the three-dimensional imaging results of the line or surface. In terms of imaging stability and speed efficiency, the imaging mode can be flexibly switched according to specific requirements.

[0123] In this embodiment, the embodiment of the present application provides a method for obtaining point cloud based on a 3D camera, and its beneficial effects are as follows: The present application can control the laser to emit laser to the reflector according to different imaging modes, so that the reflector projects the laser onto the object to be measured; call the camera to collect the reflected laser of the object to be measured to obtain reflection data, and calculate the reflection data to obtain three-dimensional point cloud data. The present application can control the laser and the galvanometer to work together according to different modes, without the need for multiple scanning processes, so as to reduce the number of processes, improve the processing speed, and thus improve the processing efficiency; moreover, the present application uses a laser to emit laser, and the laser has strong anti-high reflection, anti-multiple reflection and anti-strong light capabilities, which can reduce the noise of the reflection data and improve the data accuracy; in addition, no additional light source device is required throughout the process, which can reduce the processing cost and does not require manual adjustment. By controlling the laser and the galvanometer to work together, cameras with different 3D imaging principles can be integrated into the same set of equipment, enabling the 3D camera to take into account the characteristics of line scanning and area array imaging, so as to further improve the flexibility of imaging.

[0124] The embodiment of the present application also provides a device for obtaining point cloud based on a 3D camera. Referring to Figure 2 , a schematic structural diagram of a device for obtaining point cloud based on a 3D camera provided by an embodiment of the present application is shown.

[0125] Among them, by way of example, the device for obtaining point cloud based on a 3D camera may include:

[0126] A control module 201, configured to control the laser to emit laser to the reflector according to the imaging mode after determining the imaging mode, so that the reflector projects the laser onto the object to be measured, and the imaging modes include: fixed line mode, swinging line scan mode, and area array structured light projection mode;

[0127] An acquisition module 202, configured to call the camera to collect the reflected laser of the object to be measured to obtain reflection data, and calculate the reflection data to obtain three-dimensional point cloud data.

[0128] Optionally, the control module is further configured to:

[0129] If the imaging mode is the fixed line mode, determine a fixed angle value;

[0130] After controlling the reflector to rotate to the position of the fixed angle value, control the laser to emit laser to the reflector.

[0131] Optionally, the reflection data is a single-line laser image collected by the camera;

[0132] The acquisition module is further configured to:

[0133] After the sub-pixel center line coordinates of the laser line projected by the mirror are de-distorted and epipolar corrected by the camera, corresponding matching points are searched according to the center line of the sub-pixel center line coordinates;

[0134] After determining the coordinates of the matching points in the normalized coordinate system, a normalized coordinate point equation is constructed with the single-line laser image and the coordinates of the matching points and solved to obtain three-dimensional point cloud data.

[0135] Optionally, the control module is further configured to:

[0136] If the imaging mode is the swinging line scan mode, determine the projection angle value of the rotation of the mirror within a unit time;

[0137] Control the rotation of the mirror according to the projection angle value and control the laser to emit laser light to the mirror.

[0138] Optionally, the reflection data is a moving image of continuous laser lines projected during the scanning of the mirror by the camera;

[0139] The acquisition module is further configured to:

[0140] Acquire the single-line point cloud data of the moving image at each scanning position, where the single-line point cloud data is obtained by constructing an equation with the point cloud data of the moving image and solving it to obtain point cloud data;

[0141] Merge the single-line point clouds at each scanning position to obtain three-dimensional point cloud data.

[0142] Optionally, the control module is further configured to:

[0143] If the imaging mode is the area structured light projection mode, control the swinging of the mirror according to a preset field of view range;

[0144] Control the laser to intermittently emit laser light to the mirror.

[0145] Optionally, the reflection data is multiple area structured light images collected by the camera;

[0146] The acquisition module is further configured to:

[0147] Calculate the wrapped phase of the phase-shifted fringes of the area structured light image and expand the wrapped phase into an absolute phase image;

[0148] After de-distorting and epipolar correcting the phase map corresponding to the absolute phase using the internal and external parameters of the camera, search for phase matching pairs;

[0149] Construct a normalized coordinate point equation using the coordinates of the phase matching pair and solve it to obtain three-dimensional point cloud data.

[0150] An embodiment of the present application also provides a point cloud acquisition system based on a 3D camera. Referring to Figure 3 , a schematic structural diagram of a point cloud acquisition system based on a 3D camera provided by an embodiment of the present application is shown.

[0151] Among them, by way of example, the point cloud acquisition system based on a 3D camera may include: a 3D camera and a control unit applicable to the above-described embodiment;

[0152] The control unit is used to control the mirror of the 3D camera to be fixed or rotated; control the laser of the 3D camera to emit laser light so that the mirror sends it to the object to be measured; and control the camera of the 3D camera to collect the reflection data of the laser light reflected by the object to be measured, so as to calculate the point cloud data through the reflection data.

[0153] Among them, the laser can be a single-line laser, and the mirror can be a single-axis mechanical galvanometer, a rotating mirror or a MEMS galvanometer. The single-line laser is hit near the axis of rotation of the mirror surface, and by controlling the rotation angle of the mirror surface, the position irradiated by the laser line can be controlled. It is also possible to control the presence or absence of stripes at the irradiated position by controlling the on / off of the laser. Thus, the projection of the swinging line scan, the area array structured light mode and the fixed line scan mode is completed.

[0154] Referring to Figure 4 , a schematic structural diagram of a 3D camera provided by an embodiment of the present application is shown. In one embodiment, the 3D camera includes: a base 401 and at least one image collector 402. The base 401 is provided with a laser 403 and a movable mirror 404, and the image collector 402 is arranged on the side of the base 401.

[0155] Specifically, two image collectors 402 can be provided. Each image collector 402 can be a collector provided with a camera.

[0156] Referring to Figure 4 , the mirror 404 is provided with a galvanometer lens 405, and the galvanometer lens 405 is arranged at the central position of the base 401. The laser 403 coincides with the mirror 404 in the horizontal axis of rotation. The laser 403 can be arranged beside the galvanometer lens 405 so that the laser line of the laser 403 is almost coincident with the axis of rotation of the galvanometer lens 405.

[0157] If a monocular camera is used, the intersection points of each pixel and the phase plane projected by the mirror can be calculated through the phase unwrapping result obtained by the monocular camera. After traversing all pixels, a 3D point cloud result of the area array is obtained.

[0158] If a binocular camera is used, only the phase-unwrapping results of the left and right cameras need to be obtained. Then, by using the extrinsic parameter relationship between the left and right cameras, the epipolar relationship of each pixel in the left camera can be known. Perform phase matching search on the epipolar line, substitute the sub-pixel matching pairs obtained by the left and right cameras into the conversion equation from the 3D space points of the left and right cameras to the pixel coordinate system, and the 3D points corresponding to the matching pairs can be obtained. After traversing all the matching pairs, the planar array 3D point cloud result can be obtained.

[0159] Based on a mirror, by controlling different imaging modes, the structured light projection of the three principles of fixed line scan, swinging line scan, and planar array structured light can be completed in this application. Coupled with the controlled acquisition of a monocular or binocular camera. The structure of the monocular camera is simpler and the calculation speed is faster, but the mirror system needs to be calibrated. The binocular camera has one more camera, but only the binocular camera system needs to be calibrated, and there is no need to calibrate the mirror, and the imaging result is also more stable. The corresponding line or surface three-dimensional imaging can be realized. These three methods can be flexibly switched to complete the three-dimensional imaging functions of fixed line scan, swinging line scan, and planar array structured light under one system.

[0160] Refer to Figure 5 , which shows a computer device for a method of obtaining point cloud based on a 3D camera according to this application. Specifically, it may include the following:

[0161] The above computer device 12 is presented in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 connecting different system components (including the system memory 28 and the processing unit 16).

[0162] The bus 18 represents one or more of several types of bus 18 structures, including a memory bus 18 or a memory controller, a peripheral bus 18, a graphics acceleration port, a processor, or a local bus 18 using any bus 18 structure in a variety of bus 18 structures. For example, these architectures include, but are not limited to, Industry Standard Architecture (ISA) bus 18, Micro Channel Architecture (MAC) bus 18, Enhanced ISA bus 18, Video Electronics Standards Association (VESA) local bus 18, and Peripheral Component Interconnect (PCI) bus 18.

[0163] The computer device 12 typically includes a variety of computer system-readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0164] System memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computing device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be used for reading and writing on non-removable, non-volatile magnetic media (commonly referred to as "hard disk drive"). Although Figure 3 not shown in Figure 3 , a disk drive for reading and writing on removable non-volatile disks (such as "floppy disks"), and an optical disk drive for reading and writing on removable non-volatile optical disks (e.g., CD-ROM, DVD-ROM or other optical media) can be provided. In these cases, each drive can be connected to bus 18 through one or more data media interfaces. The memory may include at least one program product having a set (e.g., at least one) of program modules 42 that are configured to perform the functions of the embodiments of the present application.

[0165] A program / utility 40 having a set (at least one) of program modules 42 can be stored, for example, in the memory. Such program modules 42 include—but are not limited to—an operating system, one or more application programs, other program modules 42, and program data. Each or some combination of these examples may include an implementation of a network environment. The program modules 42 generally execute the functions and / or methods in the embodiments described in the present application.

[0166] The computing device 12 can also communicate with one or more external devices 14 (such as a keyboard, a pointing device, a display 24, a camera, etc.), and can also communicate with one or more devices that enable a user to interact with the computing device 12, and / or communicate with any device that enables the computing device 12 to communicate with one or more other computing devices (such as a network card, a modem, etc.). Such communication can be carried out through an input / output (I / O) interface 22. Also, the computing device 12 can communicate with one or more networks (such as a local area network (LAN)), a wide area network (WAN), and / or a public network (such as the Internet) through a network adapter 20. As shown in the figure, the network adapter 20 communicates with other modules of the computing device 12 through bus 18. It should be understood that although Figure 3 not shown in Figure 3 , other hardware and / or software modules can be used in conjunction with the computing device 12, including but not limited to: microcode, device drivers, redundant processing unit 16, external disk drive arrays, RAID systems, tape drives, and data backup storage system 34, etc.

[0167] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the method for obtaining point cloud based on a 3D camera provided in the embodiments of the present application.

[0168] That is, when the above-mentioned processing unit 16 executes the above program, it realizes:

[0169] After determining the imaging mode, controlling the laser to emit laser light to the mirror according to the imaging mode, so that the mirror projects the laser light onto the object to be measured, and the imaging modes include: fixed line mode, swinging line scan mode, and area structured light projection mode;

[0170] Invoking the camera to collect the reflected laser light of the object to be measured to obtain reflection data, and calculating the reflection data to obtain three-dimensional point cloud data.

[0171] In the embodiments of the present application, the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, it realizes the method for obtaining point cloud based on a 3D camera provided in all embodiments of the present application.

[0172] That is, when the program is executed by the processor, it realizes:

[0173] After determining the imaging mode, controlling the laser to emit laser light to the mirror according to the imaging mode, so that the mirror projects the laser light onto the object to be measured, and the imaging modes include: fixed line mode, swinging line scan mode, and area structured light projection mode;

[0174] Invoking the camera to collect the reflected laser light of the object to be measured to obtain reflection data, and calculating the reflection data to obtain three-dimensional point cloud data.

[0175] One or more arbitrary combinations of computer-readable media can be adopted. The computer-readable medium can be a computer data signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples (non-exhaustive list) of the computer-readable storage medium include: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program, and this program can be used by or in combination with an instruction execution system, apparatus, or device.

[0176] A computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which computer-readable program code is carried. Such a propagated data signal may take many forms, including - but not limited to - electromagnetic signals, optical signals, or any suitable combination of the foregoing. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device.

[0177] Computer program code for performing the operations of this application may be written in one or more programming languages or combinations thereof, including object-oriented programming languages - such as Java, Smalltalk, C++ - and also including conventional procedural programming languages - such as the "C" language or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, executed as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network - including a local area network (LAN) or a wide area network (WAN) - or, alternatively, may be connected to an external computer (e.g., through the Internet using an Internet service provider). Each embodiment in this specification is described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts among the various embodiments may be referred to each other.

[0178] Although the preferred embodiments of the embodiments of this application have been described, those skilled in the art can make additional changes and modifications once they learn the basic creative concepts. Therefore, the appended claims are intended to be construed to include the preferred embodiments and all changes and modifications falling within the scope of the embodiments of this application.

[0179] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article or terminal device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the element.

[0180] The above provides a detailed introduction to a method and system for obtaining point clouds based on a 3D camera. In this text, specific examples are used to elaborate on the principle and implementation of the present application. The description of the above embodiments is only used to help understand the method and its core idea of the present application; at the same time, for those of ordinary skill in the art, according to the idea of the present application, there will be changes in the specific implementation and application scope. In summary, the content of this specification should not be construed as a limitation to the present application.

Claims

1. A point cloud acquisition method based on a 3D camera, characterized in that: The method involves a 3D camera provided with a camera, a laser and a movable reflector, and the method comprises: After determining the imaging mode, controlling the laser to emit laser light to the reflector according to the imaging mode, so that the reflector projects the laser light onto the object to be measured, the imaging mode including: fixed line mode, swing line scanning mode and planar array structured light projection mode; The camera is called to collect reflected laser light of the object to be measured to obtain reflected data, and the reflected data is calculated to obtain three-dimensional point cloud data.

2. The point cloud acquisition method based on a 3D camera according to claim 1, characterized in that: The step of controlling the laser to emit laser light to the reflector according to the imaging mode comprises: If the imaging mode is a fixed line mode, determining a fixed angle value; After the reflector is controlled to rotate to the position of the fixed angle value, the laser is controlled to emit laser light to the reflector.

3. The point cloud acquisition method based on a 3D camera according to claim 2, characterized in that: The reflection data is a single-line laser image collected by the camera; The calculating the reflection data to obtain three-dimensional point cloud data includes: After the camera performs dedistortion and epipolar correction on the sub-pixel centerline coordinates of the laser line projected by the reflector, searching for corresponding matching points according to the centerline of the sub-pixel centerline coordinates; After the coordinates of the matching points in the normalized coordinate system are determined, a normalized coordinate point equation is constructed using the single-line laser image and the coordinates of the matching points and the equation is solved to obtain three-dimensional point cloud data.

4. The point cloud acquisition method based on a 3D camera according to claim 1, characterized in that: The step of controlling the laser to emit laser light to the reflector according to the imaging mode comprises: If the imaging mode is a swing line scan mode, determining a projection angle value of the reflector rotation per unit time; The reflector is controlled to rotate according to the projection angle value, and the laser is controlled to emit laser light to the reflector.

5. The point cloud acquisition method based on a 3D camera according to claim 4, characterized in that: The reflection data is a moving image of a continuous laser line projected by the reflector when the camera collects the moving image; The calculating the reflection data to obtain three-dimensional point cloud data includes: Acquire single-line point cloud data of the moving image at each scanning position, wherein the single-line point cloud data is obtained by constructing an equation using the point cloud data of the moving image and solving the equation; The single-line point cloud of each scanning position is merged to obtain three-dimensional point cloud data.

6. The point cloud acquisition method based on a 3D camera according to claim 1, characterized in that: The step of controlling the laser to emit laser light to the reflector according to the imaging mode comprises: If the imaging mode is a planar array structured light projection mode, the reflector is controlled to swing according to a preset field of view; The laser is controlled to intermittently emit laser light toward the reflecting mirror.

7. The point cloud acquisition method based on a 3D camera according to claim 6, characterized in that: The reflection data is a plurality of area array structured light images collected by a camera; The calculating the reflection data to obtain three-dimensional point cloud data includes: Calculate the envelope phase from the phase shift fringes of the planar array structured light image, and unfold the envelope phase into an absolute phase image; After dedistorting and performing epipolar correction on the phase image corresponding to the absolute phase using the internal and external parameters of the camera, searching for a phase matching pair; The normalized coordinate point equation is constructed using the coordinates of the phase matching pair and solved to obtain three-dimensional point cloud data.

8. A point cloud acquisition system based on a 3D camera, characterized in that: include: A 3D camera and a control unit suitable for the point cloud acquisition method based on a 3D camera as described in any one of claims 1 to 7; The control unit is used to control the reflector of the 3D camera to be fixed or rotated; control the laser of the 3D camera to emit laser so that the reflector is sent to the object to be measured; And a camera head for controlling the 3D camera to collect reflection data of the laser reflected by the object to be measured, so as to calculate point cloud data through the reflection data.

9. The point cloud acquisition system based on a 3D camera according to claim 8, characterized in that: The 3D camera comprises: a base and at least one image collector, wherein the base is provided with a laser and a movable reflector, and the image collector is arranged on the side of the base.

10. The point cloud acquisition system based on a 3D camera according to claim 8, characterized in that: The laser is a line laser, and the position of the line laser and the galvanometer lens of the reflector coincide with each other on the rotation axis in the horizontal direction.

Images