2D and 3D combined depth camera motion space range detection method

Abstract

The invention provides a 2D and 3D combined depth camera motion space range detection method, which relates to the technical field of computer vision and comprises the steps of depth camera sequence calibration, external parameter calibration, motion space range initialization and motion space range verification. Through calibration and point cloud processing of the camera, detection of a motion space range of a target object is realized. According to the method, the technical requirements of stereoscopic vision technology field implementation personnel are reduced, implementation difficulty and steps are reduced, efficiency is improved, the computing power and prop requirements are reduced, calibration and view field range determination actions are integrated, the method can be widely applied to stereoscopic vision detection and guide scenes, and through the combination of 2D and 3D technologies, the method is suitable for popularization and application. The motion space range of the depth camera can be more accurately determined, the detection error is reduced, the reliability of the detection result is improved, the complicated operation steps in the traditional detection method are simplified, the requirement on the technical level of implementation personnel is reduced, and more non-professional personnel can also carry out effective detection operation.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and more specifically, particularly relates to a 2D and 3D combined depth camera motion space range detection method. Background Art

[0002] In many application scenarios of stereoscopic vision detection and guidance, the reasonable installation of depth cameras and the accurate definition of their field of view are crucial. However, the current methods for determining whether the field of view of the depth camera meets the activity space of the detected object and the guided object have many defects.

[0003] Traditional methods usually require placing the object to be inspected and the guide object at the boundaries of each target position, or moving the camera carrier to confirm at each position whether the stereo vision camera's field of view meets the requirements. This method is not only cumbersome and inefficient, but also requires a high level of technical skills from the implementer.

[0004] In addition, existing methods often require a lot of computing resources and complex algorithm support, resulting in excessive computing power requirements and difficulty in quickly obtaining accurate results in a real-time environment. At the same time, it may also require the use of many additional tools and auxiliary equipment, increasing the cost and complexity of implementation.

[0005] Due to the limitations of the above methods, the implementation of stereo vision technology in practical applications is difficult and inefficient, which limits its wide application and development in more fields. Therefore, there is an urgent need for an innovative, efficient and easy-to-operate depth camera motion space range detection method. Summary of the invention

[0006] In order to solve the above technical problems, the present invention provides a 2D and 3D combined depth camera motion space range detection method to solve the above problems.

[0007] A 2D and 3D combined depth camera motion space range detection method comprises the following steps:

[0008] S1: Depth camera sequential calibration, calibrate multiple randomly installed cameras in the scene in order according to actual usage requirements, use the color camera detection markers provided by the depth camera, arrange the detected objects in the calibration order and save the calibration order results, and re-output the camera data stream based on the calibration order results;

[0009] S2: Depth camera external parameter calibration, based on the depth camera's color camera to detect the landmark, use the multiple aligned 3D and 2D coordinate points in the detection target to build a linear expression and solve it, obtain the position relationship of the camera relative to the landmark, and convert this position relationship to the left IR camera of the depth camera to obtain the final position relationship;

[0010] S3: Initialize the motion space range. Based on the external parameter calibration result, perform coordinate transformation on the point cloud generated by the depth camera to make it consistent with the reference coordinate system. Extract the target object in the 3D space. If the target object is not extracted, return to adjust the camera view; if the target object is extracted, initialize the up, down, left, and right motion space of the target object in the 3D space with the target object as the reference.

[0011] S4: Verify the motion space range. Obtain the 3D points corresponding to the 3D initialization motion range of the target object, project the 3D points onto the 2D plane and display them on the image, and determine whether the projection result exceeds the range for calculating the ir image. If it exceeds, adjust the camera position and re - execute the above steps; otherwise, determine that the calibration result and the initialized motion space range meet the usage requirements.

[0012] Preferably, in the sequential calibration of the depth camera, after detecting the target object, arrange them in sequence according to the calibration order and save the calibration order result.

[0013] Preferably, in the external parameter calibration of the depth camera, when calculating the position relationship between the camera and the marker based on the detected marker and its actual size, use multiple aligned 3D and 2D coordinate points in the detected target to construct a linear expression of AX = B, and then solve the expression and calculate the optimal solution.

[0014] Preferably, in the external parameter calibration of the depth camera, after obtaining the position relationship between the color camera and the target object, convert it to the left ir camera of the depth camera, and the conversion formula is Tfinal = Tmaker->color * Tcolor->leftir * Tc2w.

[0015] Preferably, in the initialization of the motion space range, perform coordinate transformation on the point cloud generated by the depth camera based on the calibrated external parameters, and the conversion relationship is PCLOUDres = PCLOUDcam * T.

[0016] Preferably, in the verification of the motion space range, the projection process of projecting the 3D points corresponding to the 3D motion range onto the 2D plane is p = K * [R.t] * P.

[0017] Preferably, in the initialization of the motion space range, if the target object is not extracted, return to S2 to adjust the camera view.

[0018] Preferably, in the verification of the motion space range, if there are 2D points in the projection result that exceed the range for calculating the ir image, adjust the camera position and re - execute the steps.

[0019] Preferably, if the verification result of the motion space range determines that the calibration result and the initialized motion space range meet the usage requirements, no further operation is required.

[0020] Preferably, the whole detection process includes steps and corresponding operations of depth camera sequential calibration, external parameter calibration, motion space range initialization and verification.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] 1. Improve detection accuracy: By combining 2D and 3D technologies, the motion space range of the depth camera can be determined more precisely, reducing detection errors and improving the reliability of detection results.

[0023] 2. Reduce implementation difficulty: Simplify the complex operation steps in traditional detection methods, reduce the requirements for the technical level of implementers, and enable more non-professionals to perform effective detection operations.

[0024] 3. Improve efficiency: Reduce unnecessary repeated operations and debugging processes, and can quickly complete detection and calibration, saving time and labor costs.

[0025] 4. Reduce computing power requirements: The calculation can be completed in real time under lower computing power conditions, without relying on high-performance computing devices, reducing hardware costs and usage thresholds.

[0026] 5. Reduce prop requirements: Only the calibration board required for calibration needs to be used, without additional auxiliary tools, reducing the preparation work and cost investment for detection.

[0027] 6. Optimize implementation steps: Combine the calibration action with the action of determining the field of view range, reduce the necessary steps in the implementation of the stereo vision camera, and further improve work efficiency.

[0028] 7. Enhance adaptability: Applicable to various different scenarios and camera installation situations, and can flexibly meet the diverse needs in practical applications.

[0029] 8. Prevent problems in advance: Effectively avoid the failure of detection or derivation caused by insufficient field of view range, and improve the stability and reliability of the application of stereo vision technology.

[0030] 9. Promote technology application: Contribute to promoting the wide application of stereo vision technology in more fields and provide technical support for the development of related industries. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a schematic diagram of the overall process of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0032] The following further describes in detail the embodiments of the present invention in conjunction with the drawings. The following embodiments are used to illustrate the present invention, but cannot be used to limit the scope of the present invention.

[0033] In the scenarios of stereo vision detection and guidance, it is crucial to accurately determine whether the field of view of the depth camera meets the activity space of the object to be detected and the guidance target. The proposed method for detecting the motion space range of a depth camera by combining 2D and 3D aims to solve the problems existing in the prior art, improve the detection efficiency and accuracy, and reduce the implementation difficulty and cost.

[0034] This detection method mainly includes four main steps: sequential calibration of the depth camera, extrinsic calibration of the depth camera, initialization of the motion space range, and verification of the motion space range. The following will provide a detailed description and implementation examples for each step.

[0035] Sequential calibration of the depth camera:

[0036] (I) Step description:

[0037] First, use the color camera built into the depth camera to detect the markers. When the target object is detected, arrange them in sequence according to the calibration order required for actual use, and save the result of this calibration order. Subsequently, re - output the camera data stream according to the calibration order result to ensure that the output order of the data stream is consistent with the calibration result.

[0038] (II) Implementation example:

[0039] Suppose on an industrial production line, multiple depth cameras need to be sequentially calibrated to detect products at different positions. Select specific markers, such as objects with specific patterns or identifiers. The color cameras of the depth cameras scan and detect these markers. For example, three markers A, B, and C are detected. According to the requirements of the actual production process, the calibration order is A, B, C. Save this order and adjust the output of the camera data stream so that subsequent data processing can be carried out in this correct order.

[0040] (III) Data processing and analysis:

[0041] In this step, it is necessary to perform real - time processing and analysis on the marker data detected by the color camera to ensure the accuracy of the calibration order. Image recognition algorithms and data analysis tools can be used to quickly judge the positions and order of the markers.

[0042] Extrinsic calibration of the depth camera:

[0043] (I) Step description:

[0044] Due to the problem of the installation angle of the camera in the actual scenario, the coordinate system of the acquired point cloud is inconsistent with the preset reference coordinate system. Therefore, it is necessary to detect the marker using the color camera based on the depth camera, and then use multiple aligned 3D and 2D coordinate points in the detected target to construct a linear expression of AX = B and solve it to calculate the positional relationship between the camera and the marker. After obtaining the positional relationship of the color camera relative to the target object, it is further transformed to the left ir camera of the depth camera to obtain the final positional relationship.

[0045] (II) Implementation example:

[0046] Taking a robot vision-guided scenario as an example, the robot needs to accurately grasp objects at different positions through the depth camera. First, detect the marker using the color camera to obtain 3D and 2D coordinate points on multiple markers, such as (x1, y1, z1, u1, v1), (x2, y2, z2, u2, v2), etc. Use these coordinate points to construct a linear expression and solve it to obtain the positional relationship between the camera and the marker, such as parameters like distance and angle. Then, according to the conversion formula Tfinal = Tmaker->color * Tcolor->leftir * Tc2w, convert the positional relationship obtained by the color camera to the left ir camera to provide accurate positional information for subsequent point cloud processing.

[0047] (III) Data processing and analysis:

[0048] In this step, a large number of coordinate point calculations and linear equation solutions are involved, and efficient mathematical calculation libraries and algorithms are required to ensure the accuracy and speed of the calculations. At the same time, perform accuracy evaluation and error analysis on the calculated positional relationship data to ensure that it meets the requirements of practical applications.

[0049] Initialization of the motion space range:

[0050] (I) Step description:

[0051] Based on the results of the extrinsic calibration of the depth camera, perform coordinate system transformation on the point cloud generated by the depth camera to make it consistent with the reference coordinate system. Then, extract the target object in the 3D space. If the target object is not extracted, return to the second step to adjust the camera view; if the target object is extracted, initialize the motion space of the target object in the up, down, left, and right directions in the 3D space with the target object as the reference.

[0052] (II) Implementation example:

[0053] In a logistics warehousing scenario, it is necessary to determine the motion space range of the goods handling area by the depth camera. According to the external parameter calibration results, the coordinate system of the point cloud obtained by the depth camera is transformed. Then, an attempt is made to extract the shape and position information of the goods in the 3D space. If the initial extraction is unsuccessful, return to adjust the installation angle or position of the camera, and re-perform the external parameter calibration and point cloud processing. Once the information of the goods is successfully extracted, centered on the goods, initialize its possible motion range in the 3D space, including the height of up and down movement, the distance of left and right translation, etc.

[0054] (III) Data processing and analysis:

[0055] In this step, a large amount of point cloud data needs to be processed and analyzed to accurately extract the information of the target object. Algorithms such as point cloud segmentation and clustering can be used to improve the accuracy of target object extraction. At the same time, a rationality assessment is carried out on the initialized motion space range to ensure that it can cover the possible motion range of the target object.

[0056] Motion space range verification:

[0057] (I) Step description:

[0058] Obtain a series of 3D points corresponding to the 3D initialized motion range of the target object, project these 3D points onto the 2D plane, and display them on the image. Determine whether the projected result exceeds the range used to calculate the ir image. If there are 2D points that exceed, it is considered that the initialized motion space range does not meet the requirements, and the position of the camera needs to be adjusted and the previous steps need to be re-executed; otherwise, it is considered that the calibration result and the initialized motion space range meet the actual use requirements.

[0059] (II) Implementation example:

[0060] Taking an automated assembly scenario as an example, after the initialization of the motion space range is completed, obtain a series of 3D points representing the motion range of the goods. Through a specific projection algorithm, such as p = K * [R.t] * P, project these 3D points onto the 2D plane and display them on the monitoring image. If it is found that some of the projected points exceed the pre-set ir image range, it means that the current camera position and motion space range settings are unreasonable, and the camera position needs to be readjusted and the previous calibration and initialization steps need to be re-performed until the projection result is completely within the reasonable range.

[0061] (III) Data processing and analysis:

[0062] In this step, it is necessary to perform rapid boundary detection and range judgment on the projected 2D image data. Image processing algorithms and boundary detection functions can be used to accurately determine whether the projection points are out of range. At the same time, record and analyze the data during multiple adjustment and verification processes to summarize experience and optimize the subsequent operation process.

[0063] Example 1:

[0064] Scene description: On an automotive parts production line, it is necessary to detect the positions and movement spaces of different models of parts on the conveyor belt to ensure that the robot can accurately grasp and assemble them.

[0065] Detection process:

[0066] Sequential calibration of depth cameras: Detect markers on different parts through a color camera, and calibrate the cameras in the order of the production process, which takes about 5 minutes.

[0067] Extrinsic calibration of depth cameras: Calculate the camera position relationship using the 3D and 2D coordinate points of the markers and convert it to the left and right cameras. The whole process takes about 8 minutes.

[0068] Initialization of the movement space range: Process the point cloud obtained by the camera, successfully extract part information, and initialize the movement space, which takes about 10 minutes.

[0069] Verification of the movement space range: Project the 3D points onto the 2D plane and judge that the results are within a reasonable range. This step takes about 3 minutes.

[0070] Effect evaluation: The accuracy rate of the robot grasping parts reaches over 98%, and the production efficiency is increased by 20%.

[0071] Example 2:

[0072] Scene description: In a large logistics warehouse, it is necessary to determine the movement space range of forklifts carrying goods to avoid collisions and improve the utilization rate of warehouse space.

[0073] Detection process:

[0074] Sequential calibration of depth cameras: Complete the sequential calibration of the cameras in about 6 minutes.

[0075] Extrinsic calibration of depth cameras: It takes about 10 minutes to calculate the camera position relationship and convert it.

[0076] Initialization of the movement space range: Successfully extract the goods information and initialize the movement space after 12 minutes.

[0077] Verification of the movement space range: Complete the verification within 4 minutes and find that some areas need to be adjusted.

[0078] Effect evaluation: The collision accidents during the handling of warehouse goods have been reduced by 80%, and the space utilization rate has been increased by 15%.

[0079] Example 3:

[0080] Scenario description: On an automated assembly line for electronic products, it is necessary to ensure that the movement space of components during the assembly process meets the requirements to improve product quality and production efficiency.

[0081] Detection process:

[0082] Sequential calibration of the depth camera: It takes about 4 minutes.

[0083] External parameter calibration of the depth camera: It takes about 7 minutes to complete the calculation and conversion.

[0084] Initialization of the movement space range: The initialization is completed within 8 minutes.

[0085] Verification of the movement space range: The verification is completed in 3 minutes, and the result meets the requirements.

[0086] Effect evaluation: The defective rate of product assembly has been reduced by 90%, and the production efficiency has been increased by 18%. Comparative analysis:

[0087]

[0088]

[0089] From the comparison of the above three examples, it can be seen that the detection method of the present invention can effectively determine the movement space range of the depth camera in different scenarios, and has achieved remarkable effects in improving production efficiency, ensuring product quality, reducing accidents, etc. Although the time required for each step in different scenarios varies, generally, the detection and optimization can be completed within an acceptable time to meet the requirements of practical applications.

[0090] The embodiments of the present invention are given for purposes of illustration and description, and are not exhaustive or limit the invention to the disclosed form. Many modifications and variations are obvious to those of ordinary skill in the art. The embodiments are selected and described to better illustrate the principles and practical applications of the present invention, and to enable those of ordinary skill in the art to understand the present invention and thus design various embodiments with various modifications suitable for specific purposes.

Claims

1. A 2D and 3D combined depth camera motion space range detection method, characterized in that: The following steps are involved: S1: Depth camera sequential calibration: calibrate multiple randomly installed cameras in the scene in order according to actual usage requirements. Use the color camera detection markers provided by the depth camera to arrange the detected objects in the calibration order and save the calibration order results. Re-output the camera data stream based on the calibration order results. S2: Depth camera external parameter calibration: Based on the depth camera's color camera for landmark detection, a linear expression is constructed and solved using multiple aligned 3D and 2D coordinate points in the detection target to obtain the position relationship of the camera relative to the landmark, and this position relationship is converted to the left IR camera of the depth camera to obtain the final position relationship; S3: Initialization of motion space range: Based on the external parameter calibration result, the coordinate system of the point cloud generated by the depth camera is transformed to make it consistent with the reference coordinate system, and the target object is extracted in the 3D space. If it is not extracted, the camera angle is adjusted. If it is extracted, the motion space of the target object in the upper, lower, left and right directions is initialized in the 3D space based on the target object; S4: Motion space range check: obtain the 3D points corresponding to the 3D initialized motion range of the target object, project the 3D points onto the 2D plane and display them on the image, and determine whether the projection result exceeds the range used to calculate the ir image. If so, adjust the camera position and re-execute the above steps. Otherwise, determine that the calibration result and the initialized motion space range meet the usage requirements.

2. The method for detecting the spatial range of a 2D and 3D combined depth camera motion according to claim 1, characterized in that: In the depth camera sequential calibration, after the target objects are detected, they are arranged in sequence according to the calibration order, and the calibration sequence results are saved.

3. The method for detecting the spatial range of a 2D and 3D combined depth camera motion according to claim 1, characterized in that: In the depth camera extrinsic calibration, when calculating the position relationship of the camera relative to the marker based on the detected marker and its actual size, a linear expression of AX=B is constructed using multiple aligned 3D and 2D coordinate points in the detected target, and then the expression is solved and the optimal solution is calculated.

4. The method for detecting the spatial range of a 2D and 3D combined depth camera motion according to claim 1, characterized in that: In the depth camera external parameter calibration, after obtaining the position relationship of the color camera relative to the target object, it is converted to the left IR camera of the depth camera. The conversion formula is: Tfinal=Tmaker->color*Tcolor->leftir*Tc2w.

5. The method for detecting the spatial range of a 2D and 3D combined depth camera motion according to claim 1, characterized in that: In the initialization of the motion space range, the point cloud generated by the depth camera is transformed into a coordinate system based on the calibrated external parameters, and the transformation relationship is: PCLOUDres = PCLOUDcam*T.

6. The method for detecting the spatial range of a 2D and 3D combined depth camera motion according to claim 1, characterized in that: In the motion space range check, the projection process of projecting the 3D points corresponding to the 3D motion range onto the 2D plane is p=K*[Rt]*P.

7. The method for detecting the spatial range of a 2D and 3D combined depth camera motion as claimed in claim 1, characterized in that: During the initialization of the motion space range, if the target object is not extracted, the process returns to S2 to adjust the camera viewing angle.

8. The method for detecting the spatial range of a 2D and 3D combined depth camera motion as claimed in claim 1, characterized in that: In the motion space range check, if there are 2D points in the projected result that are beyond the range used to calculate the ir image, adjust the camera position and re-execute the steps.

9. The method for detecting the spatial range of a 2D and 3D combined depth camera motion as claimed in claim 1, characterized in that: If the motion space range check result determines that the calibration result and the initialized motion space range meet the usage requirements, no further operation is required.

10. The method for detecting the spatial range of a 2D and 3D combined depth camera motion as claimed in claim 1, characterized in that: The entire detection process includes the steps and corresponding operations of depth camera sequential calibration, external parameter calibration, motion space range initialization and verification.

Images