A Method for Constructing and Calibrating a Three-View Depth Vision Observation Space

Abstract

The present invention discloses a method for constructing and calibrating a three-view depth vision observation space, comprising the following steps: constructing a depth vision observation unit based on a depth camera and a rangefinder; arranging three depth vision observation units in a mutually perpendicular configuration to form a three-view depth vision observation space, and establishing a main coordinate system therein; performing calibration of the X, Y, and Z axes of the camera coordinate system to make the field-of-view boundary line of the depth camera parallel to the corresponding calibration line on the reference calibration object, and the Z axis of the camera coordinate system perpendicular to the corresponding calibration plane; placing an external parameter calibration object, and calculating the transformation matrix between coordinate systems according to the three-dimensional coordinates of the center point on the surface of the external parameter calibration object in the main coordinate system and each camera coordinate system, so as to establish the mapping relationship between the main coordinate system and each camera coordinate system. By establishing a three-view depth vision observation space and completing the calibration between each view and the main coordinate system, the present invention realizes the extraction of target surface information from three directions, and improves the comprehensiveness of the observation of objects with variable postures and occlusion.

Description

Technical Field

[0001] The present invention relates to the field of visual space calibration, and particularly to a method for constructing and calibrating a three-view depth visual observation space. Background Art

[0002] In the field of computer vision, single-view target observation systems have played an important role in various application scenarios. For example, in industrial inspection, quality inspection of product appearance; in intelligent driving, real-time detection of the road surface and obstacles in front of the vehicle; in automatic picking of agricultural products and fruits and vegetables, observation and positioning of fruit targets. However, when the object to be observed has a complex shape and posture or is occluded, the single-view target observation system may not be able to directly observe some surfaces of the object, and thus cannot extract the position information of the required object surface measurement points. For example, in the fruit picking scenario, the picking point position on the fruit stalk cannot be determined due to the inclination or occlusion of the fruit stalk. The existence of this problem makes it difficult for the single-view target observation system to ensure comprehensive acquisition of the surface information of the target object in various situations.

[0003] Adopting multi-view target observation can effectively increase the comprehensiveness of extracting surface information of the target object, and particularly provide support for extracting three-dimensional coordinate information of surface points of objects with complex shapes and postures or occluded objects. Calibrating the multi-view target observation system to establish the pose mapping relationship between the coordinate systems of each view so that the measurement data under relevant views can be accurately and uniformly expressed is a prerequisite for applying multi-view target observation. However, the calibration process of the multi-view target observation system involves joint operations of multiple views and has certain difficulties. Summary of the Invention

[0004] To solve the above technical problems, the present invention provides a method for constructing and calibrating a three-view depth visual observation space to improve the comprehensiveness of observing target objects with complex shapes and postures and those subject to occlusion, and to ensure the accuracy of the three-dimensional coordinates of the extracted surface points in the common coordinate system.

[0005] To achieve the above object, the present invention provides the following technical solutions: A method for constructing and calibrating a three-view depth visual observation space, comprising the following steps:

[0006] Step 1, construct a depth visual observation unit, where the depth visual observation unit includes a depth camera, a rangefinder, and a base; the depth camera has a built-in camera coordinate system and can obtain the three-dimensional coordinates of spatial points within its field of view in the camera coordinate system, and the depth camera has been internally calibrated; the ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to mount the depth camera and the rangefinder;

[0007] Step 2: Arrange the three depth vision observation units in a mutually perpendicular configuration in space; the depth cameras in each depth vision observation unit are directed towards the same target, jointly forming a three-view depth vision observation space; a main coordinate system is also established in the three-view depth vision observation space;

[0008] Step 3: Determination of the positions of the depth vision observation units:

[0009] Prepare a reference calibration object, and the reference calibration object is a regular hexahedron; select three mutually perpendicular faces on the reference calibration object that are opposite to each depth camera as calibration faces, and set cross calibration lines on the calibration faces; place the reference calibration object in the three-view depth vision observation space, and make each edge of the reference calibration object parallel to the corresponding axis of the main coordinate system; move each depth vision observation unit so that it faces the corresponding calibration face, and make the measurement point of the rangefinder align with the center point of the cross calibration line on the opposite calibration face, thereby determining the positions of the depth vision observation units;

[0010] Step 4: Determination of the postures of the depth vision observation units:

[0011] Perform the calibration of the Z-axis of the camera coordinate system. The Z-axis calibration process includes steps such as obtaining the normal vector of the calibration face corresponding to the depth camera, and adjusting the posture of the depth vision observation unit according to the normal vector result so that the Z-axis of the camera coordinate system is perpendicular to the calibration face; perform the calibration of the X-axis and Y-axis of the camera coordinate system, rotate the depth vision observation unit around the Z-axis so that the boundary line of the depth camera's field of view is parallel to the corresponding calibration line; thereby determining the postures of the corresponding depth vision observation units;

[0012] Step 5: Extrinsic parameter calibration of the depth camera:

[0013] Prepare an extrinsic parameter calibration object, and the extrinsic parameter calibration object is a regular hexahedron; place the extrinsic parameter calibration object at the intersection position of the measurement points of each rangefinder, and make each edge of the extrinsic parameter calibration object parallel to the corresponding axis of the main coordinate system; measure the three-dimensional coordinates of the center point of the surface of the extrinsic parameter calibration object opposite to each depth camera in the main coordinate system; use the depth camera to extract the three-dimensional coordinates of the center point of the surface of the extrinsic parameter calibration object opposite to the depth camera in the camera coordinate system; according to the three-dimensional coordinate data of the center point of the surface of the extrinsic parameter calibration object in the main coordinate system and each camera coordinate system, calculate the transformation matrix between each camera coordinate system and the main coordinate system, establish the mapping relationship between the main coordinate system and each camera coordinate system, and complete the extrinsic parameter calibration of the depth cameras in the three-view depth vision observation space.

[0014] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the center point of the cross calibration line is located at the center of the calibration face of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object.

[0015] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the calculation process of the normal vector of the calibration plane includes: for any calibration plane, take three non-collinear points on it as calibration points, extract the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera, form an auxiliary plane with the calibration points, and obtain the normal vector of the auxiliary plane.

[0016] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the calculation formula for the normal vector of the auxiliary plane is as shown in Equation (1):

[0017]

[0018] In the formula, x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; when the coordinates of the normal vector are (0, 0, C) (C is an arbitrary constant), the Z-axis of the camera coordinate system of the depth camera is perpendicular to the calibration plane opposite to the depth camera.

[0019] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the three-dimensional coordinates of the center points of the surfaces of the measurement external parameter calibration objects relative to each depth camera in the main coordinate system are measured using a rangefinder and a length measurement tool.

[0020] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the measurement process of the three-dimensional coordinates of the center points of the surfaces of the external parameter calibration objects relative to each depth camera in the main coordinate system includes: respectively using three rangefinders to measure the distances from themselves to the center points of the surfaces of the corresponding external parameter calibration objects, using a length measurement tool to measure the installation positions of the rangefinders in the main coordinate system, and obtaining the three-dimensional coordinates of the center points of the surfaces of the external parameter calibration objects relative to the depth cameras in the main coordinate system through comprehensive calculation.

[0021] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the process of using the depth camera to extract the three-dimensional coordinates of the center points of the surfaces of the external parameter calibration objects relative to the depth camera in the camera coordinate system includes: extracting the pixel coordinates of the center points of the surfaces of the external parameter calibration objects within the fields of view of each depth camera, extracting the depth values corresponding to the pixel coordinates, and obtaining the three-dimensional coordinates of the center points of the surfaces in the camera coordinate system.

[0022] Further, in the above method for constructing and calibrating the three-view depth vision observation space, the calculation formula for the mapping relationship between the main coordinate system and each camera coordinate system is as shown in Equation (2):

[0023]

[0024] In the formula, (x, y, z) are the three-dimensional coordinates of the center point of the external parameter calibration object in the main coordinate system; (u x , u y , u z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the X-axis of the camera coordinate system; (v x , v y , v z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the Y-axis of the camera coordinate system; (w x , w y , w z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the Z-axis of the camera coordinate system; (x0, y0, z0) are the origin coordinates of the camera coordinate system in the main coordinate system; (x’, y’, z’) are the three-dimensional coordinates of the center point on the surface of the external parameter calibration object in the camera coordinate system.

[0025] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0026] (1) By constructing a three-view depth vision observation space, the present invention realizes the simultaneous acquisition of RGB images and depth data information of the observed target from three perspectives. When the observed target has a complex shape and posture or is occluded, the present invention can improve the comprehensiveness of the observation of the target surface information.

[0027] (2) The calibration method of the three-view depth vision observation space proposed by the present invention determines the relative position relationship of each perspective and establishes the pose mapping relationship between the corresponding camera coordinate systems of each perspective, ensuring that the measurement point data on the measured target under each perspective can be accurately and uniformly expressed in the common coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 is a flowchart of a method for constructing and calibrating a three-view depth vision observation space provided by the present invention;

[0029] Figure 2 is a schematic diagram for determining the position of the depth vision observation unit of the present invention;

[0030] Figure 3 is a schematic diagram for determining the attitude of the depth vision observation unit of the present invention;

[0031] Figure 4 is a schematic diagram for calibrating the external parameters of the depth camera of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0032] To facilitate the understanding of the present invention, the present invention will be described more comprehensively below with reference to the relevant drawings.

[0033] It should be noted that the following detailed description is illustrative and aims to provide further explanation of the present invention. The specific embodiments described are only used to explain the present invention and are not used to limit the present invention.

[0034] As Figure 1 — Figure 4 shown, the present invention provides a method for constructing and calibrating a three-view depth vision observation space, including the following steps:

[0035] Step 1, construct a depth vision observation unit, where the depth vision observation unit includes a depth camera, a rangefinder, and a base; the depth camera has a built-in camera coordinate system and can obtain the three-dimensional coordinates of spatial points within its field of view in the camera coordinate system, and the depth camera has been internally calibrated; the ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to mount the depth camera and the rangefinder;

[0036] Step 2, arrange three depth vision observation units in space in a mutually perpendicular configuration; the depth cameras in each depth vision observation unit face the same target, jointly forming a three-view depth vision observation space; a main coordinate system is also established in the three-view depth vision observation space;

[0037] Step 3, determine the positions of the depth vision observation units:

[0038] As Figure 2 shown, prepare a reference calibration object, where the reference calibration object is a regular hexahedron; select three mutually perpendicular faces on the reference calibration object that are opposite to each depth camera as calibration faces, and set cross calibration lines on the calibration faces; the center points of the cross calibration lines are located at the centers of the calibration faces of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object. Place the reference calibration object in the three-view depth vision observation space and make each edge of the reference calibration object parallel to the corresponding axes of the main coordinate system; move each depth vision observation unit so that it faces the corresponding calibration face and make the measurement point of the rangefinder align with the center point of the cross calibration line on the opposite calibration face, thereby determining the basic positions of each depth vision observation unit;

[0039] Step 4, determine the postures of the depth vision observation units:

[0040] As Figure 3As shown, perform the calibration of the Z-axis of the camera coordinate system. The Z-axis calibration process includes steps such as obtaining the normal vector of the calibration plane corresponding to the depth camera, adjusting the pose of the depth vision observation unit according to the normal vector result, and making the Z-axis of the camera coordinate system perpendicular to the calibration plane; perform the calibration of the X-axis and Y-axis of the camera coordinate system, rotate the depth vision observation unit around the Z-axis to make the boundary line of the depth camera's field of view parallel to the corresponding calibration line; thus, the pose of the corresponding depth vision observation unit is determined.

[0041] In specific implementation, the calculation process of the normal vector of the calibration plane includes: for any calibration plane, take three non-collinear points on it as calibration points, extract the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera, form an auxiliary plane with the calibration points, and obtain the normal vector of the auxiliary plane. The calculation formula of the normal vector is as shown in Equation (1):

[0042]

[0043] In the formula, x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; as Figure 3 As shown by the left-view camera, when the normal vector coordinates are (0, 0, C) (C is an arbitrary constant), the Z-axis of the camera coordinate system of the depth camera is perpendicular to the calibration plane opposite to the depth camera.

[0044] Step 5, external parameter calibration of the depth camera:

[0045] As Figure 4 shown, prepare an external parameter calibration object, and the external parameter calibration object is a regular hexahedron; place the external parameter calibration object at the intersection position of the measurement points of each rangefinder, and make each edge of the external parameter calibration object parallel to the corresponding axis of the main coordinate system; measure the three-dimensional coordinates of the center points of the surfaces of the external parameter calibration object opposite to each depth camera in the main coordinate system; use the depth camera to extract the three-dimensional coordinates of the center points of the surfaces of the external parameter calibration object opposite to the depth camera in the camera coordinate system; according to the three-dimensional coordinate data of the center points of the external parameter calibration object surface in the main coordinate system and each camera coordinate system, calculate the transformation matrix between each camera coordinate system and the main coordinate system, and establish the mapping relationship between the main coordinate system and each camera coordinate system to complete the external parameter calibration of the depth camera in the three-view depth vision observation space.

[0046] As an optional implementation manner, the measurement tools used are rangefinders and length measurement tools.

[0047] In specific implementation, the measurement process of the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to each depth camera in the main coordinate system includes: respectively using three rangefinders to measure the distances from themselves to the center points of the corresponding surfaces of the external parameter calibration object, using a length measurement tool to measure the installation positions of the rangefinders in the main coordinate system, and obtaining the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the main coordinate system through comprehensive calculation.

[0048] In specific implementation, the process of using a depth camera to extract the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the camera coordinate system includes: extracting the pixel coordinates of the center points of the surfaces of the external parameter calibration object within the fields of view of the respective depth cameras, extracting the corresponding depth values of the pixel coordinates, and obtaining the three-dimensional coordinates of the surface center points in the camera coordinate system.

[0049] In specific implementation, the calculation formula for the mapping relationship between the main coordinate system and each camera coordinate system is as shown in Equation (2):

[0050]

[0051] In the formula, (x, y, z) are the three-dimensional coordinates of the center point of the external parameter calibration object in the main coordinate system; (u x , u y , u z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the X-axis of the camera coordinate system; (v x , v y , v z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the Y-axis of the camera coordinate system; (w x , w y , w z ) is the unit vector of the coordinate axis in the main coordinate system that is in the same direction as the Z-axis of the camera coordinate system; (x0, y0, z0) are the origin coordinates of the camera coordinate system in the main coordinate system; (x’, y’, z’) are the three-dimensional coordinates of the center point of the surface of the external parameter calibration object in the camera coordinate system.

[0052] The above is the description of the implementation steps of the present invention. Through the above description, those skilled in the art can implement or use the present invention. Various modifications to these embodiments will be obvious to those skilled in the art. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but rather will conform to the widest scope consistent with the principles and novel points disclosed herein.

Claims

1. A three-view depth vision observation space construction and calibration method, characterized in that: The following steps are involved: Step 1, constructing a depth vision observation unit, the depth vision observation unit includes a depth camera, a rangefinder and a base; the depth camera has a built-in camera coordinate system, which can obtain the three-dimensional coordinates of the spatial point in the field of view in the camera coordinate system, and the depth camera has been calibrated with internal parameters; The ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to install the depth camera and the rangefinder; Step 2, arranging three depth vision observation units in a mutually perpendicular configuration in space; the depth cameras in each of the depth vision observation units face the same target, and together form a three-view depth vision observation space; A principal coordinate system is also established in the three-view depth visual observation space; Step 3: Determine the position of the depth vision observation unit: Prepare a reference calibration object, which is a regular hexahedron; select three faces of the reference calibration object that are opposite to each depth camera and perpendicular to each other as calibration faces, and set cross calibration lines on the calibration faces; Place a reference calibration object in the three-view depth vision observation space, and make the edges of the reference calibration object parallel to the corresponding axes of the principal coordinate system; move each depth vision observation unit to face the corresponding calibration surface, and align the measurement point of the rangefinder with the center point of the cross calibration line on the relative calibration surface, so as to determine the position of each depth vision observation unit; Step 4: Determine the posture of the deep vision observation unit: Execute Z-axis calibration of the camera coordinate system, the Z-axis calibration process includes obtaining the normal vector of the calibration surface corresponding to the depth camera, adjusting the posture of the depth vision observation unit according to the normal vector result, so that the Z-axis of the camera coordinate system is perpendicular to the calibration surface, etc.; perform X-axis and Y-axis calibration of the camera coordinate system, rotate the depth vision observation unit around the Z-axis, so that the boundary line of the depth camera field of view is parallel to the corresponding calibration line; thereby determining the posture of the corresponding depth vision observation unit; Step 5: Depth camera external parameter calibration: Prepare an external parameter calibration object, which is a regular hexahedron; place the external parameter calibration object at the intersection of the measurement points of each rangefinder, and make the edges of the external parameter calibration object parallel to the corresponding axes of the principal coordinate system; measure the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to each depth camera in the principal coordinate system; use the depth camera to extract the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the camera coordinate system; according to the three-dimensional coordinate data of the center point of the surface of the external parameter calibration object in the principal coordinate system and each camera coordinate system, calculate the transformation matrix between each camera coordinate system and the principal coordinate system, establish the mapping relationship between the principal coordinate system and each camera coordinate system, and complete the external parameter calibration of the depth camera in the three-view depth vision observation space.

2. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The center point of the cross calibration lines in step 3 is located at the center of the calibration surface of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object.

3. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The calculation process of the normal vector of the calibration surface in step 4 includes: for any calibration surface, randomly select three non-collinear points on it as calibration points, extract the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera, form the calibration points into an auxiliary plane, and obtain the normal vector of the auxiliary plane.

4. The method for constructing and calibrating a three-view depth vision observation space according to claim 3, characterized in that: The calculation formula of the normal vector of the auxiliary plane is as follows: Wherein x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; when the normal vector coordinate is (0, 0, C), C is an arbitrary constant, and the Z axis of the camera coordinate system of the depth camera is perpendicular to the calibration plane relative to the depth camera.

5. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: In step 5, the three-dimensional coordinates of the center point of the surface of the external reference calibration object relative to each depth camera in the principal coordinate system are measured, and the measuring tools used are a rangefinder and a length measuring tool.

6. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The process of measuring the three-dimensional coordinates of the center point of the surface relative to each depth camera of the external reference calibration object in the principal coordinate system in step 5 includes: using three rangefinders to measure the distance from themselves to the corresponding center point of the surface of the external reference calibration object, using a length measuring tool to measure the installation position of the rangefinder in the principal coordinate system, and obtaining the three-dimensional coordinates of the center point of the surface relative to the external reference calibration object and the depth camera in the principal coordinate system through comprehensive calculation.

7. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The process of using a depth camera to extract the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object relative to the depth camera in the camera coordinate system in step 5 includes: extracting the pixel coordinates of the center point of the surface of the extrinsic calibration object within the field of view of each depth camera, extracting the depth value corresponding to the pixel coordinates, and obtaining the three-dimensional coordinates of the center point of the surface in the camera coordinate system.

8. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The calculation formula for the mapping relationship between the main coordinate system and each camera coordinate system in step 5 is as follows: Where (x, y, z) is the three-dimensional coordinate of the center point of the external reference calibration object in the principal coordinate system; (u x ,u y ,u z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the X axis of the camera coordinate system; (v x ,v y ,v z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the Y axis of the camera coordinate system; (w x ,w y ,w z ) is the unit vector of the coordinate axis in the principal coordinate system in the same direction as the Z axis of the camera coordinate system; (x0, y0, z0) are the coordinates of the origin of the camera coordinate system in the principal coordinate system; (x', y', z') are the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object in the camera coordinate system.

Images