Three-view depth vision observation space construction and calibration method

Abstract

The invention discloses a three-view depth vision observation space construction and calibration method. The method comprises the following steps: constructing a depth vision observation unit based on a depth camera and a range finder; the three depth vision observation units are arranged in a mutually perpendicular configuration to form a three-view-angle depth vision observation space, and a main coordinate system is set in the three-view-angle depth vision observation space; executing camera coordinate system X-axis, Y-axis and Z-axis calibration, so that the view boundary line of the depth camera is parallel to the corresponding calibration line on the reference calibration object, and the camera coordinate system Z-axis is perpendicular to the corresponding calibration surface; an external reference calibration object is placed, a transformation matrix between coordinate systems is calculated according to the three-dimensional coordinates of the surface center point of the external reference calibration object in the main coordinate system and the camera coordinate systems, and the mapping relation between the main coordinate system and the camera coordinate systems is established. According to the method, the three-view-angle depth vision observation space is established, calibration between each view angle and the main coordinate system is completed, target surface information is extracted from three directions, and observation comprehensiveness of shielded objects with changeable attitudes is improved.

Description

Technical Field

[0001] The present invention relates to the field of visual space calibration, and in particular to a three-viewing angle depth visual observation space construction and calibration method. Background Art

[0002] In the field of computer vision, single-view target observation systems play an important role in many application scenarios, such as quality inspection of product appearance in industrial inspection; real-time detection of road conditions and obstacles in front of the vehicle in intelligent driving; observation and positioning of fruit targets in automatic picking of agricultural products and fruits and vegetables. However, when the observed object has a complex shape and posture or is occluded, the single-view target observation system may not be able to directly observe some surfaces of the object, and thus cannot extract the required position information of the measurement points on the surface of the object. For example, in the fruit picking scene, the position of the picking point on the fruit stem cannot be determined because the fruit stem is tilted or occluded. The existence of this problem makes it difficult for the single-view target observation system to ensure comprehensive acquisition of surface information of target objects in various situations.

[0003] The use of multi-view target observation can effectively increase the comprehensiveness of the surface information extraction of the target object, especially providing support for the extraction of three-dimensional coordinate information of surface points of objects with complex shapes and postures or occluded objects. The precondition for the application of multi-view target observation is to calibrate the multi-view target observation system to establish the pose mapping relationship between the coordinate systems of each perspective so that the measurement data under the relevant perspectives can be expressed accurately and uniformly. However, the calibration process of the multi-view target observation system involves the joint operation of multiple perspectives, which is difficult. Summary of the invention

[0004] In order to solve the above technical problems, the present invention provides a three-perspective depth vision observation space construction and calibration method to improve the comprehensiveness of observation of target objects with complex shapes and postures and those that are obscured, and to ensure the accuracy of the three-dimensional coordinates of the extracted surface points in the public coordinate system.

[0005] To achieve the above object, the present invention provides the following technical solution: a three-view depth vision observation space construction and calibration method, comprising the following steps:

[0006] Step 1, construct a depth vision observation unit, the depth vision observation unit includes a depth camera, a rangefinder and a base; the depth camera has a built-in camera coordinate system, which can obtain the three-dimensional coordinates of the spatial point in the field of view in the camera coordinate system, and the depth camera has been calibrated with internal parameters; the ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to install the depth camera and the rangefinder;

[0007] Step 2, arranging three depth vision observation units in a mutually perpendicular configuration in space; the depth cameras in each depth vision observation unit face the same target, and together form a three-view depth vision observation space; a principal coordinate system is also established in the three-view depth vision observation space;

[0008] Step 3: Determine the position of the depth vision observation unit:

[0009] Prepare a reference calibration object, which is a regular hexahedron; select three faces of the reference calibration object that are opposite to each depth camera and perpendicular to each other as calibration faces, and set cross calibration lines on the calibration faces; place the reference calibration object in the three-view depth vision observation space, and make the edges of the reference calibration object parallel to the corresponding axes of the principal coordinate system; move each depth vision observation unit so that it faces the corresponding calibration face, and align the measurement point of the rangefinder with the center point of the cross calibration line on the relative calibration face, so as to determine the position of each depth vision observation unit;

[0010] Step 4: Determine the posture of the deep vision observation unit:

[0011] Execute Z-axis calibration of the camera coordinate system, the Z-axis calibration process includes obtaining the normal vector of the calibration surface corresponding to the depth camera, adjusting the posture of the depth vision observation unit according to the normal vector result, so that the Z-axis of the camera coordinate system is perpendicular to the calibration surface, etc.; perform X-axis and Y-axis calibration of the camera coordinate system, rotate the depth vision observation unit around the Z-axis, so that the boundary line of the depth camera field of view is parallel to the corresponding calibration line; thereby determining the posture of the corresponding depth vision observation unit;

[0012] Step 5: Depth camera external parameter calibration:

[0013] Prepare an external parameter calibration object, which is a regular hexahedron; place the external parameter calibration object at the intersection of the measurement points of each rangefinder, and make the edges of the external parameter calibration object parallel to the corresponding axes of the principal coordinate system; measure the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to each depth camera in the principal coordinate system; use the depth camera to extract the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the camera coordinate system; according to the three-dimensional coordinate data of the center point of the surface of the external parameter calibration object in the principal coordinate system and each camera coordinate system, calculate the transformation matrix between each camera coordinate system and the principal coordinate system, establish the mapping relationship between the principal coordinate system and each camera coordinate system, and complete the external parameter calibration of the depth camera in the three-view depth vision observation space.

[0014] Furthermore, in the above-mentioned three-view depth vision observation space construction and calibration method, the center point of the cross calibration lines is located at the center of the calibration surface of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object.

[0015] Furthermore, in the above-mentioned three-view depth vision observation space construction and calibration method, the calculation process of the normal vector of the calibration surface includes: for any calibration surface, arbitrarily select three non-collinear points on it as calibration points, extract the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera, form the calibration points into an auxiliary plane, and obtain the normal vector of the auxiliary plane.

[0016] Furthermore, in the above three-view depth vision observation space construction and calibration method, the calculation formula of the normal vector of the auxiliary plane is as follows:

[0017]

[0018] Wherein x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; when the normal vector coordinate is (0, 0, C) (C is an arbitrary constant), the camera coordinate system Z axis of the depth camera is perpendicular to the calibration plane relative to the depth camera.

[0019] Furthermore, in the above-mentioned three-view depth vision observation space construction and calibration method, the three-dimensional coordinates of the center point of the surface relative to each depth camera of the external reference calibration object are measured in the principal coordinate system, and the measuring tools used are a rangefinder and a length measuring tool.

[0020] Furthermore, in the above-mentioned three-view depth vision observation space construction and calibration method, the process of measuring the three-dimensional coordinates of the center point of the surface relative to each depth camera of the external reference calibration object in the principal coordinate system includes: using three rangefinders to measure the distance from themselves to the corresponding center point of the surface of the external reference calibration object, using a length measuring tool to measure the installation position of the rangefinder in the principal coordinate system, and obtaining the three-dimensional coordinates of the center point of the surface relative to the external reference calibration object and the depth camera in the principal coordinate system through comprehensive calculation.

[0021] Furthermore, in the above-mentioned three-view depth vision observation space construction and calibration method, the process of using a depth camera to extract the three-dimensional coordinates of the center point of the surface of the external reference calibration object relative to the depth camera in the camera coordinate system includes: extracting the pixel coordinates of the center point of the surface of the external reference calibration object within the field of view of each depth camera, extracting the depth value corresponding to the pixel coordinates, and obtaining the three-dimensional coordinates of the center point of the surface in the camera coordinate system.

[0022] Furthermore, in the above three-view depth vision observation space construction and calibration method, the calculation formula of the mapping relationship between the main coordinate system and each camera coordinate system is as follows:

[0023]

[0024] Where (x, y, z) is the three-dimensional coordinate of the center point of the external reference calibration object in the principal coordinate system; (u x ,u y ,u z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the X axis of the camera coordinate system; (v x ,v y ,v z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the Y axis of the camera coordinate system; (w x ,w y ,w z ) is the unit vector of the coordinate axis in the principal coordinate system in the same direction as the Z axis of the camera coordinate system; (x0, y0, z0) are the coordinates of the origin of the camera coordinate system in the principal coordinate system; (x', y', z') are the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object in the camera coordinate system.

[0025] Compared with the prior art, the present invention has the following beneficial effects:

[0026] (1) The present invention achieves the simultaneous acquisition of RGB images and depth data information of the observed target under three viewing angles by constructing a three-viewing depth visual observation space. When the observed target has a complex shape and posture or is obscured, the present invention can improve the comprehensiveness of the observation of the target surface information.

[0027] (2) The calibration method of the three-view depth vision observation space proposed in the present invention determines the relative position relationship of each view and establishes the pose mapping relationship between the camera coordinate systems corresponding to each view, thereby ensuring that the measurement point data on the measured target under each view can be accurately and uniformly expressed in the common coordinate system. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a flow chart of a three-view depth vision observation space construction and calibration method provided by the present invention;

[0029] Figure 2 is a schematic diagram of determining the position of the depth vision observation unit of the present invention;

[0030] Figure 3 is a schematic diagram of posture determination of a depth vision observation unit of the present invention;

[0031] Figure 4 It is a schematic diagram of the depth camera extrinsic calibration of the present invention. DETAILED DESCRIPTION

[0032] To facilitate understanding of the present invention, the present invention will be described more fully below with reference to the relevant drawings.

[0033] It should be noted that the following detailed description is illustrative and is intended to provide further explanation of the present invention. The specific implementation described is only used to explain the present invention and is not intended to limit the present invention.

[0034] like Figure 1 — Figure 4 As shown, the present invention provides a three-view depth vision observation space construction and calibration method, comprising the following steps:

[0035] Step 1, construct a depth vision observation unit, the depth vision observation unit includes a depth camera, a rangefinder and a base; the depth camera has a built-in camera coordinate system, which can obtain the three-dimensional coordinates of the spatial point in the field of view in the camera coordinate system, and the depth camera has been calibrated with internal parameters; the ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to install the depth camera and the rangefinder;

[0036] Step 2, arranging three depth vision observation units in a mutually perpendicular configuration in space; the depth cameras in each depth vision observation unit face the same target, and together form a three-view depth vision observation space; a principal coordinate system is also established in the three-view depth vision observation space;

[0037] Step 3: Determine the position of the depth vision observation unit:

[0038] like Figure 2 As shown, prepare a reference calibration object, which is a regular hexahedron; select three faces of the reference calibration object that are opposite to each depth camera and perpendicular to each other as calibration surfaces, and set cross calibration lines on the calibration surface; the center point of the cross calibration lines is located at the center of the calibration surface of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object. Place the reference calibration object in the three-view depth vision observation space, and make the edges of the reference calibration object parallel to the corresponding axes of the principal coordinate system; move each depth vision observation unit so that it faces the corresponding calibration surface, and align the measurement point of the rangefinder with the center point of the cross calibration lines on the relative calibration surface, so as to determine the basic position of each depth vision observation unit;

[0039] Step 4: Determine the posture of the deep vision observation unit:

[0040] like Figure 3As shown, the Z-axis calibration of the camera coordinate system is performed, and the Z-axis calibration process includes obtaining the normal vector of the calibration surface corresponding to the depth camera, adjusting the posture of the depth vision observation unit according to the normal vector result, so that the Z-axis of the camera coordinate system is perpendicular to the calibration surface, etc.; performing the X-axis and Y-axis calibration of the camera coordinate system, rotating the depth vision observation unit around the Z-axis, so that the boundary line of the depth camera field of view is parallel to the corresponding calibration line; thereby the posture of the corresponding depth vision observation unit is determined;

[0041] In a specific implementation, the calculation process of the normal vector of the calibration surface includes: for any calibration surface, three non-collinear points are randomly selected as calibration points, the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera are extracted, the calibration points are combined into an auxiliary plane, and the normal vector of the auxiliary plane is obtained. The calculation formula of the normal vector is as follows:

[0042]

[0043] Where x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; Figure 3 As shown in the left-view camera, when the normal vector coordinate is (0, 0, C) (C is an arbitrary constant), the Z axis of the camera coordinate system of the depth camera is perpendicular to the calibration surface relative to the depth camera.

[0044] Step 5: Depth camera external parameter calibration:

[0045] like Figure 4 As shown, an external parameter calibration object is prepared, and the external parameter calibration object is a regular hexahedron; the external parameter calibration object is placed at the intersection of the measurement points of each rangefinder, and the edges of the external parameter calibration object are parallel to the corresponding axes of the principal coordinate system; the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to each depth camera in the principal coordinate system are measured; the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the camera coordinate system are extracted using the depth camera; according to the three-dimensional coordinate data of the center point of the surface of the external parameter calibration object in the principal coordinate system and each camera coordinate system, the transformation matrix between each camera coordinate system and the principal coordinate system is calculated, and the mapping relationship between the principal coordinate system and each camera coordinate system is established to complete the external parameter calibration of the depth camera in the three-view depth vision observation space.

[0046] As an optional implementation, the measuring tools used are distance meters and length measuring tools.

[0047] In the specific implementation, the measurement process of the three-dimensional coordinates of the center point of the surface relative to each depth camera of the external reference calibration object in the principal coordinate system includes: using three rangefinders to measure the distance from themselves to the corresponding center point of the surface of the external reference calibration object, using a length measuring tool to measure the installation position of the rangefinder in the principal coordinate system, and obtaining the three-dimensional coordinates of the center point of the surface relative to the external reference calibration object and the depth camera in the principal coordinate system through comprehensive calculation.

[0048] In a specific implementation, the process of using a depth camera to extract the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object relative to the depth camera in the camera coordinate system includes: extracting the pixel coordinates of the center point of the surface of the extrinsic calibration object within the field of view of each depth camera, extracting the depth value corresponding to the pixel coordinates, and obtaining the three-dimensional coordinates of the center point of the surface in the camera coordinate system.

[0049] In specific implementation, the calculation formula of the mapping relationship between the main coordinate system and each camera coordinate system is as follows:

[0050]

[0051] Where (x, y, z) is the three-dimensional coordinate of the center point of the external reference calibration object in the principal coordinate system; (u x ,u y ,u z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the X axis of the camera coordinate system; (v x ,v y ,v z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the Y axis of the camera coordinate system; (w x ,w y ,w z ) is the unit vector of the coordinate axis in the principal coordinate system in the same direction as the Z axis of the camera coordinate system; (x0, y0, z0) are the coordinates of the origin of the camera coordinate system in the principal coordinate system; (x', y', z') are the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object in the camera coordinate system.

[0052] The above is a description of the implementation steps of the present invention. Through the above description, professionals in the field can implement or use the present invention. The various modifications to these embodiments will be apparent to professionals in the field. The general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown herein, but will conform to the widest range consistent with the principles and novelties disclosed herein.

Claims

1. A three-view depth vision observation space construction and calibration method, characterized in that: The following steps are involved: Step 1, constructing a depth vision observation unit, the depth vision observation unit includes a depth camera, a rangefinder and a base; the depth camera has a built-in camera coordinate system, which can obtain the three-dimensional coordinates of the spatial point in the field of view in the camera coordinate system, and the depth camera has been calibrated with internal parameters; The ranging direction of the rangefinder is the same as the field of view direction of the depth camera; the base is used to install the depth camera and the rangefinder; Step 2, arranging three depth vision observation units in a mutually perpendicular configuration in space; the depth cameras in each of the depth vision observation units face the same target, and together form a three-view depth vision observation space; A principal coordinate system is also established in the three-view depth visual observation space; Step 3: Determine the position of the depth vision observation unit: Prepare a reference calibration object, which is a regular hexahedron; select three faces of the reference calibration object that are opposite to each depth camera and perpendicular to each other as calibration faces, and set cross calibration lines on the calibration faces; Place a reference calibration object in the three-view depth vision observation space, and make the edges of the reference calibration object parallel to the corresponding axes of the principal coordinate system; move each depth vision observation unit to face the corresponding calibration surface, and align the measurement point of the rangefinder with the center point of the cross calibration line on the relative calibration surface, so as to determine the position of each depth vision observation unit; Step 4: Determine the posture of the deep vision observation unit: Execute Z-axis calibration of the camera coordinate system, the Z-axis calibration process includes obtaining the normal vector of the calibration surface corresponding to the depth camera, adjusting the posture of the depth vision observation unit according to the normal vector result, so that the Z-axis of the camera coordinate system is perpendicular to the calibration surface, etc.; perform X-axis and Y-axis calibration of the camera coordinate system, rotate the depth vision observation unit around the Z-axis, so that the boundary line of the depth camera field of view is parallel to the corresponding calibration line; thereby determining the posture of the corresponding depth vision observation unit; Step 5: Depth camera external parameter calibration: Prepare an external parameter calibration object, which is a regular hexahedron; place the external parameter calibration object at the intersection of the measurement points of each rangefinder, and make the edges of the external parameter calibration object parallel to the corresponding axes of the principal coordinate system; measure the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to each depth camera in the principal coordinate system; use the depth camera to extract the three-dimensional coordinates of the center point of the surface of the external parameter calibration object relative to the depth camera in the camera coordinate system; according to the three-dimensional coordinate data of the center point of the surface of the external parameter calibration object in the principal coordinate system and each camera coordinate system, calculate the transformation matrix between each camera coordinate system and the principal coordinate system, establish the mapping relationship between the principal coordinate system and each camera coordinate system, and complete the external parameter calibration of the depth camera in the three-view depth vision observation space.

2. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The center point of the cross calibration lines in step 3 is located at the center of the calibration surface of the reference calibration object, and the two calibration lines are respectively parallel to the corresponding edges of the reference calibration object.

3. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The calculation process of the normal vector of the calibration surface in step 4 includes: for any calibration surface, randomly select three non-collinear points on it as calibration points, extract the three-dimensional coordinates of the calibration points in the camera coordinate system of the corresponding depth camera, form the calibration points into an auxiliary plane, and obtain the normal vector of the auxiliary plane.

4. The method for constructing and calibrating a three-view depth vision observation space according to claim 3, characterized in that: The calculation formula of the normal vector of the auxiliary plane is as follows: Wherein x1, y1, z1, x2, y2, z2, x3, y3, z3 are the coordinates of the three calibration points in the camera coordinate system; (A, B, C) is the normal vector of the auxiliary plane; when the normal vector coordinate is (0, 0, C) (C is an arbitrary constant), the camera coordinate system Z axis of the depth camera is perpendicular to the calibration plane relative to the depth camera.

5. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: In step 5, the three-dimensional coordinates of the center point of the surface of the external reference calibration object relative to each depth camera in the principal coordinate system are measured, and the measuring tools used are a rangefinder and a length measuring tool.

6. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The process of measuring the three-dimensional coordinates of the center point of the surface relative to each depth camera of the external reference calibration object in the principal coordinate system in step 5 includes: using three rangefinders to measure the distance from themselves to the corresponding center point of the surface of the external reference calibration object, using a length measuring tool to measure the installation position of the rangefinder in the principal coordinate system, and obtaining the three-dimensional coordinates of the center point of the surface relative to the external reference calibration object and the depth camera in the principal coordinate system through comprehensive calculation.

7. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The process of using a depth camera to extract the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object relative to the depth camera in the camera coordinate system in step 5 includes: extracting the pixel coordinates of the center point of the surface of the extrinsic calibration object within the field of view of each depth camera, extracting the depth value corresponding to the pixel coordinates, and obtaining the three-dimensional coordinates of the center point of the surface in the camera coordinate system.

8. The method for constructing and calibrating a three-view depth vision observation space according to claim 1, characterized in that: The calculation formula for the mapping relationship between the main coordinate system and each camera coordinate system in step 5 is as follows: Where (x, y, z) is the three-dimensional coordinate of the center point of the external reference calibration object in the principal coordinate system; (u x ,u y ,u z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the X axis of the camera coordinate system; (v x ,v y ,v z ) is the unit vector of the coordinate axis in the principal coordinate system that is in the same direction as the Y axis of the camera coordinate system; (w x ,w y ,w z ) is the unit vector of the coordinate axis in the principal coordinate system in the same direction as the Z axis of the camera coordinate system; (x0, y0, z0) are the coordinates of the origin of the camera coordinate system in the principal coordinate system; (x', y', z') are the three-dimensional coordinates of the center point of the surface of the extrinsic calibration object in the camera coordinate system.

Images