Method for creating 3D texture lookup table of mine underground looking-around system

Abstract

The invention relates to the technical field of surround view splicing of mine underground AI equipment, in particular to a method for creating a 3D texture lookup table of a mine underground surround view system. According to the method, internal parameters and external parameters of a camera are calibrated firstly, then 3D coordinate points on a bowl model are directly mapped to a 2D plane, then the 2D points are used, the image splicing principle is combined, deviation values of the points are used, a new projection matrix is regenerated for the points, the projection matrix is used for searching texture coordinates, and therefore texture mapping is achieved.

Description

Technical Field

[0001] The present invention relates to the technical field of surround view splicing of AI equipment in underground mines, and in particular to a method for creating a 3D texture lookup table of a surround view system in underground mines. Background Art

[0002] Due to the harsh environment of coal mines, a large number of underground accidents occur every year. As an important part of the construction of smart mines, mining AI equipment can replace manual labor to perform high-risk simple operations in complex underground environments. For example, when trace gases such as methane and CO are often hidden underground and are not easy to detect, and when the circulation detection equipment is operating normally and manual continuous work is required, fixed AI equipment can be used in this environment to align the equipment for detection.

[0003] In the mine underground surround view system, since the stitched surround view image is a 2D image, due to the dedistortion of the fisheye lens and the projection matrix, when texture mapping is performed using the general method, the visible range of the mapped texture is very small, resulting in the visible image presented in the 3D model being severely stretched and the visible range becoming smaller, which cannot meet the use scenario of surround view 3D. Summary of the invention

[0004] In order to overcome the above-mentioned shortcomings, the purpose of the present invention is to provide a method for creating a 3D texture lookup table for an underground mine surround view system. The camera internal and external parameters are calibrated first, and then the 3D coordinate point on the bowl mold is directly mapped to a 2D plane. Then, this 2D point is used, combined with the principle of image stitching, and the offset value of the point is used to regenerate a new projection matrix for this point for texture coordinate search, thereby realizing texture mapping.

[0005] The technical solution of the present invention to solve the technical problem is: A method for creating a 3D texture lookup table for an underground mine surround view system, comprising the following steps: Step S1, using four fisheye cameras to collect the calibration plate image; Step S2, calibrating the camera internal parameters using the calibration plate image; Step S3, using four fisheye cameras to collect calibration cloth images; Step S4, calibrating the camera external parameters using the calibration cloth image; Step S5, generating a texture lookup table of the 3D model in the surround view system using the calibrated camera internal parameters and camera external parameters; Step S6: The texture lookup table presents a surround view in the surround view system.

[0006] As an improvement of the present invention, in step S5, a 3D model is generated as a bowl mold, and the 2D image of the camera is projected and offset and mapped to the coordinate points of the 3D bowl mold using a stitching technology.

[0007] As a further improvement of the present invention, in step S2, the internal parameters of the camera are calibrated using the calibration plate image, and the internal parameters of the camera include lens parameters.

[0008] As a further improvement of the present invention, in step S2, the distorted image in the calibration plate image is subjected to dedistortion processing.

[0009] As a further improvement of the present invention, in step S2, after the de-distortion process, a table or data structure is formed to record the original texture information of each pixel in the image.

[0010] As a further improvement of the present invention, in step S4, a transparency alpha value is introduced, and the alpha value is set using a weight in the overlapping part of the images, and finally the sum of the transparency of the two images is 1.0, thereby forming an image superposition with a weight.

[0011] As a further improvement of the present invention, in step S4, a new projection matrix used for stitching is regenerated according to the offset value of the point on the plane.

[0012] As a further improvement of the present invention, in step S5, the bowl mold parameters are used to generate a set of 3D coordinate points of the bowl mold.

[0013] As a further improvement of the present invention, in step S5, a virtual viewing angle is set, and the 3D data is projected onto a plane to form a 2D projection point. Combined with the calibrated coordinate setting, the coordinate offset value used to generate a new projection matrix is ​​obtained. After the new projection matrix is ​​obtained, the texture table of the secondary point is directly obtained from the original image, and the texture table is merged into the texture mapping table for use in OpenGl texture mapping.

[0014] In the present invention, the camera's internal and external parameters are calibrated first, and then the 3D coordinate point on the bowl mold is directly mapped to a 2D plane. This 2D point is then used, in combination with the principle of image stitching, to use the offset value of the point to regenerate a new projection matrix for the point, which is used to search for texture coordinates, thereby achieving texture mapping. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] For ease of explanation, the present invention is described in detail with reference to the following preferred embodiments and the accompanying drawings.

[0016] Figure 1 A flowchart of the method for creating a 3D texture lookup table for an underground mine surround view system of the present invention; Figure 2It is a schematic diagram of converting 3D coordinate values ​​into planar 2D coordinate values ​​in the present invention. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0018] like Figure 1 and Figure 2 As shown, a method for creating a 3D texture lookup table for an underground mine surround view system of the present invention comprises the following steps: Step S1, using four fisheye cameras to collect the calibration plate image; Step S2, calibrating the camera internal parameters using the calibration plate image; Step S3, using four fisheye cameras to collect calibration cloth images; Step S4, calibrating the camera external parameters using the calibration cloth image; Step S5, generating a texture lookup table of the 3D model in the surround view system using the calibrated camera internal parameters and camera external parameters; Step S6: The texture lookup table presents a surround view in the surround view system.

[0019] Among them, in step S2, the internal parameters of the camera are calibrated using the calibration plate image, and the internal parameters of the camera include lens parameters; then, the distorted image in the calibration plate image is dedistorted; then, after the dedistortion process, a table or data structure of the original texture information of each pixel in the image is recorded; specifically, the specific contents of the internal parameter calibration, lens parameters, dedistortion and dedistorted texture table are as follows: 1. Internal reference calibration Intrinsic calibration refers to the process of calibrating the internal parameters of a camera or other sensor through tools such as calibration plates. The internal parameters usually include the focal length of the camera, the coordinates of the principal point (optical center), the distortion coefficient, etc. These parameters describe the internal characteristics of the camera and are very important for accurate image measurement and processing. The main purpose of intrinsic calibration is to improve the accuracy and reliability of the image and reduce errors in image processing. Through calibration, the internal parameters of the camera can be determined, so that the position, size, shape and other information of the objects in the image can be calculated more accurately.

[0020] 2. Lens parameters Lens parameters usually include the following aspects: 1. Focal length: Focal length is one of the main performances of a photographic lens. It refers to the distance from the optical center of the lens to the focal plane (film or image sensor CCD or CMOS). It is expressed as f and its unit is mm. The focal length of the lens is inversely proportional to the angle of view. The longer the focal length, the smaller the angle of view.

[0021] 2. Field of view: Field of view refers to the range of images that the lens can see and record.

[0022] 3. Lens aperture (aperture): Generally speaking, the larger the aperture, the better. A large aperture helps increase the shutter speed, reduce shake in dim light, achieve a smaller depth of field, highlight the subject, and blur the background.

[0023] 4. Focal length conversion factor: the relationship between focal length, viewing angle and picture frame.

[0024] 5. Optical zoom: Optical zoom is the English name of optical zoom. Digital cameras rely on the optical lens structure to achieve zoom. Digital cameras zoom in and out the scene to be photographed by moving the lens. The larger the optical zoom ratio, the farther the scene can be photographed.

[0025] 6. Digital zoom: Digital zoom is actually the electronic enlargement of the picture, which uses "interpolation" to enlarge part of the pixels on the original CCD image sensor.

[0026] 7. Anti-shake function: optical image stabilization and electronic image stabilization.

[0027] 8. Total track length: The total track length of the lens is divided into the total optical length and the total mechanical length. The total optical length refers to the distance from the first surface of the lens in the lens to the image plane, and the total mechanical length refers to the distance from the end face of the lens barrel to the image plane.

[0028] 9. Optical back focal length: the distance from the last side of the lens in the optical system to the image plane.

[0029] 10. Optical front focal length: The distance from the first surface of the lens in the optical system to the object plane, which should be distinguished from the mechanical back focal length FFL.

[0030] 11. Flange focal length: the distance from the last mechanism surface of the lens group to the image plane.

[0031] 12. Field of view: refers to the maximum field of view that the lens can capture. The field of view can be divided into diagonal field of view (FOV-D), horizontal field of view (FOV-H), and vertical field of view (FOV-V). The field of view we usually refer to is the diagonal field of view of the digital camera module.

[0032] 13. Focal number (relative aperture) (f-number): The ratio of the effective focal length to the entrance pupil diameter, used to determine the brightness of the lens.

[0033] 14. Distortion: Distortion is the off-axis phase difference, and it is the aberration of the off-axis beam of light. It is the height difference between the intersection of the off-axis point and the principal ray on the image plane and the ideal (paraxial) image plane. It is a function of the field of view and has nothing to do with the aperture. The primary distortion varies with the cube of the field of view (image height), while the percentage distortion increases with the square of the image height, so the distortion varies with different fields of view.

[0034] 15. Relative illumination: The brightness of an object or illuminated surface when illuminated by a light source is called illumination. Relative illumination is the ratio of central illumination to peripheral illumination.

[0035] 3. De-distortion Dedistortion refers to the processing of distorted images to restore their original shapes and geometric relationships.

[0036] 4. Texture table after dedistortion The dedistorted texture table refers to a table or data structure that records the original texture information of each pixel in the image after dedistortion. In image processing, dedistortion is to correct image deformation caused by factors such as lens distortion to make the image more realistic and accurate. After dedistortion, the pixel position in the image may change, so a texture table is needed to record the original texture information of each pixel so that the texture can be correctly restored or applied in subsequent processing. The texture table is usually a two-dimensional array, in which each element corresponds to a pixel in the image. The value of the element can be the color, brightness, grayscale and other information of the texture. By querying the texture table, the original texture information of each pixel can be obtained and applied to the dedistorted image to restore the real texture of the image.

[0037] In the present invention, in step S4, the transparency alpha value is introduced, and the alpha value is set using weights in the overlapping part of the image, and finally the transparency of the two images is 1.0, thereby forming a weighted image superposition; according to the offset value of the point on the plane, the projection matrix used for the new splicing is regenerated; specifically, the external parameter calibration is the process of determining the position and posture of the camera or other sensor relative to an external reference system. The measurement results of the sensor can be accurately associated with the coordinate system of the external world; the specific key steps are as follows: 1. Select appropriate calibration targets or landmarks, which usually have clear geometric features or known coordinates.

[0038] 2. Acquire images or data containing calibration targets at different positions and angles.

[0039] 3. Extract feature points or information related to the calibration target from the image or data.

[0040] 4. Through mathematical calculations and optimization algorithms, the external parameter information such as the translation vector and rotation matrix of the sensor relative to the external reference system is solved.

[0041] Specific application scenarios (as follows): 1. In the field of AI devices, it is used for precise positioning and interaction between AI devices and the environment.

[0042] 2. In autonomous driving, determine the relative position relationship between the on-board sensors and the road, other vehicles, etc.

[0043] 3. In 3D reconstruction, multiple images from different perspectives are accurately fused.

[0044] In practical applications, external parameter calibration can accurately determine the relative position and posture of the end effector of the AI ​​device and a fixed point on the workbench, thereby achieving precise grasping and operation; or in an augmented reality system, external parameter calibration can accurately place virtual objects at specific locations in the real scene to achieve a realistic fusion effect. The accuracy of external parameter calibration is crucial to the successful implementation of many applications.

[0045] In the present invention, stitching is specifically as follows: introducing a transparency alpha value into a lookup table, and setting this value using a weight in the overlapping portion of the images, so that the final transparency sum of the two images is 1.0, thereby forming a weighted image overlay.

[0046] In the present invention, generating a new projection matrix is ​​to regenerate a new projection matrix used for splicing according to the offset value of the point on the plane. The projection matrix is ​​as follows:

[0047] Moreover, the projection matrix is ​​an opencv format matrix, as shown below: project_matrix: !!opencv-matrix rows: 3 cols: 3 dt:d data: [ -5.8157421548703270e-01, -2.3783902425382557e+00, 7.9289866250759269e+02, -1.2070917822703488e-01, -2.2993309282279020e+00, 6.2401924962770136e+02, -2.5315221059651780e-04, -4.0244826334525745e-03, 1. ] The offset coordinates are in a two-dimensional coordinate system, that is, the pixel coordinate point value after being projected onto a 2D plane, for example: (256, 784).

[0048] In the present invention, in step S5, a 3D model is generated as a bowl mold, and the 2D image of the camera is projected and offset and mapped to the coordinate points of the 3D bowl mold using stitching technology; the bowl mold parameters are used to generate a set of 3D coordinate points of the bowl mold; a virtual viewing angle is set, and the 3D data is projected onto a plane to form 2D projection points. Combined with the calibrated coordinate settings, the coordinate offset values ​​used to generate a new projection matrix are obtained. After the new projection matrix is ​​obtained, the texture table of the secondary point is directly obtained from the original image, and the texture table is merged into the texture mapping table for use in OpenGl texture mapping; specifically, a 3D bowl mold is created using the bowl mold parameters to generate a set of 3D coordinate points of the bowl mold.

[0049] In the present invention, the texture lookup table is a one-to-one correspondence between vertex coordinates and image texture coordinates. The coordinate system of the 3D bowl mold is based on the x, y, and z values ​​formed by the 3D world coordinates, while the texture coordinates are values ​​between 0 and 1 based on the U and V directions of the image.

[0050] In the present invention, setting the virtual perspective is to set the position of the virtual camera, which generally only modifies the height in the Z direction. The height is different, and the position of the 2D point obtained by the bowl mold after projection to the plane is different. The higher the height, the smaller the xy value, and the smaller the texture range mapped to the model.

[0051] In the present invention, the 3D data is projected onto the plane using the position information of the virtual camera, and the 3D coordinate value is converted into the 2D coordinate value of the plane, which is used as the offset value for generating a new projection matrix later, and the offset value is the 2D coordinate value; the 3D coordinate value is converted into the 2D coordinate value of the plane to achieve the following Figure 2 As shown, Li and Ri form the camera's view plane, Lw and Rw are the visible world planes. When Li and Ri are fixed and the camera's placement height changes, the visible range of the actual world plane will also change. Vertex A is a 3D point on the bowl mold, and the second projection point A' is a pixel point on the 2D image. In this way, this method can be used to search for the texture of the points on the bowl mold one by one, thereby realizing image mapping.

[0052] In the present invention, using 2D points to generate projection offset coordinates is to use the 2D points projected onto the plane, combined with calibrated coordinate settings, to obtain coordinate offset values ​​for generating a new projection matrix.

[0053] In the present invention, the texture table of the projection point 1X1 is generated by directly obtaining the texture UV value of the secondary point from the original image after obtaining the new projection matrix. In order to improve efficiency, the secondary texture table specification uses a size of 1X1.

[0054] In the present invention, merging the texture mapping table means merging the 1x1 texture table into the texture mapping master table for subsequent use in OpenGL texture mapping.

[0055] In the present invention, the purpose of OpenGL rendering is to draw the mapped texture onto the 3D bowl mold, thereby realizing the presentation of 3D effect, which is the process of using the 3D lookup table; the ultimate function of OpenGL rendering is to draw both 2D and 3D stereoscopic effects with real-time camera plane data.

[0056] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for creating a 3D texture lookup table for an underground mine surround view system, characterized in that: The steps include: Step S1, using four fisheye cameras to collect the calibration plate image; Step S2, calibrating the camera internal parameters using the calibration plate image; Step S3, using four fisheye cameras to collect calibration cloth images; Step S4, calibrating the camera external parameters using the calibration cloth image; Step S5, generating a texture lookup table of the 3D model in the surround view system using the calibrated camera internal parameters and camera external parameters; Step S6: The texture lookup table presents a surround view in the surround view system.

2. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 1, characterized in that: In step S5, a 3D model is generated as a bowl mold, and the 2D image of the camera is projected and offset and mapped to the coordinate points of the 3D bowl mold using a stitching technology.

3. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 1, characterized in that: In step S2, the internal parameters of the camera are calibrated using the calibration plate image, and the internal parameters of the camera include lens parameters.

4. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 3, characterized in that: In step S2, the distorted image in the calibration plate image is dedistorted.

5. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 4, characterized in that: In step S2, after the de-distortion process, a table or data structure is created to record the original texture information of each pixel in the image.

6. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 2, characterized in that: In step S4, a transparency alpha value is introduced, and the alpha value is set using a weight in the overlapping portion of the images, and finally the sum of the transparency of the two images is 1.0, thereby forming an image superposition with a weight.

7. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 6, characterized in that: In step S4, a new projection matrix used for stitching is regenerated according to the offset value of the point on the plane.

8. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 7, characterized in that: In step S5, the bowl mold parameters are used to generate a set of 3D coordinate points of the bowl mold.

9. The method for creating a 3D texture lookup table for an underground mine surround view system according to claim 8, characterized in that: In step S5, a virtual viewing angle is set, and the 3D data is projected onto a plane to form a 2D projection point. Combined with the calibrated coordinate setting, the coordinate offset value used to generate a new projection matrix is ​​obtained. After the new projection matrix is ​​obtained, the texture table of the secondary point is directly obtained from the original image, and the texture table is merged into the texture mapping table for use in OpenGl texture mapping.

Images