Mining vehicle electronic fence self-calibration method, medium and equipment

Abstract

The invention discloses a mining vehicle electronic fence self-calibration method, a medium and equipment, a 360-degree camera system is used for carrying out omnibearing calibration on a mine car, an internal reference matrix, a distortion coefficient and a projection matrix of a camera are obtained, then image splicing and display in a 360-degree range around a car body are realized, and finally an electronic fence 360-degree environment view is obtained. And accurate environment perception is provided for safe driving of the mine car. According to the technical scheme, the problems that a traditional mine car electronic fence system depends on manual measurement and parameter presetting, and adaptability is poor are effectively solved. The invention aims to provide an automatic and high-precision calibration method so as to meet the requirements of mine cars with different sizes and different mounting positions.

Description

Technical Field

[0001] The present invention relates to the field of computer image analysis and processing, in particular to a parameter calibration technology of a vehicle-mounted camera in a mine working condition in the field of assisted driving, and specifically to a method, medium and device for self-calibration of an electronic fence for a mining vehicle. Background Art

[0002] Traditional mine car electronic fence systems mainly rely on manual measurement and preset parameters for calibration. This method has problems such as low calibration accuracy and poor adaptability, and it is difficult to meet the needs of complex and changing mining environments. Therefore, there is an urgent need for a new method that can automatically and accurately complete the calibration of mine car electronic fence systems. Summary of the invention

[0003] In response to the above problems, the present invention provides a self-calibration method, medium and equipment for the electronic fence of a mining vehicle. The mine vehicle is calibrated in all directions through a 360° camera acquisition system, and the camera's intrinsic parameter matrix, distortion coefficient and projection matrix are obtained, thereby realizing image stitching and display in a 360° range around the vehicle body, providing accurate environmental perception for the safe driving of the mine vehicle.

[0004] To achieve the above object, the present invention adopts the following technical solutions:

[0005] A method for self-calibration of an electronic fence for a mining vehicle comprises the following steps:

[0006] Step S1, building a data acquisition system based on a 360° camera and measuring the height of the camera from the ground to prepare for the subsequent acquisition, processing, and storage of target object image information;

[0007] Step S2: Use a 360° camera to capture images around the vehicle body and save the images in a file;

[0008] Step S3, calibrate the camera to obtain the camera projection matrix;

[0009] Step S4: transform the camera view, and use the projection matrix obtained in S3 to convert the captured image into a bird's-eye view;

[0010] Step S5, repeating the above steps S3-S4 to convert multiple images captured by the 360° camera into a bird's-eye view;

[0011] Step S6: perform 360° panoramic stitching on the converted multiple bird's-eye views to obtain a complete image around the vehicle body.

[0012] As a preferred embodiment of the above scheme, in step S1, the data acquisition system includes a 360° camera, a processor and a data storage module. The camera is arranged outside the vehicle cab and around the vehicle body for collecting image information of the target object. The processor is connected to the camera for processing image data. The data storage module is connected to the processor for storing the collected image data.

[0013] As a preferred embodiment of the above solution, in step S2, the image is automatically acquired by calling a camera through a QT interface.

[0014] As a preferred embodiment of the above solution, in step S3, the camera projection matrix is ​​obtained by using a corner point calibration method for the data collected by the camera.

[0015] As a preferred embodiment of the above scheme, the corner point calibration method specifically includes: acquiring a calibration plate image sequence of known size, then extracting feature points on the calibration plate, then determining the three-dimensional world coordinates of these feature points under the known size of the calibration plate, then using the camera calibration algorithm, inputting the feature points in the image and their three-dimensional coordinates, calculating the camera's intrinsic parameter matrix and extrinsic parameter matrix, and finally, evaluating the calibration results, including the calculation of the reprojection error, to verify the accuracy of the calibration.

[0016] As a preferred embodiment of the above solution, the intrinsic parameters of the camera include focal length, principal point offset, distortion parameters, and the extrinsic parameters of the camera include the position and posture of the camera.

[0017] As a preferred embodiment of the above solution, in step S4, the bird's-eye view conversion method adopts a projection transformation method.

[0018] As a preferred embodiment of the above solution, in step S5, bird's-eye views from four perspectives, namely, front, back, left and right, are repeatedly acquired.

[0019] A computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the above-mentioned mining vehicle electronic fence self-calibration method.

[0020] A computer device comprises: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned mining vehicle electronic fence self-calibration method by executing the computer instructions.

[0021] Due to the above structure, the beneficial effects of the present invention are:

[0022] The electronic fence self-calibration method obtains image information around the vehicle body based on multiple 360° cameras; then, the camera's external parameter calibration matrix and internal parameter matrix are calculated, and the camera projection matrix is ​​calculated by the chessboard calibration method to project the image into a bird's-eye view; finally, multiple bird's-eye views of different perspectives are spliced ​​to obtain the electronic fence calibration result. The technical solution of the present invention effectively overcomes the problems of manual measurement and preset parameters, poor calibration adaptability, etc. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for describing the embodiments are briefly introduced below.

[0024] Figure 1 It is a working principle diagram of the present invention;

[0025] Figure 2 It is a schematic diagram of the camera calibration board of the present invention;

[0026] Figure 3 It is a schematic diagram of the self-calibration effect of the present invention. DETAILED DESCRIPTION

[0027] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present invention.

[0028] like Figure 1 , Figure 2 As shown, this embodiment provides a method for self-calibration of an electronic fence for a mining vehicle, comprising the following steps:

[0029] Step S1: Build a data acquisition system based on a 360° camera and measure the height of the camera from the ground to prepare for the subsequent acquisition, processing, and storage of target object image information:

[0030] The data acquisition system includes a 360° camera, a processor and a data storage module. The camera is set outside the vehicle cab and around the vehicle body (four cameras are reasonably arranged around the vehicle body to achieve a 360° all-round view) to collect image information of the target object. The processor is connected to the camera to process the image data, and the data storage module is connected to the processor to save the collected image data. That is, the image information of the target object collected by the camera is transmitted to the processor in real time through a high-speed data transmission interface for processing, and is saved by the data storage module for subsequent analysis and use.

[0031] The construction of the data acquisition system must ensure that the camera can be stably fixed in the front, rear, left and right directions outside the vehicle's cab, and the camera should be installed at an appropriate height to avoid being blocked by obstacles and maintain a suitable horizontal and vertical angle. The measurement of the camera's height from the ground must be accurate to the millimeter level to ensure the accuracy of subsequent image processing.

[0032] Visual images can be obtained by training the image data collected by the camera using the Yolov5 deep learning model, and then the model is used to detect the target objects in the image to obtain the target category and bounding box information.

[0033] Step S2: Use a 360° camera to capture images around the vehicle body and save the files:

[0034] The image is acquired by calling the camera through the QT interface and automatically acquiring it. As a control platform for image acquisition, the QT interface can easily call the camera for automatic acquisition. The acquired image files need to be named and stored according to the time sequence or camera position for subsequent processing and stitching.

[0035] Step S3: calibrate the camera and obtain the camera projection matrix:

[0036] The camera projection matrix is ​​obtained from the data collected by the camera using the corner points. The calibration method is the chessboard calibration method. As the main method for camera calibration, the specific implementation process is as follows: First, prepare a calibration plate of known size and set obvious corner points on the calibration plate as feature points; then, use the 360° camera acquisition system to shoot the image sequence of the calibration plate at different positions and angles (by shooting a chessboard pattern of known size, at least 10 sets of images at different angles and positions are obtained to ensure the accuracy of the calibration results); then, extract the feature points on the calibration plate through the image processing algorithm (computer vision libraries such as OpenCV can be used to process the image, extract the corner points of the chessboard, and use the SIFT algorithm for feature matching), and determine the three-dimensional world coordinates of these feature points based on the known size of the calibration plate; finally, use the camera calibration algorithm to input the feature points in the image and their three-dimensional coordinates, and calculate the camera's intrinsic and extrinsic matrix through an iterative optimization algorithm. The intrinsic parameter matrix includes focal length, principal point offset, distortion parameters, etc., which are used to describe the internal parameters of the camera; the extrinsic parameter matrix describes the position and posture of the camera, that is, the position and orientation of the camera in three-dimensional space. The accuracy of the calibration result can be evaluated by calculating the reprojection error (verifying the image not involved in the calibration by reprojection). The smaller the reprojection error, the more accurate the calibration result.

[0037] Step S4: transform the camera view and use the projection matrix obtained in S3 to convert the captured image into a bird's-eye view:

[0038] The bird's-eye view conversion method can adopt the projection transformation method. As the main method for bird's-eye view conversion, the projection transformation method is specifically implemented as follows: first, according to the camera projection matrix obtained in step S3, the captured image is projected and transformed; then, the transformed image is interpolated by the interpolation algorithm to obtain a bird's-eye view with consistent resolution. The core of the projection transformation method is to use the camera projection matrix to map the image from a two-dimensional plane to a bird's-eye view plane in three-dimensional space, thereby realizing the perspective transformation of the image.

[0039] The undistorted image is transformed into a perspective image to generate a top-view image, and the camera installation angle is adjusted according to the top-view image.

[0040] The specific steps are as follows:

[0041] Set the required calibration related parameters to determine the projection area of ​​each camera.

[0042] According to the set parameters, calibration cloths are placed around the vehicle body to ensure that the overlapping area of ​​each calibration cloth covers the field of view of adjacent cameras.

[0043] Use the camera to capture the image of the calibration cloth in real time, calculate its field of view boundary according to the position and angle of the camera, and ensure that the calibration plate is completely within the visible range of the camera.

[0044] In the acquired image, the positions of the landmarks are manually selected. When selecting, it should be ensured that these landmarks have sufficient coverage in all overlapping areas for subsequent feature matching and projection matrix calculation.

[0045] According to the coordinates of the selected marker points, the two-dimensional coordinates of each selected marker point in the camera coordinate system are paired with its three-dimensional coordinates in the world coordinate system. The least squares method is used to solve the linear equations to estimate the projection matrix P. The projection matrix P can be expressed as follows:

[0046] P=K·[R|t]

[0047] Among them, K is the intrinsic parameter matrix of the camera, R is the rotation matrix, and t is the translation vector.

[0048] Reprojection validation is performed using images that were not used in the calibration to evaluate the accuracy of the projection matrix.

[0049] After obtaining the projection matrix, when performing perspective transformation, first perform coordinate transformation on each pixel point (x, y) and calculate its position (X, Y, Z) in the world coordinate system:

[0050]

[0051] By defining the projection plane of the top-down perspective and setting the corresponding Z value, the image is presented in a top-down manner. Specifically, by setting the height and top-down angle of the camera, the three-dimensional coordinates are projected onto a two-dimensional plane to generate a top-down image.

[0052] Step S5: Repeat the above steps S3-S4 to convert the multiple images captured by the 360° camera into a bird's-eye view:

[0053] Repeat steps S3 and S4 to obtain four bird's-eye views of the front, back, left and right perspectives, respectively. Each bird's-eye view must be accurately calibrated and transformed to ensure the accuracy of subsequent splicing.

[0054] Step S6: perform 360° panoramic stitching on the converted multiple bird's-eye views to obtain a complete image around the vehicle body:

[0055] According to the bird's-eye view and related parameters obtained in steps S1-S5, the multi-view images are stitched to obtain the final calibration result. The stitching process needs to consider the overlapping areas and stitching gaps between images, and seamless stitching is achieved through image registration algorithm and fusion algorithm. The final calibration result is a complete 360° panoramic image around the vehicle body, which can accurately reflect the environmental information around the vehicle body and provide accurate environmental perception for the safe driving of the mine car. Specifically:

[0056] The image after perspective transformation will fall into the area set in advance in step S4. There are overlapping areas between adjacent cameras. In order to solve the ghosting problem, the overlapping areas of the four corners are extracted and smoothed.

[0057] The weighted average method is used to process the overlapping area to ensure the natural transition of the stitched image. By defining the weight ω, a linear operation is performed based on the gradient between pixels in the overlapping area of ​​adjacent cameras to calculate the fused pixel value. The formula is:

[0058]

[0059] Among them, I(x,y) is the pixel value at (x,y) after fusion, I0(x,y) is the pixel value of image A at (x,y), I1(x,y) is the pixel value of image B at (x,y), ω is the weight corresponding to the pixel at (x,y), and the formula is:

[0060]

[0061] Where dI0(x,y) represents the distance from the (x,y) pixel to the contour of image A minus the overlapping area, and dI1(x,y) represents the distance from the contour of image B minus the overlapping area.

[0062] The top-view images of the four cameras are fused in adjacent directions to obtain a 360° panoramic image, thereby completing the 360° panoramic image stitching. Figure 3 shown.

[0063] This embodiment also provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to execute the above-mentioned mining vehicle electronic fence self-calibration method.

[0064] This embodiment also provides a computer device, including: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the above-mentioned mining vehicle electronic fence self-calibration method by executing the computer instructions.

[0065] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A method for self-calibration of electronic fence for mining vehicles, characterized in that: The following steps are involved: Step S1, building a data acquisition system based on a 360° camera and measuring the height of the camera from the ground to prepare for the subsequent acquisition, processing, and storage of target object image information; Step S2: Use a 360° camera to capture images around the vehicle body and save the images in a file; Step S3, calibrate the camera to obtain the camera projection matrix; Step S4: transform the camera view, and use the projection matrix obtained in S3 to convert the captured image into a bird's-eye view; Step S5, repeating the above steps S3-S4 to convert multiple images captured by the 360° camera into a bird's-eye view; Step S6: perform 360° panoramic stitching on the converted multiple bird's-eye views to obtain a complete image around the vehicle body.

2. A mining vehicle electronic fence self-calibration method according to claim 1, characterized in that: In step S1, the data acquisition system includes a 360° camera, a processor and a data storage module. The camera is arranged outside the vehicle cab and around the vehicle body for collecting image information of the target object. The processor is connected to the camera for processing image data. The data storage module is connected to the processor for storing the collected image data.

3. A mining vehicle electronic fence self-calibration method according to claim 1, characterized in that: In step S2, the image is automatically acquired by calling the camera through the QT interface.

4. A mining vehicle electronic fence self-calibration method according to claim 1, characterized in that: In step S3, the camera projection matrix is ​​obtained by using the corner point calibration method for the data collected by the camera.

5. A mining vehicle electronic fence self-calibration method according to claim 4, characterized in that: The corner point calibration method specifically includes: collecting a sequence of calibration plate images of known size, then extracting the feature points on the calibration plate, then determining the three-dimensional world coordinates of these feature points when the size of the calibration plate is known, and then using the camera calibration algorithm to input the feature points in the image and their three-dimensional coordinates, calculate the camera's intrinsic parameter matrix and extrinsic parameter matrix, and finally, evaluate the calibration results, including the calculation of the reprojection error, to verify the accuracy of the calibration.

6. A mining vehicle electronic fence self-calibration method according to claim 5, characterized in that: The camera's internal parameters include focal length, principal point offset, and distortion parameters, while the camera's external parameters include the camera's position and posture.

7. A mining vehicle electronic fence self-calibration method according to claim 1, characterized in that: In step S4, the bird's-eye view conversion method adopts a projection transformation method.

8. A mining vehicle electronic fence self-calibration method according to claim 1, characterized in that: In step S5, the bird's-eye views of the front, back, left and right perspectives are repeatedly acquired.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the mining vehicle electronic fence self-calibration method according to any one of claims 1 to 8.

10. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the mining vehicle electronic fence self-calibration method according to any one of claims 1 to 8 by executing the computer instructions.

Images