Work vehicle monitoring system and method based on fusion of three-dimensional point cloud and optical image

Abstract

The application discloses a kind of based on three-dimensional point cloud and optical image fusion's operation vehicle monitoring system, it includes three-dimensional point cloud and image data acquisition module acquisition three-dimensional point cloud data and image data;Three-dimensional point cloud target detection module constructs three-dimensional point cloud detection model according to three-dimensional point cloud data, obtains three-dimensional point cloud target detection frame;Image target detection module constructs image detection model according to image data, obtains image detection frame;Camera internal and external parameter calibration module carries out internal and external parameter calibration to camera, and obtains the distortion parameter of camera;Three-dimensional point cloud and camera joint calibration module carries out calibration to the external parameter between camera and laser radar, and carries out objectification processing to three-dimensional point cloud data and image data;Three-dimensional point cloud and optical image fusion detection module three-dimensional point cloud target detection frame is projected to image coordinate system and obtains three-dimensional point cloud projection detection frame, and is fused with image detection frame.The application can improve near electric operation site operation efficiency and safety.

Description

Technical Field

[0001] This invention relates to the field of intelligent operation and maintenance technology for power transmission and transformation, specifically to a monitoring system and method for operation vehicles based on the fusion of three-dimensional point clouds and optical images. Background Technology

[0002] This invention belongs to the field of intelligent operation and maintenance of power transmission and transformation. Near-electric work is a routine operation method in power transmission and transformation projects, and its safety risk prevention is a key focus of on-site safety management. Lifting operations are high-risk operations in power transmission and transformation operation and maintenance. The safe distance between the work vehicle and live conductors is a crucial factor in preventing on-site safety risks, and the main factors affecting safe distance monitoring are the detection of the work vehicle and the live conductor. During operation, vehicles enter the live area from different paths, and due to visual interference, it is impossible to accurately locate the safe distance between the robotic arm and the nearest live conductor during the swing of the boom. When the boom enters the safe distance range for operation, electric shock may occur due to insufficient safety distance, potentially causing large-scale power outages or even personal injury. Power engineering sites are high-electromagnetic-interference environments, and commonly used ranging methods such as infrared and ultrasonic ranging often cannot be used normally. Laser ranging has the characteristics of being non-contact and highly resistant to electromagnetic interference. Therefore, real-time monitoring of the work vehicle's position through three-dimensional laser point clouds has significant practical implications. Summary of the Invention

[0003] The purpose of this invention is to provide a monitoring system and method for work vehicles based on the fusion of three-dimensional point cloud and optical image, which provides an important reference for the prevention and control of safety risks in near-electric work sites.

[0004] To achieve this objective, the present invention provides a vehicle monitoring system based on the fusion of 3D point clouds and optical images, comprising:

[0005] The 3D point cloud and image data acquisition module is used to acquire 3D point cloud data and image data of the work vehicle under study and the near-electric work site environment;

[0006] The 3D point cloud target detection module is used to construct a 3D point cloud detection model of the work vehicle based on the acquired 3D point cloud data, and to obtain the 3D point cloud target detection box of the work vehicle using the 3D point cloud detection model of the work vehicle.

[0007] The image target detection module is used to construct an image detection model of the work vehicle based on the acquired image data, and to obtain the image detection box of the work vehicle using the image detection model of the work vehicle;

[0008] The camera intrinsic and extrinsic parameter calibration module is used to calibrate the intrinsic and extrinsic parameters of the camera used to acquire image data and to obtain the camera's distortion parameters.

[0009] The 3D point cloud and camera joint calibration module is used to calibrate the external parameters between the camera and the lidar based on the camera's intrinsic and extrinsic parameters and distortion parameters, and to perform object-oriented processing on the 3D point cloud data and image data based on the external parameters between the camera and the lidar and the camera's intrinsic parameters.

[0010] The 3D point cloud and optical image fusion detection module is used to project the 3D point cloud target detection box of the work vehicle onto the image coordinate system based on the object-processed 3D point cloud data and image data to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle and the image detection box of the work vehicle are fused in the image coordinate system.

[0011] Preferably, the 3D point cloud data is divided into A% training set and B% validation set, and the image data is divided into A% training set and B% validation set.

[0012] Preferably, the method for constructing a 3D point cloud detection model of the work vehicle, obtaining the 3D point cloud target detection box of the work vehicle using the 3D point cloud target detection model, and obtaining the category, coordinate position in 3D space, and physical size information of the work vehicle from the 3D point cloud target detection box is as follows:

[0013] The 3D point cloud detection model of the work vehicle is the SE-SSD model. The data augmentation module inputs the training set of the 3D point cloud data into the SE-SSD model. By connecting the center of the ground truth bounding box with the grid vertex, the target points of the work vehicle are divided into multiple subsets, and random swapping, sparsity and loss operations are performed on the multiple subsets.

[0014] The convolution module is used to input the training set of 3D point cloud data and the training set after random swapping, sparsity and loss operations into the convolution module to extract the spatial semantic features of the work vehicle in the 3D point cloud data.

[0015] The generative adversarial module trains student and teacher network models simultaneously, and performs 3D point cloud target detection of the work vehicle based on the spatial semantic features of the work vehicle in the extracted 3D point cloud data. It outputs the category of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information.

[0016] Preferably, the method for constructing a vehicle image detection model, obtaining vehicle image detection boxes using the vehicle image detection model, and obtaining the vehicle category, center point pixel coordinates, and length and width dimensions of the vehicle image detection box from the vehicle image detection box is as follows:

[0017] The image detection model for the work vehicle is the YOLOX model. The data preprocessing module inputs the training set of work vehicle image data into the YOLOX model, performs mosaic data enhancement on the training data, and stitches the enhanced images by random scaling, random cropping, and random arrangement.

[0018] The feature extraction module is used to extract the spatial semantic features of the work vehicle from the stitched image data;

[0019] The multi-scale feature fusion module takes the spatial semantic features of the work vehicle in the extracted image data as input and fuses feature maps of different receptive fields through multiple pooling kernels.

[0020] The head network module takes the fused feature maps of different receptive fields as input, performs classification and regression operations, and obtains the category of the work vehicle, the pixel coordinates of the center point, and the length and width of the work vehicle image detection box.

[0021] Preferred methods for calibrating camera intrinsic and extrinsic parameters include: using OpenCV to collect calibration images and using MATLAB toolboxes to conduct camera calibration experiments to obtain the camera's intrinsic parameter matrix, distortion vector, and extrinsic parameter matrix;

[0022] The specific process for calibrating the camera's intrinsic and extrinsic parameters is as follows:

[0023] Let the intrinsic parameter be (f) x ,f y ,c x ,c y The distortion parameters are (k1,k2,p1,p2,k3);

[0024] Among them, f x f represents the camera's focal length in the horizontal direction. y c represents the camera's focal length in the vertical direction. x c represents the offset of the camera's optical axis in the horizontal direction of the image coordinate system. y This represents the offset of the camera's optical axis in the vertical direction of the image coordinate system; k1 is the first radial distortion parameter, k2 is the second radial distortion parameter, and k3 is the distortion correction parameter; p1 is the first tangential distortion parameter, and p2 is the second tangential distortion parameter.

[0025] Select an A×B chessboard calibration board, where the actual physical side length of each black and white square is C mm;

[0026] A script written in Python was used to control the camera to acquire images. By continuously moving the checkerboard calibration board, D images were repeatedly taken. N images with obvious changes were selected for calibration experiments. During the camera shooting process, LiDAR was used to collect corresponding point cloud data.

[0027] Open the MATLAB Camera Calibrator toolbox, import the images to be calibrated, and perform corner detection on each image; solve the intrinsic and extrinsic parameters of the camera according to Zhang Zhengyou's chessboard calibration method, and optimize the calibration parameters by adding a distortion model and using the maximum likelihood estimation method; finally, obtain the calibration results of the camera's intrinsic and extrinsic parameters.

[0028] Preferably, the extrinsic parameters between the camera and the LiDAR are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The specific method for objectifying the 3D point cloud data and image data based on the extrinsic parameters between the camera and the LiDAR and the camera's intrinsic parameters is as follows:

[0029] In the lidar coordinate system, the relative position of the object and the lidar is represented as [X...]. L Y L Z L ], where the origin is the geometric center of the lidar, X L The axis is horizontal forward, Y L The axis is horizontal to the left, Y L The axis points vertically upwards, conforming to the rules of a right-handed coordinate system; in the camera coordinate system, the relative position of the object and the camera is represented as [X...]. C Y C Z C ], where the origin is the optical center O of the camera, X C The axis is parallel to the X-axis, and the Y-axis is parallel to the X-axis. C The axis is parallel to the Y-axis, and the Z-axis is parallel to the Y-axis. C The X-axis is parallel to the camera's optical axis and perpendicular to the image plane. The image coordinate system is represented as [X,Y], where the origin is the camera's principal point, the X-axis is parallel to the u-axis, and the Y-axis is parallel to the v-axis. The pixel coordinate system is represented as [u,v], where the origin is the top left corner of the image, the u-axis is horizontal to the right, and the v-axis is vertically downward.

[0030] The transformation formula from the lidar coordinate system to the camera coordinate system is as follows:

[0031]

[0032] The transformation formula from camera coordinate system to image coordinate system is as follows, where the Z-axis of the camera coordinate system is in the same direction as the Z-axis of the image coordinate system:

[0033]

[0034] The formula for transforming from image coordinates to pixel coordinates is as follows:

[0035]

[0036] Point P in the lidar coordinate system L Its pixel projection point P IThe conversion formula between them is:

[0037]

[0038] In the formula: M1 is the intrinsic parameter matrix of the camera;

[0039] f is the length of the camera's focal length (u o v o ) represents the position coordinates of the camera center point in the image coordinate system; dX represents the physical length of a single pixel in the image plane, dY represents the physical width of a single pixel in the image plane; s represents the scaling factor from three-dimensional space to two-dimensional space, f represents the length of the camera focal length; R represents the rotation parameter matrix from the LiDAR coordinate system to the camera coordinate system; T represents the translation vector matrix from the LiDAR coordinate system to the camera coordinate system.

[0040] Let the coordinate transformation from a point in the lidar coordinate system to a point in the image coordinate system be:

[0041]

[0042] In the formula, M1 is only related to the camera's internal parameters. The camera's intrinsic parameter matrix M1 is obtained by solving the formula, and then the corresponding constraint function is established by extracting target features to solve for the extrinsic parameter matrix M2 between the camera and the lidar. This is a simplified version of the above transformation formula, where s is the scaling factor from three-dimensional space to two-dimensional space, and p... I Let P be the point L Pixel projection points;

[0043] The corner and center point normal vectors of the target in the 3D point cloud and image are extracted as features. The positions of the target corner and center point normal vectors in the lidar coordinate system and the camera coordinate system are matched. The loss function is established and minimized. Then, the rotation parameter matrix R and translation vector matrix T between the camera and lidar are solved to obtain the extrinsic parameter matrix between the camera and lidar.

[0044] Preferably, the specific method for projecting the target detection box of the 3D point cloud of the work vehicle onto the image coordinate system is as follows:

[0045] The 3D point cloud target detection module performs 3D point cloud target detection and outputs the coordinate position (x1, y1, z1) of the work vehicle in 3D space and the physical dimensions (Δx, Δy, Δz) of the work vehicle.

[0046] Therefore, the range of the work vehicle in three-dimensional space can be obtained as follows:

[0047]

[0048] In the formula: x1min Let x represent the minimum value of the target point cloud in the x-direction of 3D space. 1max The value of the target point cloud in the x-direction of 3D space is represented by y. 1min This represents the minimum value of the target point cloud in the y-direction of 3D space. 1max The z-axis represents the maximum value of the target point cloud in the 3D space along the y-direction. 1min Let z represent the minimum value of the target point cloud in the z-direction of 3D space. 1max This represents the maximum value of the target point cloud in the z-direction of three-dimensional space;

[0049] Therefore, the target detection bounding box of the 3D point cloud of the work vehicle is obtained as p. 1i =(x 1i y 1i , z 1i (i = 1, 2, ..., 8), where i represents the 8 vertices of the target detection box of the 3D point cloud of the work vehicle; based on the extrinsic parameter matrix M2 and the intrinsic parameter matrix M1 between the camera and the lidar obtained by the camera intrinsic and extrinsic parameter calibration module and the 3D point cloud and camera joint calibration module, the target detection box of the 3D point cloud of the work vehicle is projected onto the image coordinate system, and the projection of the target detection box of the 3D point cloud of the work vehicle onto the image coordinate system is p. ci =(u i v i (i = 1, 2, ..., 8) is:

[0050]

[0051] Among them, u i This indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci The value on the x-axis, v i This indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci Value on the y-axis; x 1i This represents the x-coordinates and y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system. 1i This represents the y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system, and the z-coordinates of the bounding box on the point cloud. 1i This represents the coordinates of the eight vertices of the target detection box in the 3D point cloud of the work vehicle on the z-axis of the point cloud coordinate system;

[0052] Let L be the rectangle with the largest area formed by the above 3D point cloud projection results. _BBox Let C be the bounding box detected by the YOLOX model. _BBox .

[0053] Preferably, the specific method for fusing the 3D point cloud projection detection box of the work vehicle with the image detection box of the work vehicle in the image coordinate system by calculating the intersection-union ratio and the center point distance is as follows:

[0054] S1: Let {L} be the projected bounding box of the 3D point cloud target detection box of the work vehicle. i_BBox}(i=1,2,…,m), the bounding box for the work vehicle image is {C j_BBox}(j=1,2,…,n);

[0055] S2: Calculate {L i_BBox} and {C j_BBox The intersection-union ratio (CUI) between each detection box and the pixel distance between their center points are given in the diagram. The CUI between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image is k. ij_IOU The calculation is as follows:

[0056]

[0057] Let the coordinates of the center point of the image detection box of the work vehicle be (u c v c The center point coordinates of the target detection box projection value of the 3D point cloud of the work vehicle are (u1, v1), and the center point distance D is defined as... LC for:

[0058] D LC =|u c -u1|+|v c -v1|

[0059] By k ih_IOU and D LC The calculation formula yields the intersection-union matrix (IUU) between the 3D point cloud projection detection box and the image detection box of the work vehicle:

[0060]

[0061] The center point distance matrix is:

[0062]

[0063] Among them, D mn_LC This represents the distance to the center point in the m-th row and n-th column of the center point distance matrix.

[0064] S3: Let the IOU threshold be α, and when the intersection-union ratio (IU) between the image detection box of the work vehicle and the projection value of the target detection box of the 3D point cloud of the work vehicle is k ij_IOU When ∈[0,α), it is determined that it belongs to 2 separate target objects;

[0065] S4: Let k gh_IOU =maxk,kgU_IOU This represents the intersection-union ratio (IUR) between the g-th row and the h-th column in the IUR matrix k, when k gh_IOU >α and D gh_LC When D < β, the fusion is considered successful. gh_LC Let K be the distance between the center points in the g-th row and h-th column of the center point distance matrix; take the union between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image as the new target detection box, denoted as K. i_LC_BBox_gh Meanwhile, the serial number of the 3D point cloud projection detection frame of the work vehicle that was successfully integrated this time is recorded as g. i The successfully fused detection box number for the work vehicle image is h. i Delete the g-th row and h-th column from matrix k;

[0066] S5: Repeat step S4 until no expression satisfies k in the intersection-union matrix and the center-distance matrix. gh_IOU >α and D gh_LC The value of β is used to output the fusion detection result K. LC_BBox ={K i_LC_BBox_gh}(i=1,2,…,k).

[0067] A method for monitoring work vehicles based on the fusion of 3D point cloud and optical imagery, comprising the following steps:

[0068] Collect 3D point cloud data and image data of the work vehicle under study and the near-electric work site environment;

[0069] Based on the acquired 3D point cloud data, a 3D point cloud detection model for the work vehicle is constructed. The 3D point cloud detection model is used to obtain the target detection box of the work vehicle in 3D point cloud. The type of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information are obtained through the target detection box of the work vehicle in 3D point cloud.

[0070] Based on the acquired image data, a work vehicle image detection model is constructed, and a work vehicle image detection box is obtained using the work vehicle image detection model. The work vehicle category, center point pixel coordinates, and length and width dimensions of the work vehicle image detection box are obtained through the work vehicle image detection box.

[0071] For the camera used to acquire image data, calibrate the camera's intrinsic and extrinsic parameters and obtain the camera's distortion parameters;

[0072] The external parameters between the camera and the lidar are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The 3D point cloud data and image data are then objectified based on the external parameters between the camera and the lidar and the camera's intrinsic parameters.

[0073] Based on the object-processed 3D point cloud data and image data, the 3D point cloud target detection box of the work vehicle is projected onto the image coordinate system to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle is fused with the image detection box of the work vehicle in the image coordinate system.

[0074] A computer program product includes a computer program that, when executed by a processor, implements the steps of the above-described method.

[0075] The beneficial effects of this invention are:

[0076] This invention improves the accuracy and reliability of monitoring, enabling real-time monitoring of work vehicles and their operating environment. This helps to promptly identify potential safety hazards and provides timely warnings and guidance to work vehicle drivers and operators, thereby ensuring safe operation. The invention also reduces data processing time, improves system response speed, and avoids false alarms and missed alarms caused by data misalignment. Furthermore, by combining advanced sensor technology, image processing technology, and data processing technology, this invention provides a new solution for the intelligent monitoring and management of work vehicles, which helps promote the intelligent development of the work vehicle industry and improves operational efficiency and safety. Attached Figure Description

[0077] Figure 1 This is a schematic diagram of the structure of the present invention;

[0078] Figure 2 A diagram illustrating the effect of joint calibration of camera and LiDAR;

[0079] Figure 3 A calibration board for a 9×6 chessboard grid;

[0080] Figure 4 This is a diagram illustrating IOU calculation.

[0081] Figure 5 This is a schematic diagram for calculating the distance to the center point;

[0082] Figure 6 This is a flowchart of the present invention. Detailed Implementation

[0083] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments:

[0084] Example 1

[0085] A vehicle monitoring system based on the fusion of 3D point cloud and optical imagery, such as Figure 1 As shown, it includes:

[0086] The 3D point cloud and image data acquisition module is used to acquire 3D point cloud data and image data of the work vehicle under study and the near-electric work site environment. This design can improve detection accuracy and reliability by acquiring two types of data at the same time.

[0087] The 3D point cloud target detection module is used to construct a 3D point cloud detection model of the work vehicle based on the acquired 3D point cloud data, and to obtain the 3D point cloud target detection box of the work vehicle using the 3D point cloud detection model of the work vehicle.

[0088] The image target detection module is used to construct an image detection model of the work vehicle based on the acquired image data, and to obtain the image detection box of the work vehicle using the image detection model of the work vehicle;

[0089] The camera intrinsic and extrinsic parameter calibration module is used to calibrate the intrinsic and extrinsic parameters of the camera used to acquire image data. (The intrinsic parameters of the camera describe the inherent properties of the camera, including focal length, physical size of pixels, center pixel coordinates of the image, and radial distortion coefficient; the extrinsic parameters of the camera describe the position and orientation of the camera in space, including rotation parameters (ω, δ, θ) and translation parameters (Tx, Ty, Tz)). It also obtains the distortion parameters of the camera. This design improves the accuracy and reliability of image processing.

[0090] The 3D point cloud and camera joint calibration module is used to calibrate the external parameters between the camera and the LiDAR based on the camera's intrinsic and extrinsic parameters and distortion parameters. (The LiDAR's external parameters mainly relate to its position and orientation in space, specifically including three rotation parameters and three translation parameters; these parameters describe the relative transformation relationship between the LiDAR measurement coordinate system and other sensors or the global coordinate system, i.e., the rotation and translation transformation matrix, which the LiDAR uses to acquire the aforementioned point cloud information.) It performs object-oriented processing on the 3D point cloud data and image data based on the external parameters between the camera and the LiDAR and the camera's intrinsic parameters. The effect of the camera-LiDAR joint calibration is shown in the image below. Figure 2 As shown, this design realizes the external parameter calibration between the camera and the lidar, enabling the 3D point cloud data and image data of the work vehicle under study, the charged body, and the near-electric work site environment to be fused in the same coordinate system;

[0091] The 3D point cloud and optical image fusion detection module is used to project the 3D point cloud target detection box of the work vehicle onto the image coordinate system based on the object-processed 3D point cloud data and image data to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle and the image detection box of the work vehicle are fused in the image coordinate system to obtain the final fusion detection result. The final fusion detection result provides the type of work vehicle, the coordinate position of the work vehicle in 3D space, physical size information, center point pixel coordinates, and the length and width dimensions of the image detection box of the work vehicle. This design can effectively help identify the specific type and location of the work vehicle, and effectively help the work vehicle driver and operator understand the spatial layout of the work vehicle, which is crucial for avoiding potential collisions.

[0092] The above design improves the accuracy and robustness of target detection by projecting the 3D point cloud target detection box of the work vehicle into the image coordinate system and fusing it with the image detection box of the work vehicle, thus enabling comprehensive and multi-angle monitoring and analysis of the work vehicle.

[0093] In the above technical solution, the 3D point cloud data is divided into a training set of A% (70%) and a validation set of B% (30%), and the image data is divided into a training set of A% (70%) and a validation set of B% (30%). This design helps to ensure the independence and objectivity of the training of the 3D point cloud detection model and the image detection model of the work vehicle, improve the generalization ability of the 3D point cloud detection model and the image detection model of the work vehicle, and optimize the data utilization rate.

[0094] In the above technical solution, the specific method for constructing a 3D point cloud detection model of the work vehicle, obtaining the 3D point cloud target detection box of the work vehicle using the 3D point cloud target detection model, and obtaining the type of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information through the 3D point cloud target detection box of the work vehicle is as follows:

[0095] The 3D point cloud detection model of the work vehicle is the SE-SSD model (Self-Integrated Single-Stage Target Detector Model). The data augmentation module inputs the training set of 3D point cloud data into the SE-SSD model. By connecting the center of the ground truth bounding box (the ground truth bounding box is the set of coordinate values ​​of the smallest bounding cube of the actual position contour of the work vehicle in the 3D point cloud space) with the grid vertices (8 vertices), the target points of the work vehicle are divided into multiple subsets (6 subsets). Random swapping, sparsity and loss operations are performed on multiple subsets (6 subsets) to simulate the occlusion, sparsity and missing problems in the point cloud acquisition process.

[0096] The convolution module is used to input the training set of 3D point cloud data and the training set after random swapping, sparsity and loss operations into the convolution module to extract the spatial semantic features of the work vehicle in the 3D point cloud data (extract the spatial and semantic features of the work vehicle that have recognizability. Spatial and semantic features are just descriptive words. They are all quantified in the calculated feature map and cannot specifically distinguish which data are spatial features and which data are semantic features).

[0097] The generative adversarial module trains both student and teacher network models simultaneously. The teacher network is trained with the spatial semantic features before data augmentation, and the student network is trained with the spatial semantic features after data augmentation. The teacher's prediction results are used as the soft target of the student model to supervise the prediction results of the student model. Finally, the coordinates of the work vehicle in the 3D point cloud data are obtained through the student network. Based on the spatial semantic features of the work vehicle in the extracted 3D point cloud data, the module performs 3D point cloud target detection of the work vehicle and outputs the category of the work vehicle, the coordinates of the work vehicle in 3D space, and the physical size information.

[0098] The above design achieves high-precision and high-efficiency 3D point cloud target detection for work vehicles, and provides comprehensive information extraction capabilities for work vehicles.

[0099] In the above technical solution, the specific method for constructing a work vehicle image detection model, obtaining a work vehicle image detection box using the work vehicle image detection model, and obtaining the work vehicle category, center point pixel coordinates, and length and width dimensions of the work vehicle image detection box through the work vehicle image detection box is as follows:

[0100] The image detection model for the work vehicle is the YOLOX model. The data preprocessing module inputs the training set of work vehicle image data into the YOLOX model, performs mosaic data augmentation on the training data, and stitches the augmented images by random scaling, random cropping, and random arrangement.

[0101] The feature extraction module is used to extract the spatial semantic features of the work vehicle from the stitched image data;

[0102] The multi-scale feature fusion module takes the spatial semantic features of the work vehicle in the extracted image data as input and fuses feature maps of different receptive fields through multiple pooling kernels (small-sized pooling kernels);

[0103] The head network module takes the feature maps of different receptive fields after fusion as input, performs classification and regression operations, and obtains the category of the work vehicle, the pixel coordinates of the center point, and the length and width of the work vehicle image detection box.

[0104] The above design increases the diversity of image data from the work vehicle, which helps the YOLOX model generalize better and improves detection accuracy.

[0105] The above technical solution includes the following methods for calibrating the camera's intrinsic and extrinsic parameters: using OpenCV to collect calibration images and using the MATLAB toolbox to conduct camera calibration experiments to obtain the camera's intrinsic parameter matrix, distortion vector, and extrinsic parameter matrix;

[0106] The specific process for calibrating the camera's intrinsic and extrinsic parameters is as follows:

[0107] Let the intrinsic parameter be (f) x ,f y ,c x ,c y The distortion parameters are (k1,k2,p1,p2,k3);

[0108] Among them, f x f represents the camera's focal length in the horizontal direction (in pixels). y c represents the camera's focal length in the vertical direction (in pixels). x c represents the offset of the camera's optical axis in the horizontal direction of the image coordinate system (in pixels). y This represents the offset of the camera's optical axis in the vertical direction of the image coordinate system (in pixels); k1 is the first radial distortion parameter, k2 is the second radial distortion parameter, k3 is the distortion correction parameter; p1 is the first tangential distortion parameter, p2 is the second tangential distortion parameter;

[0109] A checkerboard calibration board of A×B (9×6) is selected, where the actual physical side length of each black and white square is Cmm (16.5mm), as follows. Figure 3 As shown;

[0110] A script written in Python was used to control the camera to acquire images. Gesture recognition was used to remotely operate the camera to trigger the shutter without contact. The specific operation was as follows: fix the camera position, run the script, move the checkerboard calibration board, raise the index finger, and acquire images. By continuously moving the checkerboard calibration board (moving forward, backward, left, and right, and rotating within ±10° to 30°), D images (20 images) were repeatedly taken. N images (15 images) with obvious changes were selected for calibration experiments. During the camera shooting process, a LiDAR was used to collect corresponding point cloud data for the 3D point cloud and camera joint calibration module.

[0111] Open the MATLAB Camera Calibrator toolbox, import the images to be calibrated, and perform corner detection on each image; solve for the camera's intrinsic and extrinsic parameters using the Zhang Zhengyou chessboard calibration method (a general camera calibration method that is built into the MATLAB Camera Calibrator toolbox), and optimize the calibration parameters by incorporating a distortion model and using maximum likelihood estimation; finally, obtain the camera's intrinsic and extrinsic parameter calibration results.

[0112] The above design calibrates the camera's internal and external parameters to accurately understand the camera's focal length, principal point offset, and distortion, and performs distortion correction on the image to improve image clarity and accuracy, thereby enhancing the camera's imaging quality.

[0113] In the above technical solution, the extrinsic parameters between the camera and the LiDAR are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The specific method for objectifying the 3D point cloud data and image data based on the extrinsic parameters between the camera and the LiDAR and the camera's intrinsic parameters is as follows:

[0114] In the lidar coordinate system, the relative position of the object and the lidar is represented as [X...]. L Y L Z L ], where the origin is the geometric center of the lidar, X L The axis is horizontal forward, Y L The axis is horizontal to the left, Y L The axis points vertically upwards, conforming to the rules of a right-handed coordinate system; in the camera coordinate system, the relative position of the object and the camera is represented as [X...]. C Y C Z C ], where the origin is the optical center O of the camera, X C The axis is parallel to the X-axis, and the Y-axis is parallel to the X-axis. C The axis is parallel to the Y-axis, and the Z-axis is parallel to the Y-axis. C The X-axis is parallel to the camera's optical axis and perpendicular to the image plane. The image coordinate system (which refers to a coordinate system established in the image pixel coordinate system and expressed in physical units (millimeters) to give the pixel scale physical meaning) is represented as [X,Y], where the origin (i.e., the intersection of the camera's optical axis and the image plane, usually located at the center of the image plane) is the camera's principal point, the X-axis is parallel to the u-axis, and the Y-axis is parallel to the v-axis; the pixel coordinate system is represented as [u,v], where the origin is the upper left corner of the image, the u-axis is horizontal to the right, and the v-axis is vertically downward.

[0115] The transformation formula from the lidar coordinate system to the camera coordinate system is as follows:

[0116]

[0117] The transformation formula from camera coordinate system to image coordinate system is as follows, where the Z-axis of the camera coordinate system is in the same direction as the Z-axis of the image coordinate system:

[0118]

[0119] The formula for transforming from image coordinates to pixel coordinates is as follows:

[0120]

[0121] In the lidar coordinate system (without considering camera imaging distortion), point P... L Its pixel projection point P I The conversion formula between them is:

[0122]

[0123] In the formula: M1 is the intrinsic parameter matrix of the camera; f is the length of the camera's focal length (u o v o dX is the position coordinate of the camera center point in the image coordinate system; dY is the physical dimension of the length of a single pixel in the image plane; s is the scaling factor from three-dimensional space to two-dimensional space; f is the length of the camera focal length; R is the rotation parameter matrix from the LiDAR coordinate system to the camera coordinate system (a 3×3 identity orthogonal matrix); T is the translation vector matrix from the LiDAR coordinate system to the camera coordinate system (a 3×1 matrix).

[0124] Let the coordinate transformation from a point in the lidar coordinate system to a point in the image coordinate system be:

[0125]

[0126] In the formula, M1 is only related to the camera's internal parameters. The camera's intrinsic parameter matrix M1 is obtained by solving the equation. Then, the corresponding constraint function (i.e., the transformation relationship formula) is established by extracting target features to solve for the extrinsic parameter matrix M2 between the camera and the lidar. This is a simplified version of the above transformation formula, where s is the scaling factor from three-dimensional space to two-dimensional space, and p... I Let P be the point L Pixel projection points;

[0127] The corner and center point normal vectors of the target in the 3D point cloud and image are extracted as features. The positions of the target corner and center point normal vectors in the lidar coordinate system and camera coordinate system are matched. The loss function is established and minimized. Then, the rotation parameter matrix R and translation vector matrix T between the camera and lidar are solved to obtain the extrinsic parameter matrix between the camera and lidar.

[0128] The above design can accurately establish the transformation relationship between the lidar coordinate system and the image coordinate system through the calibrated camera intrinsic parameter matrix M1 and the extrinsic parameter matrix M2 between the camera and the lidar, so that the 3D point cloud data can be accurately mapped onto the 2D image, and the data can be accurately fused.

[0129] In the above technical solution, the specific method for projecting the target detection box of the 3D point cloud of the work vehicle onto the image coordinate system is as follows:

[0130] The 3D point cloud target detection module performs 3D point cloud target detection and outputs the coordinate position (x1, y1, z1) of the work vehicle in 3D space and the physical dimensions (Δx, Δy, Δz) of the work vehicle.

[0131] Therefore, the range of the work vehicle in three-dimensional space can be obtained as follows:

[0132]

[0133] In the formula: x 1min Let x represent the minimum value of the target point cloud in the x-direction of 3D space. 1max The value of the target point cloud in the x-direction of 3D space is represented by y. 1min This represents the minimum value of the target point cloud in the y-direction of 3D space. 1max The z-axis represents the maximum value of the target point cloud in the 3D space along the y-direction. 1min Let z represent the minimum value of the target point cloud in the z-direction of 3D space. 1max This represents the maximum value of the target point cloud in the z-direction of three-dimensional space;

[0134] Therefore, the target detection bounding box of the 3D point cloud of the work vehicle is obtained as p. 1i =(x 1i y 1i , z 1i (i = 1, 2, ..., 8), where i represents the 8 vertices of the target detection box of the 3D point cloud of the work vehicle; based on the extrinsic parameter matrix M2 and the intrinsic parameter matrix M1 between the camera and the lidar obtained by the camera intrinsic and extrinsic parameter calibration module and the 3D point cloud and camera joint calibration module, the target detection box of the 3D point cloud of the work vehicle is projected onto the image coordinate system, and the projection of the target detection box of the 3D point cloud of the work vehicle onto the image coordinate system is p. ci =(u i v i (i = 1, 2, ..., 8) is:

[0135]

[0136] Among them, u iThis indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci The value on the x-axis, v i This indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci Value on the y-axis; x 1i This represents the x-coordinates and y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system. 1i This represents the y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system, and the z-coordinates of the bounding box on the point cloud. 1i This represents the coordinates of the eight vertices of the target detection box in the 3D point cloud of the work vehicle on the z-axis of the point cloud coordinate system;

[0137] Let L be the rectangle with the largest area formed by the above 3D point cloud projection results. _BBox Let C be the bounding box detected by the YOLOX model. _BBox ;

[0138] The above design can accurately project the 3D point cloud target detection box of the work vehicle onto the image coordinate system, realizing the fusion of 3D perception and 2D visual information, which helps to improve the accuracy and reliability of target detection.

[0139] In the above technical solution, the specific method for fusing the 3D point cloud projection detection box of the work vehicle with the image detection box of the work vehicle in the image coordinate system by calculating the intersection-union ratio and the center point distance is as follows:

[0140] S1: Let {L} be the projected bounding box of the 3D point cloud target detection box of the work vehicle. i_BBox}(i=1,2,…,m), the bounding box for the work vehicle image is {C j_BBox}(j=1,2,…,n);

[0141] S2: Calculate {L i_BBox} and {C j_BBox The intersection-union ratio (CUI) between each detection box and the pixel distance between their center points are given in the diagram. The CUI between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image is k. ij_IOU The calculation formula is as follows, and the diagram for IOU calculation is shown below. Figure 4 As shown:

[0142]

[0143] Let the coordinates of the center point of the image detection box of the work vehicle be (u c v cThe center point coordinates of the target detection box projection value of the 3D point cloud of the work vehicle are (u1, v1), and the center point distance calculation diagram is shown below. Figure 5 As shown, the distance D between the center points is defined. LC for:

[0144] D LC =|u c -u1|+|v c -v1|

[0145] By k ij_IOU and D LC The calculation formula yields the intersection-union matrix (IUU) between the 3D point cloud projection detection box and the image detection box of the work vehicle:

[0146]

[0147] The center point distance matrix is:

[0148]

[0149] Among them, D mn_LC This represents the distance to the center point in the m-th row and n-th column of the center point distance matrix.

[0150] S3: Let the IOU threshold be α (α takes the value of 0.5), when the intersection-union ratio k between the image detection box of the work vehicle and the projection value of the target detection box of the 3D point cloud of the work vehicle is... ij_IOU When ∈[0,α), it is determined that it belongs to 2 separate target objects;

[0151] S4: Let k gh_IOU =maxk (takes the maximum value of k), k gh_IOU This represents the intersection-union ratio (IUR) between the g-th row and the h-th column in the IUR matrix k, when k gh_IOU >α and D gh_LC If the value of D is less than β (β is 10), then the fusion is considered successful. gh_LC Let K be the distance between the center points in the g-th row and h-th column of the center point distance matrix; take the union between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image as the new target detection box, denoted as K. i_LC_BBox_gh Meanwhile, the serial number of the 3D point cloud projection detection frame of the work vehicle that was successfully integrated this time is recorded as g. i The successfully fused detection box number for the work vehicle image is h. i Delete the g-th row and h-th column from matrix k;

[0152] S5: Repeat step S4 until no expression satisfies k in the intersection-union matrix and the center-distance matrix. gh_IOU >α and D gh_LC The value of β is used to output the fusion detection result K. LC_BBox ={K i_LC_BBox_gh}(i=1,2,…,k), the category of the work vehicle, the coordinate position of the work vehicle in three-dimensional space, physical size information, center point pixel coordinates, and length and width size information of the work vehicle image detection box are obtained through the final fusion detection results;

[0153] S6: Let g = {g i},h={h i}(i=1,2,…,k), for the 3D point cloud projection detection boxes of the work vehicles that have not yet been successfully matched. and Output {Li_BBox}; for the image detection boxes of the work vehicle that have not yet been successfully matched. and Output {Cj_BBox};

[0154] The above design improves the accuracy and robustness of target detection on the work vehicle by integrating the 3D point cloud projection detection box and the image detection box of the work vehicle, and optimizes the selection of the target detection box.

[0155] Example 2

[0156] A method for monitoring work vehicles based on the fusion of 3D point cloud and optical imagery, such as Figure 6 As shown, 3D point cloud data and image data are acquired; based on the 3D point cloud data, a 3D point cloud detection model is constructed to obtain 3D point cloud target detection boxes; based on the image data, an image detection model is constructed to obtain image detection boxes; the camera's intrinsic and extrinsic parameters are calibrated, and the camera's distortion parameters are obtained; the extrinsic parameters between the camera and the LiDAR are calibrated, and the 3D point cloud data and image data are processed in an object-oriented manner; the 3D point cloud target detection boxes are projected onto the image coordinate system to obtain 3D point cloud projection detection boxes, and then fused with the image detection boxes.

[0157] The specific methods for monitoring work vehicles include the following steps:

[0158] Collect 3D point cloud data and image data of the work vehicle under study and the near-electric work site environment;

[0159] Based on the acquired 3D point cloud data, a 3D point cloud detection model for the work vehicle is constructed. The 3D point cloud detection model is used to obtain the target detection box of the work vehicle in 3D point cloud. The type of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information are obtained through the target detection box of the work vehicle in 3D point cloud.

[0160] Based on the acquired image data, a work vehicle image detection model is constructed, and a work vehicle image detection box is obtained using the work vehicle image detection model. The work vehicle category, center point pixel coordinates, and length and width dimensions of the work vehicle image detection box are obtained through the work vehicle image detection box.

[0161] For the camera used to acquire image data, calibrate the camera's intrinsic and extrinsic parameters and obtain the camera's distortion parameters;

[0162] The external parameters between the camera and the lidar are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The 3D point cloud data and image data are then objectified based on the external parameters between the camera and the lidar and the camera's intrinsic parameters.

[0163] Based on the object-processed 3D point cloud data and image data, the 3D point cloud target detection box of the work vehicle is projected onto the image coordinate system to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle is fused with the image detection box of the work vehicle in the image coordinate system.

[0164] Example 3

[0165] A computer program product includes a computer program, characterized in that, when the computer program is executed by a processor, it implements the steps of the method described in Embodiment 2.

[0166] The contents not described in detail in this specification are existing technologies known to those skilled in the art.

Claims

1. A monitoring system for work vehicles based on the fusion of three-dimensional point clouds and optical images, characterized in that it include: The 3D point cloud and image data acquisition module is used to acquire 3D point cloud data and image data of the work vehicle under study and the near-electric work site environment; The 3D point cloud target detection module is used to construct a 3D point cloud detection model of the work vehicle based on the acquired 3D point cloud data, and to obtain the 3D point cloud target detection box of the work vehicle using the 3D point cloud detection model of the work vehicle. The image target detection module is used to construct an image detection model of the work vehicle based on the acquired image data, and to obtain the image detection box of the work vehicle using the image detection model of the work vehicle; The camera intrinsic and extrinsic parameter calibration module is used to calibrate the intrinsic and extrinsic parameters of the camera used to acquire image data and to obtain the camera's distortion parameters. The 3D point cloud and camera joint calibration module is used to calibrate the external parameters between the camera and the lidar based on the camera's intrinsic and extrinsic parameters and distortion parameters, and to perform object-oriented processing on the 3D point cloud data and image data based on the external parameters between the camera and the lidar and the camera's intrinsic parameters. The 3D point cloud and optical image fusion detection module is used to project the 3D point cloud target detection box of the work vehicle onto the image coordinate system based on the object-processed 3D point cloud data and image data to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle and the image detection box of the work vehicle are fused in the image coordinate system. The specific method for fusing the 3D point cloud projection detection box of the work vehicle with the image detection box of the work vehicle in the image coordinate system by calculating the intersection-union ratio and the center point distance is as follows: S1: Let {L} be the projected bounding box of the 3D point cloud target detection box of the work vehicle. i_BBox }(i=1,2,…,m), the bounding box for the work vehicle image is {C j_BBox }(j=1,2,…,n); S2: Calculate {L i_BBox } and {C j_BBox The intersection-union ratio (CUI) between each detection box and the pixel distance between their center points are given in the diagram. The CUI between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image is k. ij_IOU The calculation is as follows: Let the coordinates of the center point of the image detection box of the work vehicle be (u c v c The center point coordinates of the target detection box projection value of the 3D point cloud of the work vehicle are (u1, v1), and the center point distance D is defined as... LC for: D LC =|u c -u1|+|v c -v1| By k ij_IOU and D LC The calculation formula yields the intersection-union matrix (IUU) between the 3D point cloud projection detection box and the image detection box of the work vehicle: The center point distance matrix is: Among them, D mn_LC This represents the distance to the center point in the m-th row and n-th column of the center point distance matrix. S3: Let the IOU threshold be α, and when the intersection-union ratio (IU) between the image detection box of the work vehicle and the projection value of the target detection box of the 3D point cloud of the work vehicle is k ij_IOU When ∈[0,α), it is determined that it belongs to two separate target objects; S4: Let k gh_IOU =maxk,k gh_IOU This represents the intersection-union ratio (IUR) between the g-th row and the h-th column in the IUR matrix k, when k gh_IOU >α and D gh_LC When D < β, the fusion is considered successful. gh_LC Let K be the distance between the center points in the g-th row and h-th column of the center point distance matrix; take the union between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image as the new target detection box, denoted as K. i_LC_BBox_gh Meanwhile, the serial number of the 3D point cloud projection detection frame of the work vehicle that was successfully integrated this time is recorded as g. i The successfully fused detection box number for the work vehicle image is h. i Delete the g-th row and h-th column from matrix k; S5: Repeat step S4 until no expression satisfies k in the intersection-union matrix and the center-distance matrix. gh_IOU >α and d gh_LC The value of β is used to output the fusion detection result K. LC_BBox ={K i_LC_BBox_gh }(i=1,2,…,k).

2. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 1, characterized in that: The 3D point cloud data is divided into A% training set and B% validation set, and the image data is divided into A% training set and B% validation set.

3. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 2, characterized in that: The specific method for constructing a 3D point cloud detection model of the work vehicle, obtaining the 3D point cloud target detection box of the work vehicle using the 3D point cloud target detection model, and obtaining the work vehicle's category, coordinate position in 3D space, and physical size information from the 3D point cloud target detection box is as follows: The 3D point cloud detection model of the work vehicle is the SE-SSD model. The data augmentation module inputs the training set of the 3D point cloud data into the SE-SSD model. By connecting the center of the ground truth bounding box with the grid vertex, the target points of the work vehicle are divided into multiple subsets, and random swapping, sparsity and loss operations are performed on the multiple subsets. The convolution module is used to input the training set of 3D point cloud data and the training set after random swapping, sparsity and loss operations into the convolution module to extract the spatial semantic features of the work vehicle in the 3D point cloud data. The generative adversarial module trains student and teacher network models simultaneously, and performs 3D point cloud target detection of the work vehicle based on the spatial semantic features of the work vehicle in the extracted 3D point cloud data. It outputs the category of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information.

4. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 1, characterized in that: The specific method for constructing a work vehicle image detection model, obtaining work vehicle image detection boxes using the work vehicle image detection model, and obtaining the work vehicle category, center point pixel coordinates, and length and width dimensions of the work vehicle image detection box from the work vehicle image detection box is as follows: The image detection model for the work vehicle is the YOLOX model. The data preprocessing module inputs the training set of work vehicle image data into the YOLOX model, performs mosaic data enhancement on the training data, and stitches the enhanced images by random scaling, random cropping, and random arrangement. The feature extraction module is used to extract the spatial semantic features of the work vehicle from the stitched image data; The multi-scale feature fusion module takes the spatial semantic features of the work vehicle in the extracted image data as input and fuses feature maps of different receptive fields through multiple pooling kernels. The head network module takes the fused feature maps of different receptive fields as input, performs classification and regression operations, and obtains the category of the work vehicle, the pixel coordinates of the center point, and the length and width of the work vehicle image detection box.

5. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 1, characterized in that: Methods for calibrating camera intrinsic and extrinsic parameters include: using OpenCV to collect calibration images and using MATLAB toolboxes to conduct camera calibration experiments to obtain the camera's intrinsic parameter matrix, distortion vector, and extrinsic parameter matrix; The specific process for calibrating the camera's intrinsic and extrinsic parameters is as follows: Let the intrinsic parameter be (f) x ,f y ,c x ,c y The distortion parameters are (k1,k2,p1,p2,k3); Among them, f x f represents the camera's focal length in the horizontal direction. y c represents the camera's focal length in the vertical direction. x c represents the offset of the camera's optical axis in the horizontal direction of the image coordinate system. y This represents the offset of the camera's optical axis in the vertical direction of the image coordinate system; k1 is the first radial distortion parameter, k2 is the second radial distortion parameter, and k3 is the distortion correction parameter; p1 is the first tangential distortion parameter, and p2 is the second tangential distortion parameter. Select an A×B chessboard calibration board, where the actual physical side length of each black and white square is C mm; A script written in Python was used to control the camera to acquire images. By continuously moving the checkerboard calibration board, D images were repeatedly taken. N images with obvious changes were selected for calibration experiments. During the camera shooting process, LiDAR was used to collect corresponding point cloud data. Open the MATLAB Camera Calibrator toolbox, import the images to be calibrated, and perform corner detection on each image; solve the intrinsic and extrinsic parameters of the camera according to Zhang Zhengyou's chessboard calibration method, and optimize the calibration parameters by adding a distortion model and using the maximum likelihood estimation method; finally, obtain the calibration results of the camera's intrinsic and extrinsic parameters.

6. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 5, characterized in that: The extrinsic parameters between the camera and the LiDAR are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The specific method for objectifying 3D point cloud data and image data based on the extrinsic parameters between the camera and the LiDAR and the camera's intrinsic parameters is as follows: In the lidar coordinate system, the relative position of the object and the lidar is represented as [X...]. L Y L Z L ], where the origin is the geometric center of the lidar, X L The axis is horizontal forward, Y L The axis is horizontal to the left, Y L The axis points vertically upwards, conforming to the rules of a right-handed coordinate system; in the camera coordinate system, the relative position of the object and the camera is represented as [X...]. C Y C Z C ], where the origin is the optical center O of the camera, X C The axis is parallel to the X-axis, and the Y-axis is parallel to the X-axis. C The axis is parallel to the Y-axis, and the Z-axis is parallel to the Y-axis. C The X-axis is parallel to the camera's optical axis and perpendicular to the image plane. The image coordinate system is represented as [X,Y], where the origin is the camera's principal point, the X-axis is parallel to the u-axis, and the Y-axis is parallel to the v-axis. The pixel coordinate system is represented as [u,v], where the origin is the top left corner of the image, the u-axis is horizontal to the right, and the v-axis is vertically downward. The transformation formula from the lidar coordinate system to the camera coordinate system is as follows: The transformation formula from camera coordinate system to image coordinate system is as follows, where the Z-axis of the camera coordinate system is in the same direction as the Z-axis of the image coordinate system: The formula for transforming from image coordinates to pixel coordinates is as follows: Point P in the lidar coordinate system L Its pixel projection point P I The conversion formula between them is: In the formula: M1 is the intrinsic parameter matrix of the camera; f is the length of the camera's focal length (u o v o dX represents the position coordinates of the camera center point in the image coordinate system; dX represents the physical length of a single pixel in the image plane; and dY represents the physical width of a single pixel in the image plane. s is the scaling factor from 3D space to 2D space, f is the length of the camera focal length; R is the rotation parameter matrix from the lidar coordinate system to the camera coordinate system; T is the translation vector matrix from the lidar coordinate system to the camera coordinate system. Let the coordinate transformation from a point in the lidar coordinate system to a point in the image coordinate system be: In the formula, M1 is only related to the camera's internal parameters. The camera's intrinsic parameter matrix M1 is obtained by solving the formula, and then the corresponding constraint function is established by extracting target features to solve for the extrinsic parameter matrix M2 between the camera and the lidar. This is a simplified version of the above transformation formula, where s is the scaling factor from three-dimensional space to two-dimensional space, and p... I Let P be the point L Pixel projection points; The corner and center point normal vectors of the target in the 3D point cloud and image are extracted as features. The positions of the target corner and center point normal vectors in the lidar coordinate system and the camera coordinate system are matched. The loss function is established and minimized. Then, the rotation parameter matrix R and translation vector matrix T between the camera and lidar are solved to obtain the extrinsic parameter matrix between the camera and lidar.

7. The vehicle monitoring system based on the fusion of three-dimensional point cloud and optical image as described in claim 6, characterized in that: The specific method for projecting the target detection bounding box of the 3D point cloud of the work vehicle onto the image coordinate system is as follows: The 3D point cloud target detection module performs 3D point cloud target detection and outputs the coordinate position (x1, y1, z1) of the work vehicle in 3D space and the physical dimensions (Δx, Δy, Δz) of the work vehicle. Therefore, the range of the work vehicle in three-dimensional space can be obtained as follows: In the formula: x 1min Let x represent the minimum value of the target point cloud in the x-direction of 3D space. 1max The value of the target point cloud in the x-direction of 3D space is represented by y. 1min This indicates that the target point cloud is in three-dimensional space. y Minimum value in the direction, y 1max The z-axis represents the maximum value of the target point cloud in the 3D space along the y-direction. 1min Let z represent the minimum value of the target point cloud in the z-direction of 3D space. 1max This represents the maximum value of the target point cloud in the z-direction of three-dimensional space; Therefore, the target detection bounding box of the 3D point cloud of the work vehicle is obtained as p. 1i =(x 1i y 1i , z 1i (i = 1, 2, ..., 8), where i represents the 8 vertices of the target detection box of the 3D point cloud of the work vehicle; based on the extrinsic parameter matrix M2 and the intrinsic parameter matrix M1 between the camera and the lidar obtained by the camera intrinsic and extrinsic parameter calibration module and the 3D point cloud and camera joint calibration module, the target detection box of the 3D point cloud of the work vehicle is projected onto the image coordinate system, and the projection of the target detection box of the 3D point cloud of the work vehicle onto the image coordinate system is p. ci =(u i v i (i = 1, 2, ..., 8) is: Among them, u i This indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci The value on the x-axis, v i This indicates the projected positions p of the eight vertices of the 3D point cloud target detection bounding box of the work vehicle onto the image coordinate system. ci Value on the y-axis; x 1i This represents the x-coordinates and y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system. 1i This represents the y-coordinates of the eight vertices of the target detection bounding box in the 3D point cloud of the work vehicle on the point cloud coordinate system, and the z-coordinates of the bounding box on the point cloud. 1i This represents the coordinates of the eight vertices of the target detection box in the 3D point cloud of the work vehicle on the z-axis of the point cloud coordinate system; Let L_ be the rectangle with the largest area formed by the above 3D point cloud projection results. BBox Let C_ be the bounding box detected by the YOLOX model. BBox .

8. A method for monitoring work vehicles based on the fusion of 3D point clouds and optical images, characterized in that, It involves the following steps: Collect three-dimensional point cloud data and image data of the work vehicle, the live conductor, and the near-electric work site environment; Based on the acquired 3D point cloud data, a 3D point cloud detection model for the work vehicle is constructed. The 3D point cloud detection model is used to obtain the target detection box of the work vehicle in 3D point cloud. The type of the work vehicle, the coordinate position of the work vehicle in 3D space, and the physical size information are obtained through the target detection box of the work vehicle in 3D point cloud. Based on the acquired image data, a work vehicle image detection model is constructed, and a work vehicle image detection box is obtained using the work vehicle image detection model. The work vehicle category, center point pixel coordinates, and length and width dimensions of the work vehicle image detection box are obtained through the work vehicle image detection box. For the camera used to acquire image data, calibrate the camera's intrinsic and extrinsic parameters and obtain the camera's distortion parameters; The external parameters between the camera and the lidar are calibrated based on the camera's intrinsic and extrinsic parameters and distortion parameters. The 3D point cloud data and image data are then objectified based on the external parameters between the camera and the lidar and the camera's intrinsic parameters. Based on the object-processed 3D point cloud data and image data, the 3D point cloud target detection box of the work vehicle is projected onto the image coordinate system to obtain the 3D point cloud projection detection box of the work vehicle. By calculating the intersection-union ratio and the center point distance, the 3D point cloud projection detection box of the work vehicle is fused with the image detection box of the work vehicle in the image coordinate system. The specific method for fusing the 3D point cloud projection detection box of the work vehicle with the image detection box of the work vehicle in the image coordinate system by calculating the intersection-union ratio and the center point distance is as follows: S1: Let {L} be the projected bounding box of the 3D point cloud target detection box of the work vehicle. i_BBox }(i=1,2,…,m), the bounding box for the work vehicle image is {C j_BBox }(j=1,2,…,n); S2: Calculate {L i_BBox } and {C j_BBox The intersection-union ratio (CUI) between each detection box and the pixel distance between their center points are given in the diagram. The CUI between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image is k. ij_IOU The calculation is as follows: Let the coordinates of the center point of the image detection box of the work vehicle be (u c v c The center point coordinates of the target detection box projection value of the 3D point cloud of the work vehicle are (u1, v1), and the center point distance D is defined as... LC for: D LC =|u c -u1|+|v c -v1| By k ij_IOU and D LC The calculation formula yields the intersection-union matrix (IUU) between the 3D point cloud projection detection box and the image detection box of the work vehicle: The center point distance matrix is: Among them, D mn_LC This represents the distance to the center point in the m-th row and n-th column of the center point distance matrix. S3: Let the IOU threshold be α, and when the intersection-union ratio (IU) between the image detection box of the work vehicle and the projection value of the target detection box of the 3D point cloud of the work vehicle is k ij_IOU When ∈[0,α), it is determined that it belongs to two separate target objects; S4: Let k gh_IOU =maxk,k gh_IOU This represents the intersection-union ratio (IUR) between the g-th row and the h-th column in the IUR matrix k, when k gh_IOU >α and D gh_LC When D < β, the fusion is considered successful. gh_LC Let K be the distance between the center points in the g-th row and h-th column of the center point distance matrix; take the union between the 3D point cloud projection detection box of the work vehicle and the detection box in the work vehicle image as the new target detection box, denoted as K. i_LC_BBox_gh Meanwhile, the serial number of the 3D point cloud projection detection frame of the work vehicle that was successfully integrated this time is recorded as g. i The successfully fused detection box number for the work vehicle image is h. i Delete the g-th row and h-th column from matrix k; S5: Repeat step S4 until no expression satisfies k in the intersection-union matrix and the center-distance matrix. gh_IOU >α and D gh_LC The value of β is used to output the fusion detection result K. LC_bBox ={K i_LC_BBox_gh }(i=1,2,…,k).

9. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method of claim 8.

Images