Electricity approaching behavior monitoring method based on cross-modal stereoscopic perception

Abstract

The invention discloses a near-electricity behavior monitoring method based on cross-modal stereoscopic perception, and belongs to the technical field of electric power safety monitoring. According to the invention, the problem of poor early warning reliability caused by low precision of the existing monitoring method is solved. Through the hierarchical processing flow of electric field guiding, visual locking, point cloud ranging and cross checking, the system can efficiently and reliably lock an electrified body target, the electric field sensor firstly provides orientation guidance, large-range searching of the target by visual and laser radar is avoided, and the computing power requirement and the false alarm rate of the system are greatly reduced; the visual camera and the laser radar carry out accurate locking and distance measurement in a limited area guided by an electric field, mutual verification of a physical level is realized through a cross check mechanism, the design constructs redundancy safety guarantee based on different physical principles, and the early warning reliability of a single sensor in a severe environment is greatly improved. The method can be applied to power safety monitoring.

Description

Technical Field

[0001] This invention belongs to the field of power safety monitoring technology, specifically relating to a near-electric behavior monitoring method based on cross-modal three-dimensional sensing. Background Technology

[0002] In live-line work sites such as power grid construction and line renovation, the movement and operation of construction machinery pose significant safety hazards. Insufficient safety distance between the machinery and live equipment can easily lead to electric shock accidents, seriously threatening the personal safety of workers and damaging electrical equipment, resulting in substantial economic losses. Therefore, precise safety distance monitoring is necessary at live-line work sites. Multi-source sensors deployed on construction machinery can quickly collect on-site data. The spatial relationship between the machinery and live equipment dynamically changes with the movement of the machinery, causing variations in the measurement data from different sensors. Therefore, it is essential to utilize multi-source sensor data to remotely monitor the distance between the machinery and live equipment, providing early warnings before danger occurs, which can significantly improve the safety of electrical work.

[0003] Currently, mainstream near-electric safety monitoring methods mainly rely on multi-source sensors such as power frequency electric field sensors, binocular vision cameras, and lidar, combined with artificial intelligence models for analysis. The main methods currently employed include:

[0004] 1) Directly using a single sensor model for processing: Due to the complex environment of the construction site, factors such as electromagnetic interference, changes in lighting, and severe weather have a huge impact on single sensor data. However, this method lacks an effective environmental adaptability mechanism, so this processing mode is usually not accurate and has poor stability.

[0005] 2) Using pre-modeled integrated systems: This method can address some environmental issues relatively effectively; however, due to changes in site layout and the emergence of new obstacles, there will be significant differences between the data obtained in the laboratory and the data in real application scenarios. This approach usually yields good experimental results in laboratory data, but in practical applications, this method is prone to fitting to specific patterns, leading to prediction failure. Especially when construction machinery is at a critical safe distance, the differences in sensor data characteristics caused by distance changes are extremely subtle, making it difficult for both of the above methods to effectively distinguish them, resulting in a sharp decline in the reliability of early warnings.

[0006] 3) Using a more complex deep neural network model to distinguish the core difference between safety and danger: The biggest problem with this model is that the number of "dangerous working conditions" samples is too small to support the full training of a large neural network, which leads to overfitting and makes the warning accuracy insufficient.

[0007] Therefore, existing monitoring methods still suffer from poor early warning reliability due to low monitoring accuracy. It is an urgent need to propose a method that can adapt to the subtle differences between safe and dangerous critical states, as well as the differences in sensor data characteristics caused by different construction machinery and different work site environments. Summary of the Invention

[0008] The purpose of this invention is to solve the problem of poor early warning reliability caused by the low accuracy of existing monitoring methods, and to propose a near-electric behavior monitoring method based on cross-modal three-dimensional sensing.

[0009] The technical solution adopted by the present invention to solve the above-mentioned technical problems is as follows:

[0010] A method for monitoring near-electrical behavior based on cross-modal stereo sensing, the method specifically includes the following steps:

[0011] Step S1: Deploy multi-source sensors on the construction machinery, establish a historical data table SenList based on the historical data collected by the multi-source sensors, and initialize the cross-modal fusion feature list FusionList;

[0012] The multi-source sensors include a power frequency electric field sensor array, a binocular vision camera, and a lidar.

[0013] Step S2: Establish the electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the power frequency electric field sensor array EFInput, and the output of the module DirectionCalibrationModel is the charged body azimuth array EFOuput.

[0014] Step S3: Establish a cross-modal data fusion and target recognition module FusionModel. The input of module FusionModel is Lidar point cloud data LidarData, binocular vision image data ImageData and array EFInput. The output of module FusionModel is the cross-modal fusion result FusionOutput.

[0015] Step S4: Use the FusionModel module to process each group of data in the historical data table SenList, and store the processing results in the list FusionList;

[0016] Step S5: Establish a hierarchical early warning decision model, WarningModel, and train the model using the data in the list FusionList;

[0017] Step S6: Collect multi-source sensor data TestData in real time using sensors, and use the FusionModel module and the WarningModel model to determine whether a near-electric hazard occurs. If a near-electric hazard occurs, an early warning needs to be issued; otherwise, no action is required.

[0018] Furthermore, the specific process of step S1 is as follows:

[0019] Step S101: Install a power frequency electric field sensor array, a binocular vision camera, and a lidar on the top of the boom of the construction machinery;

[0020] Furthermore, spatiotemporal joint calibration of the binocular vision camera and LiDAR was performed to obtain the camera intrinsic parameters and the transformation relationship between the LiDAR coordinate system and the binocular vision camera coordinate system.

[0021] Step S102: Use the power frequency electric field sensor array, binocular vision camera and lidar to collect time series data simultaneously, and use a total station or laser rangefinder to obtain the actual distance between the geometric center of the power frequency electric field sensor array and the target charged equipment simultaneously.

[0022] Based on the acquired data, a historical data table SenList is created. The SenList table contains four fields:

[0023] Lidar point cloud data (LidarData): A collection of three-dimensional coordinate points;

[0024] Binocular vision image data: contains two color images, each with a resolution of 1280×720, simultaneously acquired by the left and right cameras;

[0025] Electric field sensor data EFieldData: An array containing 4 floating-point numbers, the array includes... , , , and These represent the electric field intensity values ​​in the four directions: top, front, left, and right, respectively.

[0026] SafetyLabel: A floating-point number representing the actual distance D between the geometric center of the electric field sensor array and the target charged device. true ;

[0027] Step S103: Record the number of elements in the historical data table SenList as SenNum;

[0028] Step S104: Create an empty cross-modal fusion feature list FusionList. Each element of the FusionList contains 5 fields:

[0029] Orientation of charged body: a tuple containing horizontal and vertical azimuth angles;

[0030] LidarDistance: A floating-point number representing the nearest distance D measured by the lidar. lidar ;

[0031] Binocular vision distance feature VisionDistance: A floating-point number representing the nearest distance D measured by binocular vision. vision ;

[0032] Final FusionDistance: A floating-point number representing the final distance D after fusion verification. fusion ;

[0033] SafetyDecision: An integer variable used to express the warning level;

[0034] Step S105: Initialize all five fields of the elements in the FusionList to be 0;

[0035] Step S106: Through parameter calibration, obtain the consistency threshold Th1, deviation threshold Th2, voxel downsampling side length, and electric field inverse calculation coefficient values.

[0036] Step S107, Step S1 ends.

[0037] Furthermore, the specific process of step S2 is as follows:

[0038] Step S201: Establish an electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the electric field sensor array EFInput, and the output is the charged body azimuth array EFOuput. Step S202: Calculate the electric field composition index and azimuth index based on the array EFInput: (1) Composite index of electric field level :

[0039]

[0040] (2) Vertical composite index of electric field :

[0041]

[0042] (3) Horizontal azimuth radian index :

[0043]

[0044] Where atan2 represents the arctangent function in the four quadrants;

[0045] (4) Vertical azimuth radian index :

[0046]

[0047] (5) Horizontal azimuth index :

[0048]

[0049] (6) Vertical azimuth index :

[0050]

[0051] in, , , and All are conversion factors;

[0052] Step S203: Create a charged body azimuth array EFOuput with dimension 2;

[0053] Step S204, calculate the result in S202 and Stored in the array EFOuput;

[0054] Step S205: The array EFOuput is used as the output of the module DirectionCalibrationModel.

[0055] Furthermore, the specific process of step S3 is as follows:

[0056] Step S301: Establish a cross-modal data fusion and target locking module FusionModel. Define the input of module FusionModel as LidarData, ImageData and array EFInput, and define the output of module FusionModel as cross-modal fusion result FusionOutput.

[0057] The DirectionCalibrationModel module is called to process the EFInput array;

[0058] Step S302: The first temporary variable FusionResult is assigned the result of processing the array EFInput using the module DirectionCalibrationModel. The processing result includes the horizontal azimuth angle. and vertical azimuth ;

[0059] Step S303: Perform a voxel downsampling algorithm on LidarData to generate a downsampled point cloud;

[0060] Based on the transformation relationship between the lidar coordinate system and the binocular vision camera coordinate system, each point in the downsampled point cloud is transformed to the camera coordinate system;

[0061] Based on horizontal azimuth and vertical azimuth A three-dimensional view frustum is defined with the geometric center of the electric field sensor as its vertex, and the horizontal angular range of the three-dimensional view frustum is... The vertical angle range is The point cloud transformed to the camera coordinate system and located inside the 3D view frustum is used as the candidate point cloud; the four corner points of the 3D view frustum are projected onto the image plane according to the camera intrinsic parameters, and the quadrilateral region formed by the four projection points is used as the ROI region in the image;

[0062] Step S304: Input the ROI region in the image into the YOLOv5-lite model, and output the bounding box coordinates, the category of charged body in the bounding box, and the confidence score of the bounding box through the YOLOv5-lite model.

[0063] If the highest confidence level is higher than 0.6, then the center point of the bounding box with the highest confidence level is used as the pixel coordinate of the charged body, and the coordinates of the charged body on the image are marked as follows: And continue with step S307;

[0064] Otherwise, proceed to step S305;

[0065] Step S305, Point cloud rollback mechanism for failed visual detection:

[0066] The candidate point cloud is clustered using the DBSCAN clustering algorithm to obtain the point cloud cluster closest to the geometric center of the electric field sensor array. The centroid of the obtained point cloud cluster is used as the candidate three-dimensional position of the charged body, and then step S306 is executed.

[0067] Step S306, the secondary verification and attribute analysis of the rollback mechanism, specifically includes:

[0068] Step 1: Azimuth Verification

[0069] The calculated azimuth angle of the centroid of the point cloud cluster relative to the geometric center of the electric field sensor Then compare with the azimuth information Relationship:

[0070] like If the target recognition fails, the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values. Then, the process jumps directly to step S310. Otherwise, the azimuth check passes, and proceed to step 2; Step 2, Attribute Analysis;

[0071] The linearity of the obtained point cloud clusters was determined using principal component analysis (PCA). And compare linearity Size relative to threshold 0.8:

[0072] If linearity If the target does not conform to the wire feature, it is regarded as an unknown obstacle. The output array FusionOutput corresponding to the current group of data is marked as an unknown obstacle. The FusionDistance field in the output FusionOutput corresponding to the current group of data is set to -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FusionDistance field in FusionOutput are all marked as invalid values, and the process jumps directly to step S310.

[0073] If linearity If so, proceed to step 3;

[0074] Step 3: Reflection Intensity Analysis

[0075] The average laser reflection intensity of the point cloud cluster is calculated, and then compared with a preset feature library of power equipment reflection intensity. If a match is successful, the target is confirmed as a charged object, and the pixel coordinates of the charged object in the image are determined based on the candidate 3D position of the charged object. ;

[0076] And proceed to step S307; If the matching fails, the target is determined to be an unknown obstacle, and the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1, and the LidarDistance field, the binocular visual distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values; and the process jumps directly to step S310.

[0077] Step S307: Calculate the visual distance between the lidar and the binocular camera, and then execute step S308;

[0078] The calculation process for the visual distance between the lidar and the binocular camera is as follows:

[0079] Based on the camera intrinsic parameters, all downsampled point clouds transformed to the camera coordinate system are back-projected onto the image plane, and the points falling into the image plane are determined by their positions. The back-projected pixels within a square region with a side length of 10 pixels centered on the electric field sensor array are stored in a candidate set. The nearest distance from all point clouds in the candidate set to the geometric center of the electric field sensor array is calculated, and the obtained nearest distance is taken as the lidar range. ;

[0080] According to the camera focal length and baseline distance Calculate the visual distance of the binocular camera :

[0081]

[0082] in, Indicates parallax;

[0083] Step S308: Construct the fused output structure

[0084] Create a cross-modal fusion output, FusionOutput, with the same structure as the elements in the list FusionList, and assign the azimuth information to the Orientation field of FusionOutput. Assign the value to the LidarDistance field. Assign the value to the VisionDistance field. ; Step S309, based on the consistency threshold pair and Cross-verification and fusion are performed to determine the final fusion distance, FusionDistance; specifically: Step (1): Calculate the distance difference ; like If the data is consistent, the FusionDistance field of FusionOutput will be assigned a value. ; Step (2), if If the data is found to be biased, a weighted fusion calculation will be used. :

[0085]

[0086] in, and All are weighting coefficients;

[0087] Assign a value to the FusionDistance field of FusionOutput. ;

[0088] Step (3), if If a data conflict is detected, the distance is calculated by back-calculating the electric field. :

[0089]

[0090] in, The total electric field strength is ;

[0091] Calculate the dynamic fault tolerance threshold :

[0092]

[0093] in, , and All are constants. As environmental factors, For the theoretical electric field strength, The variance of the electric field data;

[0094]

[0095] in, For the field voltage level, The equivalent radius of the line; Step (4) Execute the final arbitration

[0096] like If the conflict is deemed interpretable, the FusionDistance field will be assigned a value. ;

[0097] Otherwise, mark the FusionDistance field as invalid -1 and immediately trigger the highest level alert;

[0098] Step S310: Output the fusion result.

[0099] Furthermore, the specific process of step S303 is as follows:

[0100] The three-dimensional space is divided into cubic voxels, and the three-dimensional points contained within each cubic voxel are determined. Then, for any cubic voxel, the representative point of the cubic voxel is determined based on the point cloud within the cubic voxel:

[0101]

[0102] in, This represents the set of all points contained within the cube voxel. Represents a set The number of points inside, For set The first The coordinates of the points The coordinates of the representative point of the cube voxel;

[0103] A downsampling point cloud is constructed using representative points of each cubic voxel.

[0104] Furthermore, the specific process of step S304 is as follows:

[0105] Points in the lidar coordinate system Transform to the camera coordinate system to obtain the point :

[0106]

[0107] Where R represents a rotation matrix of dimension 3×3, and T represents a translation vector of dimension 3×1.

[0108] Furthermore, the specific process of step S4 is as follows:

[0109] Step S401: Initialize the processing counter ProcCounter=1;

[0110] Step S402: Assign the first temporary variable ProcessedData to the ProcCounter element of SenList;

[0111] Step S403: The cross-modal data fusion and target recognition module FusionModel is called to process the ProcCounter element in the list SenList. The output of the module is the real-time fusion result RealTimeFusionResult, which is a structure with the same structure as the element in the list FusionList. Step S404: Assign the values ​​of each field of RealTimeFusionResult to the ProcCounter element in the list FusionList. The fields to be assigned include: Orientation of charged body, LidarDistance, VisionDistance, and FusionDistance. Step S405, verify the value of the FusionDistance field of RealTimeFusionResult:

[0112] If the FusionDistance field value is -1, it means that the target recognition failed or was identified as an unknown obstacle, and the process will jump directly to S410; otherwise, continue to execute step S406. Step S406, calculate the error between the fused distance and the actual distance.

[0113]

[0114] Step S407: Determine the data confidence level based on the Error value and add the Confidence field to the ProcCounter element of the FusionList;

[0115] If the error is greater than 0.5 meters, the confidence flag for the current data is set to 0, indicating low confidence.

[0116] Otherwise, the confidence flag for the current data is set to 1, indicating high confidence. Step S408: Compare the SafetyLabel with a preset safety distance threshold, generate a graded safety decision label based on the comparison result, and assign the generated graded safety decision label to the SafetyDecision field of the ProcCounter element of the FusionList; specifically:

[0117] If D true < D safe If so, the SafetyDecision field will be assigned the value 2;

[0118] If D safe ≤ D true < D warning If so, the SafetyDecision field will be assigned the value 1;

[0119] Otherwise, the SafetyDecision field is set to 0; Step S409: Increment the value of the processing counter ProcCounter by 1; Step S410, determine the loop condition:

[0120] If ProcCounter is less than or equal to the total number of elements in the historical data table, then return to step S402.

[0121] Otherwise, proceed to step S411; Step S411: Filter out elements in the list FusionList whose confidence field is 0 and elements whose FusionDistance field is -1, to obtain the remaining elements in the list FusionList.

[0122] Furthermore, the specific process of step S5 is as follows:

[0123] Step S501: Establish a hierarchical early warning decision model, WarningModel. WarningModel is a random forest classifier. The input features of WarningModel include... , , and ;

[0124] The output of the WarningModel model is a multi-class decision, specifically the value of the SafetyDecision field.

[0125] In this context, a value of 0 in the SafetyDecision field indicates a safe state, a value of 1 in the SafetyDecision field indicates a Level 1 warning state, and a value of 2 in the SafetyDecision field indicates a Level 2 warning state.

[0126] Step S502: Train the model WarningModel using the remaining data in the list FusionList.

[0127] Furthermore, the specific process of step S6 is as follows:

[0128] Step S601: Collect multi-source sensor data TestData in real time;

[0129] Step S602: Use the FusionModel module to perform fusion processing on the collected data to obtain the real-time fusion result RealTimeFusionResult;

[0130] Step S603: Determine whether the value of the FusionDistance field in the RealTimeFusionResult is -1;

[0131] If the FusionDistance field value is not -1, then , , and As the output of the trained model WarningModel, it is used to determine whether a near-electric hazard has occurred.

[0132] If the FusionDistance field value is -1, then proceed to step S604;

[0133] Step S604: Determine whether there are invalid values ​​in the remaining fields of the RealTimeFusionResult;

[0134] If no invalid values ​​are found in the remaining fields, the highest level alert will be triggered immediately.

[0135] If the remaining fields contain invalid values, proceed to step S605;

[0136] Step S605: Make a judgment based on the data collected by the electric field sensor:

[0137] If E total > E critical If so, a Level II warning will be issued;

[0138] If E critical ≥E total > E warning If so, a Level 1 warning will be issued;

[0139] Otherwise, the decision is that no danger of near-electricity has occurred.

[0140] Furthermore, the method uses an edge computing unit as the core processor, and all sensors are connected to the edge computing unit via gigabit Ethernet and powered and synchronized via PoE.

[0141] The beneficial effects of this invention are:

[0142] This invention employs a hierarchical processing flow of electric field guidance, visual locking, point cloud ranging, and cross-verification, enabling the system to efficiently and reliably lock onto charged targets. The electric field sensor first provides azimuth guidance, avoiding extensive target searches by visual and lidar systems, significantly reducing system computational requirements and false alarm rates. The visual camera and lidar precisely lock and range within the defined area guided by the electric field, and achieve physical-level mutual verification through a cross-verification mechanism. This design constructs redundant safety assurance based on different physical principles, enabling more effective monitoring of near-electric behavior and achieving accurate monitoring of near-electric behavior, thus making the safe operating distance measurable, visible, and controllable. Attached Figure Description

[0143] Figure 1 This is a flowchart of a near-electric behavior monitoring method based on cross-modal stereo sensing according to the present invention. Detailed Implementation

[0144] Specific implementation method one: Combining Figure 1 This embodiment describes a method for monitoring near-electrical behavior based on cross-modal stereo sensing. The method specifically includes the following steps:

[0145] Step S1: Deploy multi-source sensors on the construction machinery, establish a historical data table SenList based on the historical data collected by the multi-source sensors, and initialize the cross-modal fusion feature list FusionList;

[0146] The multi-source sensors include a power frequency electric field sensor array, a binocular vision camera, and a lidar.

[0147] Step S2: Establish the electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the power frequency electric field sensor array EFInput, and the output of the module DirectionCalibrationModel is the charged body azimuth array EFOuput.

[0148] Step S3: Establish a cross-modal data fusion and target recognition module FusionModel. The input of module FusionModel is Lidar point cloud data LidarData, binocular vision image data ImageData, and array EFInput. Module FusionModel uses module DirectionCalibrationModel to process array EFInput. The output of module FusionModel is cross-modal fusion result FusionOutput.

[0149] Step S4: Use the FusionModel module to process each group of data in the historical data table SenList, and store the processing results in the list FusionList;

[0150] Step S5: Establish a hierarchical early warning decision model, WarningModel, and train the model using the data in the list FusionList;

[0151] Step S6: Collect multi-source sensor data TestData in real time using sensors, and use the FusionModel module and the WarningModel model to determine whether a near-electric hazard occurs. If a near-electric hazard occurs, an early warning needs to be issued; otherwise, no action is required.

[0152] Specific Implementation Method Two: This implementation method is a further limitation of Specific Implementation Method One. The specific process of step S1 is as follows:

[0153] Step S101: Install a power frequency electric field sensor array, a binocular vision camera, and a lidar on the top of the boom of the construction machinery;

[0154] Furthermore, spatiotemporal joint calibration of the binocular vision camera and LiDAR was performed to obtain the camera intrinsic parameters and the transformation relationship between the LiDAR coordinate system and the binocular vision camera coordinate system.

[0155] Step S102: Under various typical operating scenarios and different safety distances, use the power frequency electric field sensor array, binocular vision camera and lidar to synchronously collect time-series data, take the synchronously collected data as a set of data, and use a total station or high-precision laser rangefinder to synchronously obtain the actual distance between the geometric center of the power frequency electric field sensor array and the target energized equipment.

[0156] The acquired data forms a historical data table SenList, where each set of data is an element. The SenList table contains four fields:

[0157] Lidar point cloud data: A collection containing tens of thousands of 3D coordinate points, for example, points... The coordinates are as The data was obtained by scanning a 70° cone-shaped area in front using a lidar at a rate of 240,000 points / second.

[0158] Binocular vision image data: contains two color images, each with a resolution of 1280×720, simultaneously acquired by the left and right cameras;

[0159] Electric field sensor data EFieldData: An array containing 4 floating-point numbers, the array includes... , , , and These represent the electric field intensity values ​​in the four directions: top, front, left, and right, respectively.

[0160] SafetyLabel: A floating-point number representing the actual distance D between the geometric center of the electric field sensor array and the target charged device. true ;

[0161] Step S103: Record the number of elements in the historical data table SenList as SenNum;

[0162] Step S104: Create an empty cross-modal fusion feature list FusionList. Each element of the FusionList contains 5 fields:

[0163] Orientation of charged body: a tuple containing horizontal and vertical azimuth angles;

[0164] LidarDistance: A floating-point number representing the nearest distance D measured by the lidar. lidar ;

[0165] Binocular vision distance feature VisionDistance: A floating-point number representing the nearest distance D measured by binocular vision. vision ;

[0166] Final FusionDistance: A floating-point number representing the final distance D after fusion verification. fusion ;

[0167] SafetyDecision: An integer variable used to express the warning level;

[0168] Step S105: Initialize all five fields of the elements in the FusionList to be 0;

[0169] Step S106: Through parameter calibration, obtain the consistency threshold Th1, deviation threshold Th2, voxel downsampling side length, and electric field inverse calculation coefficient values.

[0170] Step S107, Step S1 ends.

[0171] The other steps and parameters are the same as in Specific Implementation Method 1.

[0172] The power frequency electric field sensor array consists of a signal conditioning circuit and four hemispherical metal electrode probes. The array is fixed with an insulating bracket, precisely oriented in four directions: up, front, left, and right. A binocular vision camera is mounted on the top of the crane cab via a universal bracket, ensuring a panoramic view of the entire work area. The lidar has a scanning speed of no less than 240,000 points per second and a field of view of no less than 70°. The lidar and binocular camera module are installed side-by-side, ensuring full overlap between their scanning cone angle and the camera's field of view. The device's installation structure is designed to accommodate both magnets and clamps for easy and secure installation, meeting the requirements for rapid installation on the crane boom during field use.

[0173] The specific process of step S106 is described below:

[0174] Perform sensor consistency calibration: Place a standard reflector at a known distance from the sensor array and simultaneously acquire LiDAR and binocular vision data N times. Calculate the difference sequence between the LiDAR distance measurement and the actual distance, and the difference sequence between the binocular vision distance value and the actual distance, in the N measurements. Then calculate the joint standard deviation of the two difference sequences. Then set the consistency threshold Th1 equal to Set the deviation threshold Th2 equal to ;

[0175] Perform voxel grid size calibration: based on the nominal ranging accuracy of the lidar. Set the side length for voxel downsampling ;

[0176] Configure electric field inverse calculation coefficient values: Based on the voltage level and typical weather conditions at the work site, load the corresponding electric field inverse calculation coefficient values ​​from the system's built-in database.

[0177] Specific Implementation Method Three: This implementation method is a further limitation of Specific Implementation Method Two. The specific process of step S2 is as follows:

[0178] Step S201: Establish an electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the electric field sensor array EFInput, and the output is the charged body azimuth array EFOuput. Step S202: Calculate the electric field composition index and azimuth index based on the array EFInput: (1) Composite index of electric field level :

[0179]

[0180] (2) Vertical composite index of electric field :

[0181]

[0182] (3) Horizontal azimuth radian index :

[0183]

[0184] Where atan2 represents the arctangent function in the four quadrants;

[0185] (4) Vertical azimuth radian index :

[0186]

[0187] (5) Horizontal azimuth index :

[0188]

[0189] (6) Vertical azimuth index :

[0190]

[0191] in, , , and All are conversion factors, obtained through factory calibration;

[0192] Step S203: Create a charged body azimuth array EFOuput with dimension 2; Step S204, calculate the result in S202 and Stored in the array EFOuput; Step S205: The array EFOuput is used as the output of the module DirectionCalibrationModel.

[0193] The other steps and parameters are the same as in Specific Implementation Method Two.

[0194] The conversion factor is determined by the following calibration method:

[0195] An electric field sensor array is placed in a uniform, strong electric field. Four hemispherical electrodes are sequentially aligned with the electric field direction, and the horizontal and vertical radian measurements are recorded each time. The four horizontal radian measurements are sorted in ascending order, and the minimum value in the sorting is taken as the minimum value. (Horizontal offset), and calculate the difference between the maximum and minimum values, then divide 90 degrees by this difference to obtain the result. (Horizontal scaling factor).

[0196] Similarly, sort the four vertical radians in ascending order, and take the minimum value in the sorting results as... (Vertical offset), and calculate the difference between the maximum and minimum values, then divide 90 degrees by this difference to obtain the result. (Vertical scaling factor). The above calibration process can be completed at the factory or performed as a field calibration step.

[0197] Specific Implementation Method Four: This implementation method is a further limitation of Specific Implementation Method Three. The specific process of step S3 is as follows:

[0198] Step S301: Establish a cross-modal data fusion and target locking module FusionModel. Define the input of module FusionModel as LidarData, ImageData and array EFInput, and define the output of module FusionModel as cross-modal fusion result FusionOutput.

[0199] The DirectionCalibrationModel module is called to process the EFInput array;

[0200] Step S302: The first temporary variable FusionResult is assigned the result of processing the array EFInput using the module DirectionCalibrationModel. The processing result includes the horizontal azimuth angle. and vertical azimuth This directional guidance information is used to limit the target search area in subsequent steps;

[0201] Step S303: Perform a voxel downsampling algorithm on LidarData to generate a downsampled point cloud;

[0202] Based on the transformation relationship between the lidar coordinate system and the binocular vision camera coordinate system, each point in the downsampled point cloud is transformed to the camera coordinate system to achieve coordinate alignment between the point cloud data and the image data.

[0203] Based on horizontal azimuth and vertical azimuth A three-dimensional view frustum is defined with the geometric center of the electric field sensor as its vertex, and the horizontal angular range of the three-dimensional view frustum is... The vertical angle range is The depth ranges from 2 meters to 30 meters. The point cloud that is transformed to the camera coordinate system and located inside the 3D view frustum is used as the candidate point cloud; the four corner points of the 3D view frustum are projected onto the image plane according to the camera intrinsic parameters, and the quadrilateral region formed by the four projection points is used as the ROI region in the image.

[0204] The image here refers to the fused image captured by the left and right cameras;

[0205] Step S304: Input the ROI region in the image into the YOLOv5-lite model, and output the bounding box coordinates, the category of charged body in the bounding box, and the confidence score of the bounding box through the YOLOv5-lite model.

[0206] The backbone network of the YOLOv5-lite model is CSPDarknet53, the feature aggregation network is PANet, the initial learning rate is 0.01, the batch size is 32, the total number of training iterations is 300,000, and the loss function is CIoU Loss.

[0207] If the highest confidence level is higher than 0.6, then the center point of the bounding box with the highest confidence level is used as the pixel coordinate of the charged body, and the coordinates of the charged body on the image are marked as follows: And continue with step S307;

[0208] Otherwise, proceed to step S305;

[0209] Step S305, Point cloud rollback mechanism for failed visual detection:

[0210] The candidate point cloud is clustered using the DBSCAN clustering algorithm to obtain the point cloud cluster closest to the geometric center of the electric field sensor array. The centroid of the obtained point cloud cluster is used as the candidate three-dimensional position of the charged body, and then step S306 is executed.

[0211] In this invention, the neighborhood radius of the clustering algorithm DBSCAN is set to 0.2 meters, and the minimum number of points is set to 5.

[0212] Step S306, the secondary verification and attribute analysis of the rollback mechanism, specifically includes:

[0213] Step 1: Azimuth Verification

[0214] The calculated azimuth angle of the centroid of the point cloud cluster relative to the geometric center of the electric field sensor Then compare with the azimuth information Relationship:

[0215] like If the target recognition fails, the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values. Then, the process jumps directly to step S310. Otherwise, the azimuth check passes, and proceed to step 2; Step 2: Attribute analysis. By performing attribute analysis on the point cloud clusters, it is determined whether they are typical charged bodies such as wires.

[0216] The linearity of the obtained point cloud clusters was determined using principal component analysis (PCA). And compare linearity Size relative to threshold 0.8:

[0217] If linearity If the target does not conform to the wire feature, it is regarded as an unknown obstacle. The output array FusionOutput corresponding to the current group of data is marked as an unknown obstacle. The FusionDistance field in the output FusionOutput corresponding to the current group of data is set to -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FusionDistance field in FusionOutput are all marked as invalid values, and the process jumps directly to step S310.

[0218] If linearity If so, the confidence level for it to be a conductor is increased, and step 3 is continued;

[0219] It should be noted that linearity is the ratio of the projection variance of the point cloud cluster along the principal direction to the total variance.

[0220]

[0221] in, and These are the variances of the point cloud projections onto the planes in the principal direction and perpendicular to the principal direction, respectively.

[0222] Step 3: Reflection Intensity Analysis

[0223] The average laser reflection intensity of the point cloud cluster is calculated, and then compared with a preset feature library of power equipment reflection intensity. If a match is successful, the target is confirmed as a charged object, and the pixel coordinates of the charged object in the image are determined based on the candidate 3D position of the charged object. ;

[0224] And proceed to step S307; If the matching fails, the target is determined to be an unknown obstacle, and the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1, and the LidarDistance field, the binocular visual distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values; and the process jumps directly to step S310.

[0225] Step S307: Calculate the visual distance between the lidar and the binocular camera, and then execute step S308;

[0226] The calculation process for the visual distance between the lidar and the binocular camera is as follows:

[0227] Based on the camera intrinsic parameters, all downsampled point clouds transformed to the camera coordinate system are back-projected onto the image plane, and the points falling into the image plane are determined by their positions. The back-projected pixels within a square region with a side length of 10 pixels centered on the electric field sensor array are stored in a candidate set. The nearest distance from all point clouds in the candidate set to the geometric center of the electric field sensor array is calculated, and the obtained nearest distance is taken as the lidar range. ;

[0228] According to the camera focal length and baseline distance Calculate the visual distance of the binocular camera :

[0229]

[0230] in, Indicates parallax; both cameras have a focal length of [missing information]. ;

[0231] Step S308: Construct the fused output structure

[0232] Create a cross-modal fusion output, FusionOutput, with the same structure as the elements in the list FusionList, and assign the azimuth information to the Orientation field of FusionOutput. Assign the value to the LidarDistance field. Assign the value to the VisionDistance field. ; Step S309, based on the consistency threshold pair and Cross-verification and fusion are performed to determine the final fusion distance, FusionDistance; specifically: Step (1): Calculate the distance difference ; like If the data is consistent, the FusionDistance field of FusionOutput will be assigned a value. ; Step (2), if If the data is found to be biased, a weighted fusion calculation will be used. :

[0233]

[0234] in, and All are weighting coefficients;

[0235] The weighting coefficients are adaptively adjusted according to the environment: if the average brightness of the pixels in the ROI region of the image is less than 20 Lux, the weight of the LiDAR is 0.7 and the weight of the visual system is 0.3; if the average brightness of the pixels in the ROI region of the image is not less than 20 Lux, it is determined whether the point cloud density in the ROI region is less than 100 points / cubic meter. If it is less than 100 points / cubic meter, the weight of the LiDAR is 0.4 and the weight of the visual system is 0.6. If it is not less than 100 points / cubic meter, both weights are 0.5.

[0236] Assign a value to the FusionDistance field of FusionOutput. ;

[0237] Step (3), if If a data conflict is detected, the distance is calculated by back-calculating the electric field. :

[0238]

[0239] in, The total electric field strength is ;

[0240] Calculate the dynamic fault tolerance threshold :

[0241]

[0242] in, , and All are constants. As environmental factors, For the theoretical electric field strength, The variance of the electric field data;

[0243] In this invention, The value is 0.3m. The value is 0.1 m / (kV / m)². The value is 0.05 m / (kV / m), environmental factor Defined as follows: 0 for sunny days, 0.2m for light rain or fog, and 0.4m for heavy rain.

[0244]

[0245] in, For the field voltage level, The equivalent radius of the line. The same method as step (2) is used for calculation; Step (4): Execute the final arbitration.

[0246] like If the conflict is deemed interpretable, the FusionDistance field will be assigned a value. ;

[0247] Otherwise, mark the FusionDistance field as invalid -1 and immediately trigger the highest level alert;

[0248] Step S310: Output the fusion result.

[0249] The other steps and parameters are the same as in Specific Implementation Method 3.

[0250] Specific Implementation Method Five: This implementation method is a further limitation of Specific Implementation Method Four. The specific process of step S303 is as follows:

[0251] The three-dimensional space is divided into cubic voxels, and the three-dimensional points contained within each cubic voxel are determined. Then, for any cubic voxel, the representative point of the cubic voxel is determined based on the point cloud within the cubic voxel:

[0252]

[0253] in, This represents the set of all points contained within the cube voxel. Represents a set The number of points inside, For set The first The coordinates of the points The coordinates of the representative point of the cube voxel;

[0254] A downsampling point cloud is constructed using representative points of each cubic voxel.

[0255] The other steps and parameters are the same as in Specific Implementation Method Four.

[0256] Specific Implementation Method Six: This implementation method is a further limitation of Specific Implementation Method Five. The specific process of step S304 is as follows:

[0257] Points in the lidar coordinate system Transform to the camera coordinate system to obtain the point :

[0258]

[0259] Where R represents a rotation matrix of dimension 3×3, and T represents a translation vector of dimension 3×1.

[0260] The other steps and parameters are the same as in Specific Implementation Method 5.

[0261] Specific Implementation Method Seven: This implementation method is a further limitation of Specific Implementation Method Six. The specific process of step S4 is as follows:

[0262] Step S401: Initialize the processing counter ProcCounter=1;

[0263] Step S402: Assign the first temporary variable ProcessedData to the ProcCounter element of SenList;

[0264] Step S403: The cross-modal data fusion and target recognition module FusionModel is called to process the ProcCounter element in the list SenList. The output of the module is the real-time fusion result RealTimeFusionResult, which is a structure with the same structure as the element in the list FusionList. Step S404: Assign the values ​​of each field of RealTimeFusionResult to the ProcCounter element in the list FusionList. The fields to be assigned include: Orientation of charged body, LidarDistance, VisionDistance, and FusionDistance. Step S405, verify the value of the FusionDistance field of RealTimeFusionResult:

[0265] If the FusionDistance field value is -1, it means that the target recognition failed or was identified as an unknown obstacle, and the process will jump directly to S410; otherwise, continue to execute step S406. Step S406, calculate the error between the fused distance and the actual distance.

[0266]

[0267] Step S407: Determine the data confidence level based on the Error value and add the Confidence field to the ProcCounter element of the FusionList;

[0268] If the error is greater than 0.5 meters, the confidence flag for the current data is set to 0, indicating low confidence.

[0269] Otherwise, the confidence flag for the current data is set to 1, indicating high confidence. Step S408: Compare the SafetyLabel with a preset safety distance threshold, generate a graded safety decision label based on the comparison result, and assign the generated graded safety decision label to the SafetyDecision field of the ProcCounter element of the FusionList; specifically:

[0270] The safe distance threshold is loaded from the system configuration file based on the voltage level of the work site. The configuration file contains the following mapping relationship:

[0271] 10kV:D safe = 1.5m, D warning = 2.0m;

[0272] 220kV:D safe = 4.0m, D warning = 5.0m;

[0273] 500kV:D safe = 6.0m, D warning = 7.0m;

[0274] If D true < D safe If so, the SafetyDecision field will be assigned the value 2;

[0275] If D safe ≤ D true < D warning If so, the SafetyDecision field will be assigned the value 1;

[0276] Otherwise, the SafetyDecision field is set to 0; Step S409: Increment the value of the processing counter ProcCounter by 1; Step S410, determine the loop condition:

[0277] If ProcCounter is less than or equal to the total number of elements in the historical data table, then return to step S402.

[0278] Otherwise, proceed to step S411; Step S411: Filter out elements in the list FusionList whose confidence field is 0 and whose FusionDistance field is -1, to obtain the remaining elements in the list FusionList.

[0279] The other steps and parameters are the same as in Specific Implementation Method Six.

[0280] Specific Implementation Method Eight: This implementation method is a further limitation of Specific Implementation Method Seven. The specific process of step S5 is as follows:

[0281] Step S501: Establish a hierarchical early warning decision model, WarningModel. WarningModel is a random forest classifier. The input features of WarningModel include... , , and ;

[0282] The output of the WarningModel model is a multi-class decision, specifically the value of the SafetyDecision field.

[0283] In this context, a value of 0 in the SafetyDecision field indicates a safe state, a value of 1 in the SafetyDecision field indicates a Level 1 warning state, and a value of 2 in the SafetyDecision field indicates a Level 2 warning state.

[0284] The hyperparameters of the WarningModel model are set as follows: 100 decision trees, 10 maximum depth, 2 minimum leaf node samples, and the model supports online incremental updates.

[0285] Step S502: Train the model WarningModel using the remaining data in the list FusionList.

[0286] The other steps and parameters are the same as in Specific Implementation Method Seven.

[0287] It should be noted that this invention uses a 5-fold cross-validation method for training.

[0288] Specific Implementation Method Nine: This implementation method is a further limitation of Specific Implementation Method Eight. The specific process of step S6 is as follows:

[0289] Step S601: Collect multi-source sensor data TestData in real time;

[0290] Step S602: Use the FusionModel module to perform fusion processing on the collected data to obtain the real-time fusion result RealTimeFusionResult;

[0291] Step S603: Determine whether the value of the FusionDistance field in the RealTimeFusionResult is -1;

[0292] If the FusionDistance field value is not -1, then , , and As the output of the trained model WarningModel, it is used to determine whether a near-electric hazard has occurred.

[0293] If a near-electrical hazard occurs, a tiered early warning strategy will be implemented;

[0294] When FinalDecision is set to "Level 1 Warning", the driver's cabin audible and visual alarm and seat vibration alert will be triggered.

[0295] When FinalDecision is set to "Level 2 Warning", a lockout signal is sent to the control system of the construction machinery to force it to stop dangerous actions.

[0296] If no near-electric shock hazard is detected, the system will continue to monitor.

[0297] If the FusionDistance field value is -1, then proceed to step S604;

[0298] Step S604: Determine whether there are invalid values ​​in the remaining fields of the RealTimeFusionResult;

[0299] If no invalid values ​​are found in the remaining fields, the highest level alert will be triggered immediately.

[0300] If the remaining fields contain invalid values, proceed to step S605;

[0301] Step S605: Set E based on experiencecritical and E warning The value is determined based on the data collected by the electric field sensor.

[0302] If E total > E critical If so, a Level II warning will be issued;

[0303] If E critical ≥E total > E warning If so, a Level 1 warning will be issued;

[0304] Otherwise, the decision is that no danger of near-electricity has occurred.

[0305] The other steps and parameters are the same as in Specific Implementation Method 8.

[0306] Specific Implementation Method Ten: This implementation method is a further limitation of Specific Implementation Method Nine. The method uses an edge computing unit as the core processor, and all sensors are connected to the edge computing unit through Gigabit Ethernet and are powered and synchronized with data via PoE.

[0307] The other steps and parameters are the same as in Specific Implementation Method Nine.

[0308] The above examples of the present invention are merely illustrative of the computational model and process of the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is impossible to exhaustively list all possible implementations here. Any obvious variations or modifications derived from the technical solutions of the present invention are still within the scope of protection of the present invention.

Claims

1. A method for monitoring near-electrical behavior based on cross-modal stereo sensing, characterized in that, The method specifically includes the following steps: Step S1: Deploy multi-source sensors on the construction machinery, establish a historical data table SenList based on the historical data collected by the multi-source sensors, and initialize the cross-modal fusion feature list FusionList; The multi-source sensors include a power frequency electric field sensor array, a binocular vision camera, and a lidar. Step S2: Establish the electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the power frequency electric field sensor array EFInput, and the output of the module DirectionCalibrationModel is the charged body azimuth array EFOuput. Step S3: Establish a cross-modal data fusion and target recognition module FusionModel. The input of module FusionModel is Lidar point cloud data LidarData, binocular vision image data ImageData and array EFInput. The output of module FusionModel is the cross-modal fusion result FusionOutput. Step S4: Use the FusionModel module to process each group of data in the historical data table SenList, and store the processing results in the list FusionList; Step S5: Establish a hierarchical early warning decision model, WarningModel, and train the model using the data in the list FusionList; Step S6: Collect multi-source sensor data TestData in real time using sensors, and use the FusionModel module and the WarningModel model to determine whether a near-electric hazard occurs. If a near-electric hazard occurs, an early warning needs to be issued; otherwise, no action is required.

2. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 1, characterized in that, The specific process of step S1 is as follows: Step S101: Install a power frequency electric field sensor array, a binocular vision camera, and a lidar on the top of the boom of the construction machinery; Furthermore, spatiotemporal joint calibration of the binocular vision camera and LiDAR was performed to obtain the camera intrinsic parameters and the transformation relationship between the LiDAR coordinate system and the binocular vision camera coordinate system. Step S102: Use the power frequency electric field sensor array, binocular vision camera and lidar to collect time series data simultaneously, and use a total station or laser rangefinder to obtain the actual distance between the geometric center of the power frequency electric field sensor array and the target charged equipment simultaneously. Based on the acquired data, a historical data table SenList is created. The SenList table contains four fields: Lidar point cloud data (LidarData): A collection of three-dimensional coordinate points; Binocular vision image data: contains two color images, each with a resolution of 1280×720, simultaneously acquired by the left and right cameras; Electric field sensor data EFieldData: An array containing 4 floating-point numbers, the array includes... , , , and These represent the electric field intensity values ​​in the four directions: top, front, left, and right, respectively. SafetyLabel: A floating-point number representing the actual distance D between the geometric center of the electric field sensor array and the target charged device. true ; Step S103: Record the number of elements in the historical data table SenList as SenNum; Step S104: Create an empty cross-modal fusion feature list FusionList. Each element of the FusionList contains 5 fields: Orientation of charged body: a tuple containing horizontal and vertical azimuth angles; LidarDistance: A floating-point number representing the nearest distance D measured by the lidar. lidar ; Binocular vision distance feature VisionDistance: A floating-point number representing the nearest distance D measured by binocular vision. vision ; Final FusionDistance: A floating-point number representing the final distance D after fusion verification. fusion ; SafetyDecision: An integer variable used to express the warning level; Step S105: Initialize all five fields of the elements in the FusionList to be 0; Step S106: Through parameter calibration, obtain the consistency threshold Th1, deviation threshold Th2, voxel downsampling side length, and electric field inverse calculation coefficient values. Step S107, Step S1 ends.

3. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 2, characterized in that, The specific process of step S2 is as follows: Step S201: Establish an electric field direction calculation and data preprocessing module DirectionCalibrationModel. The input of the module DirectionCalibrationModel is the electric field sensor array EFInput, and the output is the charged body azimuth array EFOuput. Step S202: Calculate the electric field synthesis index and azimuth index based on the array EFInput: (1) Electric field horizontal synthesis index : (2) Vertical composite index of electric field : (3) Horizontal azimuth radian index : Where atan2 represents the arctangent function in the four quadrants; (4) Vertical azimuth radian index : (5) Horizontal azimuth index : (6) Vertical azimuth index : in, , , and All are conversion factors; Step S203: Create a charged body azimuth array EFOuput with dimension 2; Step S204, calculate the result in S202 and Stored in the array EFOuput; Step S205: The array EFOuput is used as the output of the module DirectionCalibrationModel.

4. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 3, characterized in that, The specific process of step S3 is as follows: Step S301: Establish a cross-modal data fusion and target locking module FusionModel. Define the input of module FusionModel as LidarData, ImageData and array EFInput, and define the output of module FusionModel as cross-modal fusion result FusionOutput. The DirectionCalibrationModel module is called to process the EFInput array; Step S302: The first temporary variable FusionResult is assigned the result of processing the array EFInput using the module DirectionCalibrationModel. The processing result includes the horizontal azimuth angle. and vertical azimuth ; Step S303: Perform a voxel downsampling algorithm on LidarData to generate a downsampled point cloud; Based on the transformation relationship between the lidar coordinate system and the binocular vision camera coordinate system, each point in the downsampled point cloud is transformed to the camera coordinate system; Based on horizontal azimuth and vertical azimuth A three-dimensional view frustum is defined with the geometric center of the electric field sensor as its vertex, and the horizontal angular range of the three-dimensional view frustum is... The vertical angle range is ; The point cloud that is transformed to the camera coordinate system and located inside the 3D view frustum is used as the candidate point cloud; the four corner points of the 3D view frustum are projected onto the image plane according to the camera intrinsic parameters, and the quadrilateral region formed by the four projection points is used as the ROI region in the image. Step S304: Input the ROI region in the image into the YOLOv5-lite model, and output the bounding box coordinates, the category of charged body in the bounding box, and the confidence score of the bounding box through the YOLOv5-lite model. If the highest confidence level is higher than 0.6, then the center point of the bounding box with the highest confidence level is used as the pixel coordinate of the charged body, and the coordinates of the charged body on the image are marked as follows: And continue with step S307; Otherwise, proceed to step S305; Step S305, Point cloud rollback mechanism for failed visual detection: The candidate point cloud is clustered using the DBSCAN clustering algorithm to obtain the point cloud cluster closest to the geometric center of the electric field sensor array. The centroid of the obtained point cloud cluster is used as the candidate three-dimensional position of the charged body, and then step S306 is executed. Step S306, the secondary verification and attribute analysis of the rollback mechanism, specifically includes: Step 1: Azimuth Verification The calculated azimuth angle of the centroid of the point cloud cluster relative to the geometric center of the electric field sensor Then compare with the azimuth information Relationship: like If the target recognition fails, the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values. Then, the process jumps directly to step S310. Otherwise, the azimuth check passes, and proceed to step 2; Step 2, Attribute Analysis; The linearity of the obtained point cloud clusters was determined using principal component analysis (PCA). And compare linearity Size relative to threshold 0.8: If linearity If the target does not conform to the wire feature, it is regarded as an unknown obstacle. The output array FusionOutput corresponding to the current group of data is marked as an unknown obstacle. The FusionDistance field in the output FusionOutput corresponding to the current group of data is set to -1. The LidarDistance field, the binocular vision distance feature VisionDistance field, and the final fusion distance FusionDistance field in FusionOutput are all marked as invalid values, and the process jumps directly to step S310. If linearity If so, proceed to step 3; Step 3: Reflection Intensity Analysis The average laser reflection intensity of the point cloud cluster is calculated, and then compared with a preset feature library of power equipment reflection intensity. If a match is successful, the target is confirmed as a charged object, and the pixel coordinates of the charged object in the image are determined based on the candidate 3D position of the charged object. ; And proceed to step S307; If the matching fails, the target is determined to be an unknown obstacle, and the output FusionOutput corresponding to the current group of data is marked as locking failure. The FusionDistance field in the output FusionOutput corresponding to the current group of data is assigned a value of -1, and the LidarDistance field, the binocular visual distance feature VisionDistance field, and the final fusion distance FuDistance field in FusionOutput are all marked as invalid values; and the process jumps directly to step S310. Step S307: Calculate the visual distance between the lidar and the binocular camera, and then execute step S308; The calculation process for the visual distance between the lidar and the binocular camera is as follows: Based on the camera intrinsic parameters, all downsampled point clouds transformed to the camera coordinate system are back-projected onto the image plane, and the points falling into the image plane are determined by their positions. The back-projected pixels within a square region with a side length of 10 pixels centered on the electric field sensor array are stored in a candidate set. The nearest distance from all point clouds in the candidate set to the geometric center of the electric field sensor array is calculated, and the obtained nearest distance is taken as the lidar range. ; According to the camera focal length and baseline distance Calculate the visual distance of the binocular camera : in, Indicates parallax; Step S308: Construct the fused output structure Create a cross-modal fusion output, FusionOutput, with the same structure as the elements in the list FusionList, and assign the azimuth information to the Orientation field of FusionOutput. Assign the value to the LidarDistance field. Assign the value to the VisionDistance field. ; Step S309, based on the consistency threshold pair and Cross-verification and fusion are performed to determine the final fusion distance, FusionDistance; specifically: Step (1): Calculate the distance difference ; like If the data is consistent, the FusionDistance field of FusionOutput will be assigned a value. ; Step (2), if If the data is found to be biased, a weighted fusion calculation will be used. : in, and All are weighting coefficients; Assign a value to the FusionDistance field of FusionOutput. ; Step (3), if If a data conflict is detected, the distance is calculated by back-calculating the electric field. : in, The total electric field strength is ; Calculate the dynamic fault tolerance threshold : in, , and All are constants. As environmental factors, For the theoretical electric field strength, The variance of the electric field data; in, For the field voltage level, The equivalent radius of the line; Step (4): Perform final arbitration. like If the conflict is deemed interpretable, the FusionDistance field will be assigned a value. ; Otherwise, mark the FusionDistance field as invalid -1 and immediately trigger the highest level alert; Step S310: Output the fusion result.

5. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 4, characterized in that, The specific process of step S303 is as follows: The three-dimensional space is divided into cubic voxels, and the three-dimensional points contained within each cubic voxel are determined. Then, for any cubic voxel, the representative point of the cubic voxel is determined based on the point cloud within the cubic voxel: in, This represents the set of all points contained within the cube voxel. Represents a set The number of points inside, For set The first The coordinates of the points The coordinates of the representative point of the cube voxel; A downsampling point cloud is constructed using representative points of each cubic voxel.

6. The method for monitoring near-electrical behavior based on cross-modal stereo sensing according to claim 5, characterized in that, The specific process of step S304 is as follows: Points in the lidar coordinate system Transform to the camera coordinate system to obtain the point : Where R represents a rotation matrix of dimension 3×3, and T represents a translation vector of dimension 3×1.

7. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 6, characterized in that, The specific process of step S4 is as follows: Step S401: Initialize the processing counter ProcCounter=1; Step S402: Assign the first temporary variable ProcessedData to the ProcCounter element of SenList; Step S403: The cross-modal data fusion and target recognition module FusionModel is called to process the ProcCounter element in the list SenList. The output of the module is the real-time fusion result RealTimeFusionResult, which is a structure with the same structure as the element in the list FusionList. Step S404: Assign the values ​​of each field of RealTimeFusionResult to the ProcCounter element in the list FusionList. The fields to be assigned include: Orientation of charged body, LidarDistance, VisionDistance, and FusionDistance. Step S405, verify the value of the FusionDistance field of RealTimeFusionResult: If the FusionDistance field value is -1, it means that the target recognition failed or was identified as an unknown obstacle, and the process will jump directly to S410; otherwise, continue to execute step S406. Step S406, calculate the error between the fused distance and the actual distance. Step S407: Determine the data confidence level based on the Error value and add the Confidence field to the ProcCounter element of the FusionList; If the error is greater than 0.5 meters, the confidence flag for the current data is set to 0, indicating low confidence. Otherwise, the confidence flag for the current data is set to 1, indicating high confidence. Step S408: Compare the SafetyLabel with a preset safety distance threshold, generate a graded safety decision label based on the comparison result, and assign the generated graded safety decision label to the SafetyDecision field of the ProcCounter element of the FusionList; specifically: If D true < D safe If so, the SafetyDecision field will be assigned the value 2; If D safe ≤ D true < D warning If so, the SafetyDecision field will be assigned the value 1; Otherwise, the SafetyDecision field is set to 0; Step S409: Increment the value of the processing counter ProcCounter by 1; Step S410, determine the loop condition: If ProcCounter is less than or equal to the total number of elements in the historical data table, then return to step S402. Otherwise, proceed to step S411; Step S411: Filter out elements in the list FusionList whose confidence field is 0 and elements whose FusionDistance field is -1, to obtain the remaining elements in the list FusionList.

8. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 7, characterized in that, The specific process of step S5 is as follows: Step S501: Establish a hierarchical early warning decision model, WarningModel. WarningModel is a random forest classifier. The input features of WarningModel include... , , and ; The output of the WarningModel model is a multi-class decision, specifically the value of the SafetyDecision field. In this context, a value of 0 in the SafetyDecision field indicates a safe state, a value of 1 in the SafetyDecision field indicates a Level 1 warning state, and a value of 2 in the SafetyDecision field indicates a Level 2 warning state. Step S502: Train the model WarningModel using the remaining data in the list FusionList.

9. The near-electric behavior monitoring method based on cross-modal stereo sensing according to claim 8, characterized in that, The specific process of step S6 is as follows: Step S601: Collect multi-source sensor data TestData in real time; Step S602: Use the FusionModel module to perform fusion processing on the collected data to obtain the real-time fusion result RealTimeFusionResult; Step S603: Determine whether the value of the FusionDistance field in the RealTimeFusionResult is -1; If the FusionDistance field value is not -1, then , , and As the output of the trained model WarningModel, it is used to determine whether a near-electric hazard has occurred. If the FusionDistance field value is -1, then proceed to step S604; Step S604: Determine whether there are invalid values ​​in the remaining fields of the RealTimeFusionResult; If no invalid values ​​are found in the remaining fields, the highest level alert will be triggered immediately. If the remaining fields contain invalid values, proceed to step S605; Step S605: Make a judgment based on the data collected by the electric field sensor: If E total > E critical If so, a Level II warning will be issued; If E critical ≥E total > E warning If so, a Level 1 warning will be issued; Otherwise, the decision is that no near-electricity hazard has been detected.

10. A method for monitoring near-electrical behavior based on cross-modal stereo sensing according to claim 9, characterized in that, The method uses an edge computing unit as the core processor, and all sensors are connected to the edge computing unit via gigabit Ethernet and powered and synchronized via PoE.

Images