Guard board posture estimation method and system based on 4d millimeter wave radar and infrared image

Abstract

A kind of fender posture estimation method and system based on 4D millimeter wave radar and infrared image, method: synchronously obtain infrared image and 4D millimeter wave radar point cloud, construct double-branch joint detection network;Image branch extracts fender semantic features and is mapped into BEV feature, radar branch is encoded to point cloud and generates radar BEV pseudo-image feature, after fusion, output target center, size, orientation, two-dimensional posture skeleton and fusion confidence;Utilize posture skeleton to constrain radar point cloud matching and attribution, when point cloud is sparse or missing, through template deformation, time sequence memory and graph structure completion, generate the structured completion point with confidence;Construct joint optimization target, solve fender three-dimensional posture and space occupation, output credibility evaluation and abnormality judgment.The system: through the modularization architecture of dual-mode acquisition, double-branch detection, constraint matching, structured completion, optimization solution, credibility evaluation.The present application can realize the high-precision, high-robustness estimation of fender posture.

Description

Technical Field

[0001] This invention belongs to the technical fields of intelligent sensing in coal mines, hydraulic support status monitoring, 4D millimeter-wave radar point cloud processing, infrared image recognition, and multimodal BEV fusion detection. Specifically, it relates to a method and system for estimating the attitude of a side support plate based on 4D millimeter-wave radar and infrared images. Background Technology

[0002] During fully mechanized mining operations, the hydraulic support sidewall plates are primarily used to support the coal face. The deployment, retraction, partial deployment, swaying, and jamming states of the sidewall plates directly determine the operational safety of the coal mining machine drum. If the sidewall plates are not retracted in time or exhibit abnormal posture, their free ends can easily intrude into the movement space of the coal mining machine drum, causing cutting interference problems. This can not only damage the equipment structure but also lead to the interruption of underground production operations. Therefore, accurately, in real-time, and stably acquiring the spatial position and posture parameters of the sidewall plates is a core key to ensuring safe and efficient production in fully mechanized mining faces.

[0003] Currently, the industry mainly uses image recognition technology to detect sidewall targets, which can effectively identify sidewalls using infrared or visible light images. However, this technology has certain shortcomings. Monocular images cannot output reliable absolute depth information, and factors such as thermal noise, equipment obstruction, and camera installation angle deviations under complex downhole conditions can significantly reduce the stability and accuracy of the detection results.

[0004] 4D millimeter-wave radar is adaptable to harsh working environments such as dust, water mist, and low light in underground mines. It can output multi-dimensional information such as the target's three-dimensional coordinates, movement speed, and reflection intensity, effectively compensating for the shortcomings of image recognition technology. However, in underground applications, the point cloud data collected by radar is generally sparse, and the echo signals from multiple working face structures such as side panels, roof beams, support columns, and coal walls are mixed together. This results in semantic information loss and local point cloud gaps when relying solely on radar point cloud data for side panel attitude recovery, making it impossible to accurately determine the true attitude of the side panels.

[0005] Current mainstream fusion detection solutions mostly rely on image detection boxes or image segmentation regions to filter and extract radar point cloud data, and then use point cloud plane fitting algorithms to solve the 3D attitude of the side guard plate. Although this type of solution can narrow the point cloud search range by leveraging image semantic information and improve data matching efficiency to some extent, it is prone to problems such as point cloud mismatch and frequent jitter in attitude detection results under conditions such as scarce point cloud quantity, missing free point cloud of the side guard plate, and image detection boxes containing a large amount of background point cloud. The detection accuracy and stability are difficult to meet the requirements of downhole safe production. In summary, there is an urgent need to provide a new detection technology solution that combines image and radar BEV joint detection with image attitude prior constraints to achieve point cloud matching, completion, and 3D correction, and accurately and stably obtain the attitude information of the side guard plate. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a method and system for estimating the attitude of the sidewall panel based on 4D millimeter-wave radar and infrared images. This method achieves high-precision and robust estimation of the sidewall panel attitude under underground conditions of obstruction, dust, and sparse point clouds, providing a reliable sensing foundation for sidewall panel state perception and interference prevention in coal mining faces. The system deeply integrates infrared visual semantics with radar spatial geometry in BEV space, enabling stable estimation and anomaly warning of the sidewall panel attitude under complex underground conditions, providing a complete system-level solution for intelligent perception in fully mechanized mining faces.

[0007] To achieve the above objectives, this invention provides a method for estimating the attitude of a protective panel based on 4D millimeter-wave radar and infrared images, comprising the following steps: Step 1: Synchronously acquire multi-source data and perform time matching; synchronously acquire infrared image frames and 4D millimeter-wave radar point cloud frames in the target operation area, and perform data matching based on timestamps or reception times; Step 2: Construct a dual-branch joint detection network and perform multimodal feature extraction; Construct a dual-branch joint detection network containing image branch and radar branch, and perform feature extraction and position modeling on infrared image frames and radar point cloud frames respectively; Step 3: Perform BEV spatial feature fusion and output joint detection results; fuse image BEV features and radar BEV pseudo-image features in BEV space, and output the side guard plate target detection results, two-dimensional attitude skeleton and fusion confidence based on the fused features; Step 4: Project the radar point cloud onto the infrared image plane; Based on the pre-calibrated extrinsic parameters from the radar coordinate system to the infrared camera coordinate system and the intrinsic parameters of the infrared camera, project the radar point cloud onto the infrared image plane to establish the spatial correspondence between radar points and image pixels. Step 5: Match radar point cloud based on attitude skeleton constraint; using the attitude skeleton of the fused detection results and the target region of the image, perform attitude constraint matching on the radar points to obtain the set of measured radar points corresponding to each guard plate. Step 6: Structured completion of missing radar points; When the set of measured radar points does not meet the preset integrity conditions, a set of structured completion points is generated using the two-dimensional attitude skeleton, the guard plate size model, the set of measured radar points, and the historical attitude state. Step 7: Establish a joint attitude optimization objective and solve for the three-dimensional attitude; use the two-dimensional attitude skeleton, the set of measured radar points, the set of complete points, and the side guard plate size model to establish a joint attitude optimization objective and solve for the three-dimensional attitude state of the side guard plate. Step 8: Attitude result credibility assessment and anomaly judgment; calculate the attitude result credibility based on multi-dimensional quality indicators, and output the three-dimensional attitude, spatial occupancy and abnormal attitude judgment results of the guardrail.

[0008] As a preferred option, in step 1, the process of synchronously acquiring multi-source data and performing time matching is as follows: S11: Install an infrared camera at the rocker arm end of the coal mining machine or on the top of the hydraulic support, and at the same time, install a 4D millimeter-wave radar on the stable base adjacent to the infrared camera. S12: Infrared image frames and radar point cloud frames within the target operation area are acquired using an infrared camera and a 4D millimeter-wave radar, respectively. S13: Select matching infrared image frames and radar point cloud frames from the cache based on the timestamp or system reception time; when the infrared camera and radar do not have a unified hardware clock, use the system reception time of the data arriving at the same data processing device as the common time axis, and select the frame with the smallest absolute time difference as the matching frame; when the minimum time difference is greater than the preset maximum allowable time difference, terminate the current attitude estimation and output the data desynchronization status.

[0009] As a preferred embodiment, the process of constructing the dual-branch joint detection network and extracting multimodal features in step 2 is as follows: S21: Construct a dual-branch joint detection network; the image branch uses a YOLO network, and multi-level features of the infrared image are extracted through the backbone network and the feature pyramid fusion layer. Then through the target detection head Obtain the detection bounding box, class label, and detection confidence score of the side panel, i.e., the image detection result. The attitude key point set of the guardrail is extracted using attitude key point extraction. ;in, For the current infrared image, For YOLO Layer image features, This represents the YOLO backbone and feature fusion operator, and the image detection results. Includes target bounding box, category, and detection confidence, as well as a set of image pose keypoints. This includes at least two types of attitude structures among the key points of the hinged end of the side guard plate, key points of the free end, edge points of the plate surface, and centerline parameters of the plate surface. For the target detection head mapping function, This is the attitude key point mapping function; The radar branch employs a PointPillars network, dividing the point cloud into cylinders using a BEV grid. Point feature vectors are constructed for points within each cylinder, comprising at least three of the following: 3D coordinates, reflection intensity, radial velocity, offset relative to the cylinder center, and offset relative to the centroid of the point set. Cylinder features are extracted via a shared multilayer perceptron and a max-pooling layer. To achieve multimodal feature extraction of radar BEV pseudo-images; S22: Multimodal features aligned in BEV space; Image features Extrinsic parameter matrix from radar coordinate system to infrared camera coordinate system and camera projection function Converted into image BEV features within the radar BEV space. ,in The center point of the BEV element in the coordinate system. The coordinates of the column center are For the first Image BEV features of each BEV unit, Sample represents the fusion weights for image features at different scales, and it denotes the bilinear sampling operation. For each point pair in the radar point cloud Construct feature vectors ,in, Let the coordinates of the point be in three-dimensional space. Radial velocity, This refers to either reflection intensity or signal-to-noise ratio. The coordinates of the centroid of the point set within the cylinder; Radar BEV pseudo-image features are obtained through MLP and max pooling. ,in For the first Radar BEV pseudo-image features of a BEV cell.

[0010] As a preferred option, in step 3, the process of fusing BEV spatial features and outputting the joint detection results is as follows: S31: Extract BEV features from the image and radar BEV pseudo-image features Attention-weighted fusion is performed to obtain fused BEV features. ,in and These are the fusion weights for the image branch and the radar branch, respectively; This indicates a channel-wise weighted operation; S32: Based on integrated BEV features The detection head outputs the target detection results of the guardrail, the two-dimensional attitude skeleton, and the fusion confidence score, and the first... Complete test output of individual protective board ,in For the first The center coordinates of the protective side panel on the BEV plane, in meters; , , These are the length, width, and height dimensions of the side panel, in meters (m). The angle of the side guard plate on the BEV plane is expressed in rad. The two-dimensional posture skeleton of the side guard plate is composed of the key points connecting the hinged end, the free end, and the edge of the plate. The confidence level for fusion detection is set to [0,1]. This is the mapping function for the fusion detection head.

[0011] As a preferred embodiment, the process of projecting the radar point cloud onto the infrared image plane in step 4 is as follows: For the There are 1 radar point, whose homogeneous coordinates in the radar coordinate system are: According to the extrinsic parameter matrix Calculate its homogeneous coordinates in the infrared camera coordinate system. The projected pixel coordinates are calculated based on the infrared camera's intrinsic parameters and the pinhole imaging model, as shown in the following formula: ; ; In the formula, , These are the x and y coordinates of the projected pixels, respectively. , This refers to the focal length of the infrared camera in the horizontal and vertical directions of the image. , These are the x and y coordinates of the principal point of the infrared camera on the image plane. The depth of the radar point in the camera coordinate system; Only when And projected pixel coordinates If the radar point is within the effective image range, it is retained and proceeds to step 5 for attitude constraint matching calculation and determination of the actual radar point's affiliation; otherwise, the radar point is not involved in subsequent processing.

[0012] As a preferred option, in step 5, the process of matching the radar point cloud based on attitude skeleton constraints is as follows: S51: Regarding the first Individual care board and the first The attitude constraint matching score is determined based on several radar points, including whether the projection point is located in the target area of ​​the guardrail image, the distance from the projection point to the attitude skeleton, the consistency of the projection point along the direction from the hinge end to the free end, the consistency of the projection point with the historical spatial position, and the fusion detection confidence. As shown in the following formula: ; In the formula, The indicative function is the pixel coordinates of the radar point projection. Located in the Image target area of ​​individual protective board If the condition is met, the value is 1; otherwise, the value is 0. Projecting pixels to the pose skeleton Normalized distance; This is the width parameter relative to the Gaussian kernel; The consistency score of the projected pixels along the direction from the hinged end to the free end is given. Scoring is given for the consistency between the projected pixels and their historical spatial locations; To determine the confidence level of the fusion detection; , , , , These are the non-negative weight coefficients for each sub-item, satisfying... ; S52: Attitude Constraint-Based Matching Get the Set of measured radar points for individual protective panels ,in To preset the matching score threshold, when multiple sidewall targets overlap, the same radar point is only assigned to the target with the highest matching score.

[0013] As a preferred option, the process of structurally completing the missing radar points in step 6 is as follows: S61: When The number of points in the middle is less than the minimum number of measured points. If the measured points do not cover the hinged end, free end, or key areas of the plate surface, the comprehensive completion operator is invoked. Using the two-dimensional attitude skeleton, the side plate size model, the measured radar point set, and the attitude state at the previous moment as constraints, a structured completion point set is generated for missing edge points, free endpoints, corner points, plate surface mesh points, or equivalent geometric constraints. ,in This indicates the comprehensive completion operator. It is a two-dimensional attitude skeleton. This is a model of the side guard plate dimensions, including the actual length of the side guard plate. ,width and thickness , This represents the attitude state at the previous moment. For the first The historical posture memory queue of the individual protective board; S62: Perform template deformation completion; In template deformation completion, a standard template point set is established based on the actual dimensions of the side panel. Standard template point set This includes sampling points at the hinge end, free end, long side, short side, and plate surface mesh. By solving for the rigid body transformation and small deformation parameters of the template, the template is projected to fit the image's attitude skeleton, and the template's three-dimensional surface fits the measured radar points, as shown in the following formula: ; In the formula, For template rotation matrix, The template translation vector, For small deformations of the template point, Represents the projection function. This represents the error between the template projection contour and the image pose skeleton. This indicates the anchoring error between the template surface and the measured radar point. , , These are the non-negative weighting coefficients for each error term; Set of completion points obtained by template deformation completion As shown in the following formula: The corresponding area is Missing ; In the formula, For template sampling points, This refers to the small-amplitude deformation corresponding to the template point; S63: Complete the chronological memory; In temporal memory completion, the system maintains the most recent... Frame-high reliability pose results and corresponding point clouds are used to form a historical pose memory queue. When the point cloud in the current frame is missing, the point clouds in the reliable historical frames are mapped to the current frame according to the pose transformations of adjacent frames, and then filtered according to the pose skeleton of the current image. The completed point set is obtained by time-series memory completion. Satisfy the following formula: ; In the formula, Indicates the attitude state A defined three-dimensional rigid body transformation matrix. The current pose is predicted from the pose of the previous time step. For the first time in history The pose state of the frame. For the first time in history The set of measured radar points for the frame. To assess the reliability of pose in historical frames, The historical frame credibility threshold. The threshold for skeleton projection distance; S64: Perform graph structure completion; pass Layered graph message passing aggregates the spatial location, skeletal structure, and node type information of domain nodes in each layer to update node features. , final layer features Input a multilayer perceptron and output the 3D position offset of the missing node. and completion confidence ; in, For the first Nodes in a layered graph structure eigenvectors, For nodes The set of neighboring nodes, The attention weights are determined by the image skeleton distance, 3D distance, and node type. For activation function, , Given two learnable weight matrices, For the predicted position offset, To complete the confidence level, the value range is [0,1]; All missing nodes in the graph structure are offset by the predicted position. Corrected 3D coordinates and As the completion point, and corresponding completion confidence level. Relationships form the set of points for completing the graph structure. As shown in the following formula; Missing node ; S65: The comprehensive set of completion points is obtained by merging the three types of completion results. As shown in the following formula: ; In the formula, This represents the operations of deduplication, spatial consistency filtering, and confidence fusion. For any completion point Its completion confidence level As shown in the following formula: ; In the formula, To compensate for the error between the points projected onto the image and the pose skeleton, To compensate for the consistency error between the depth anchor point and the measured radar point, To compensate for the consistency error between the point and historical attitude predictions, For template deformation residuals, , , , These are non-negative weighting coefficients. Represents the amplitude limiting function; S66: Calculate the completion ratio and average completion confidence ,in It is a very small positive number, used to prevent the denominator from being zero.

[0014] As a preferred option, in step 7, the process of establishing a joint attitude optimization objective and solving for the three-dimensional attitude is as follows: S71: with Indicates the attitude state and size model Define the point cloud matching error for the determined 3D surface of the side panel. As shown in the following formula: ; In the formula, Point To three-dimensional surface The shortest Euclidean distance; S72: Define completion consistency error ; S73: A joint pose optimization objective is established based on image pose error, point cloud matching error, completion consistency error, geometric constraint error, and temporal continuity error. The final 3D pose state is obtained by minimizing the joint optimization objective. As shown in the following formula: ; In the formula, The constrained 3D pose projection is consistent with the image key points and pose skeleton. Constrain the dimensions of the side guard plate, hinge axis, and range of mechanical movement. Constrain the pose continuity of adjacent frames , , , , These are the non-negative weighting coefficients for each error term.

[0015] As a preferred embodiment, the process of attitude result reliability assessment and anomaly detection in step 8 is as follows: S81: Order For optimal posture Normalized residuals at the location; define radar coverage score for measured points. Optimize residual scoring Rating in sync with time ,in To optimize the residual normalization scale; This represents the time difference between the image frame and the point cloud frame. This is the maximum permissible time difference; S82: Define the confidence level of attitude results ,in Image pose confidence. To determine the confidence level of BEV fusion detection, To supplement the proportional penalty index; S83: The system determines the reliability of the attitude results. Output the following results: when When the reliability exceeds the preset reliability threshold, output the first... The three-dimensional pose, spatial positioning, and reliability of the pose results of the personal protective equipment board. ;when When the reliability is below the preset reliability threshold, the system outputs a low-reliability attitude or a state pending verification, and marks the corresponding depth range of the guard plate. At the same time, the system judges whether there is an abnormal attitude or cutting interference risk based on the guard plate's unfolding angle, sway angle, free end spatial position, and spatial occupancy. When the free end of the guard plate intrudes into the preset safe area, the system outputs a cutting interference warning and the corresponding risk level.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This invention employs a dual-branch architecture that parallelly processes an infrared image branch and a 4D millimeter-wave radar point cloud branch. The infrared image branch extracts the semantics, attitude key points, edge lines, and image attitude confidence of the sidewall panel through a YOLO network, generating image BEV features. The radar branch encodes the point cloud into radar BEV pseudo-image features through a PointPillars network, providing three-dimensional scale and spatial echo distribution. The two types of BEV features are fused in a unified space through attention-weighted fusion, and the fusion detection head outputs the sidewall panel target center, size, orientation, two-dimensional attitude skeleton, and fusion confidence. This architecture enables the complementary fusion of the two-dimensional semantic prior provided by the image and the three-dimensional geometric information provided by the radar in the BEV space, overcoming the limitations of a single sensor's insufficient perception capability under conditions of downhole obstruction, dust, and low illumination, and improving the stability of sidewall panel target detection and attitude structure extraction.

[0017] 2. This invention uses attitude skeleton constraints for radar point cloud matching, defining an attitude constraint matching score based on five factors: fused projection position, skeleton distance, orientation consistency, historical consistency, and fusion confidence. This score is then used to select the set of measured radar points corresponding to each sidewall panel. This matching mechanism effectively reduces the interference of background structures such as roof beams, pillars, and coal walls on the sidewall panel point cloud. Simultaneously, the unique attribution constraint resolves the ambiguity in point cloud attribution in overlapping areas of multiple sidewall panel targets, providing high-quality measured point cloud input for subsequent attitude estimation.

[0018] 3. This invention constructs a structured completion mechanism comprising three types of sub-operators: template deformation completion, temporal memory completion, and graph structure completion. When measured radar points fail to cover key areas of the side guard plate due to obstruction, missing echoes, or insufficient reflectivity, the comprehensive completion operator, constrained by the two-dimensional attitude skeleton, the side guard plate size model, and historical attitude states, generates structured completion points such as missing edge points, free endpoints, corner points, and plate surface mesh points, and assigns a quantized confidence level to each completion point. This mechanism enables the side guard plate attitude estimation to maintain the continuity and interpretability of attitude output even under conditions of sparse or locally missing 4D millimeter-wave radar point clouds.

[0019] 4. This invention introduces a completion confidence-driven differentiated constraint strategy in joint attitude optimization. In the point cloud matching error, completed points participate in the matching constraints with completion confidence as a weight. High-confidence completed points contribute to the attitude constraints close to the measured points, while low-confidence completed points automatically reduce their influence. Simultaneously, a completion consistency error is defined, using complementary confidence as a weight to impose additional constraints on low-confidence completed points, preventing them from dominating the final attitude. This strategy achieves an organic balance between fully utilizing completed information and risk control, improving the robustness of attitude estimation under point cloud missing conditions.

[0020] 5. This invention constructs a multi-dimensional attitude result reliability evaluation system that integrates seven factors: image attitude confidence, BEV fusion confidence, radar measured point coverage score, optimized residual score, time synchronization score, completion ratio penalty, and average completion confidence. This evaluation system comprehensively assesses the reliability of attitude estimation results from four dimensions: data source quality, optimized solution effect, completion dependency, and time alignment accuracy. Based on the reliability, the system outputs a valid attitude or a state to be verified, and judges abnormal attitudes and cutting interference risks based on the three-dimensional attitude and spatial occupancy of the side guard plate. This provides a complete and reliable technical solution for side guard plate status perception and cutting interference prevention in coal mining faces.

[0021] This method achieves high-precision and robust estimation of the attitude of the sidewall under underground conditions of obstruction, dust, and sparse point cloud, providing a reliable sensing basis for the state perception of the sidewall in coal mining faces and the prevention of cutting interference.

[0022] This invention provides a fender attitude estimation system based on 4D millimeter-wave radar and infrared images, used to implement a fender attitude estimation method based on 4D millimeter-wave radar and infrared images, including: The infrared image acquisition module is used to continuously acquire infrared image frames of the target area of ​​the protective panel and transmit the image frames to the dual-branch joint detection module and the time synchronization and frame matching module. The 4D millimeter-wave radar point cloud acquisition module is used to continuously acquire radar point cloud frames containing three-dimensional spatial coordinates, radial velocity, and reflection intensity, and transmit the point cloud frames to the dual-branch joint detection module and the time synchronization and frame matching module. The time synchronization and frame matching module is used to select matching frame pairs from the infrared image frame buffer and the radar point cloud frame buffer according to the timestamp or the system reception time. The sensor calibration module is used to provide spatial transformation parameters for image BEV feature projection and radar point cloud projection through offline joint calibration or on-site calibration. The dual-branch joint detection module is used to construct a dual-branch joint detection network that includes an image branch and a radar branch. It processes the time-synchronized infrared image frames and radar point cloud frames in parallel, fuses the two types of features in the BEV space, and outputs the target detection results of the guardrail, a two-dimensional attitude skeleton, and the fusion confidence score. The point cloud projection module is used to project the radar point cloud onto the infrared image plane based on the radar-infrared extrinsic matrix and infrared camera intrinsic parameters provided by the sensor calibration module, through homogeneous coordinate transformation and pinhole imaging model, and to establish the spatial mapping relationship between radar points and image pixels. The attitude constraint point cloud matching module is used to filter and determine the affiliation of the projected radar points by using the two-dimensional attitude skeleton, the image target region and the fusion detection confidence as prior constraints. The structured point cloud completion module is used to generate a structured completion point set containing missing edge points, free endpoints, corner points, and plate surface grid points when the measured radar point set does not meet the preset integrity conditions, with the two-dimensional attitude skeleton, the guard plate size model, the measured radar point set, and the attitude state at the previous moment as constraints. The joint attitude estimation module is used to establish a joint attitude optimization objective using a two-dimensional attitude skeleton, a set of measured radar points, a set of complete points, and a side plate size model. The three-dimensional attitude state of the side plate is solved by minimizing this objective. The credibility evaluation and anomaly detection module is used to calculate the credibility of the attitude results. It detects whether there is an abnormal posture or risk of cutting interference; when the free end of the guard plate intrudes into the preset safe area, it outputs a cutting interference warning and the corresponding risk level.

[0023] The infrared image acquisition module provides stable and continuous infrared visual input for subsequent YOLO image branches, effectively capturing the thermal radiation characteristics and structural contours of the sidewall in low-light and dusty environments downhole. The 4D millimeter-wave radar point cloud acquisition module provides realistic three-dimensional spatial echo information for subsequent PointPillars radar branches, compensating for the lack of absolute depth information in single infrared images. The time synchronization and frame matching module effectively solves the spatiotemporal alignment problem between multi-source heterogeneous sensors, ensuring input consistency for cross-modal fusion from the data source. The sensor calibration module provides precise spatial transformation parameters for image BEV feature projection and radar point cloud projection, ensuring that subsequent cross-modal data fusion and point cloud projection matching are performed in a unified spatial coordinate system. The dual-branch joint detection module enables dynamic complementary enhancement of the two-dimensional semantic prior provided by the image and the three-dimensional geometric information provided by the radar within a unified BEV space, overcoming the limitations of a single sensor's insufficient sensing capability under downhole obstruction, dust, and low-light conditions. Based on the point cloud projection module, a precise geometric mapping relationship between 3D spatial points and 2D image pixels was established, providing a spatially effective cross-modal data association foundation for subsequent attitude constraint matching. Based on the attitude constraint point cloud matching module, interference from background structures such as roof beams, pillars, and coal walls was effectively filtered out, resolving the ambiguity in point cloud attribution in overlapping areas of multiple sidewall targets, and providing high-quality, low-noise measured point cloud input for attitude estimation. Based on the structured point cloud completion module, the attitude estimation of the sidewall can maintain the continuity and interpretability of attitude output even under conditions of sparse or locally missing 4D millimeter-wave radar point clouds. Based on the joint attitude estimation module, differentiated utilization and risk control of measured radar points and structured completion points were achieved, enabling the attitude estimation results to fully utilize multi-source constraint information while effectively suppressing the uncertainty caused by point cloud missingness, significantly improving the accuracy and robustness of sidewall attitude estimation under complex downhole conditions. Based on the credibility evaluation and anomaly judgment module, a comprehensive quantitative evaluation of the reliability of attitude estimation results is realized. Based on credibility, differentiated results and cutting interference early warning are output, providing a complete decision-making link from attitude estimation to reliability assessment and cutting interference prevention for the status perception of the coal face side protection plate.

[0024] This system deeply integrates infrared visual semantics with radar spatial geometry in BEV space, enabling stable estimation and anomaly warning of the sidewall attitude under complex underground working conditions, and providing a complete system-level solution for intelligent perception of fully mechanized mining faces. Attached Figure Description

[0025] Figure 1 This is a flowchart of the attitude estimation method in this invention; Figure 2 This is a flowchart of the BEV spatial feature fusion and joint detection result output in this invention. Detailed Implementation

[0026] To further illustrate the technical means and effects of the present invention in achieving its intended purpose, the following detailed description of the specific implementation methods, structures, features, and effects of the present invention, in conjunction with the accompanying drawings and preferred embodiments, is provided below.

[0027] Example 1:

[0028] like Figure 1 and Figure 2 As shown, this embodiment provides a method for estimating the attitude of a protective panel based on 4D millimeter-wave radar and infrared images, including the following steps: Step 1: Simultaneously acquire multi-source data and perform time matching; Simultaneously acquire infrared image frames and 4D millimeter-wave radar point cloud frames in the target operation area, and perform data matching according to timestamp or reception time to provide spatiotemporally aligned input data for subsequent dual-branch network processing and attitude estimation; As a preferred approach, the process of simultaneously acquiring multi-source data and performing time matching is as follows: S11: An infrared camera is installed at the rocker arm end of the coal mining machine or on the top of the hydraulic support. At the same time, a 4D millimeter-wave radar is installed on the stable base adjacent to the infrared camera. The infrared camera is used to acquire continuous infrared images of the target area of ​​the side guard plate, and the 4D millimeter-wave radar is used to collect and output radar point clouds containing three-dimensional spatial coordinates, radial velocity and reflection intensity. S12: Infrared image frames and radar point cloud frames within the target operation area are acquired using an infrared camera and a 4D millimeter-wave radar, respectively. S13: Select matching infrared image frames and radar point cloud frames from the buffer based on the timestamp or system reception time; when the infrared camera and radar do not have a unified hardware clock, use the system reception time of data arriving at the same data processing device as the common time axis, and select the frame with the smallest absolute time difference as the matching frame; when the minimum time difference is greater than the preset maximum allowable time difference, terminate the current attitude estimation and output the data desynchronization status. In this technical solution, an infrared camera and a 4D millimeter-wave radar are respectively deployed at the rocker arm end of the coal mining machine or the top of the hydraulic support and its adjacent stable base, establishing a dual-modal perception hardware foundation that complements visual semantics and three-dimensional spatial information. Time synchronization matching is performed through timestamps or system reception time, and a maximum allowable time difference threshold is set. When the infrared camera and radar do not have a unified hardware clock, the system reception time is used as a common time axis to select the optimal matching frame. When the time difference exceeds the limit, attitude estimation is automatically terminated and the data desynchronization status is output. This effectively solves the spatiotemporal alignment problem between multi-source heterogeneous sensors, ensuring the input consistency and reliability of subsequent dual-branch feature extraction and cross-modal fusion from the data source.

[0029] Step 2: Construct a dual-branch joint detection network and perform multimodal feature extraction; Construct a dual-branch joint detection network containing image branch and radar branch, and perform feature extraction and position modeling on infrared image frames and radar point cloud frames respectively; like Figure 1 As shown, this invention employs parallel processing of an infrared image branch and a 4D millimeter-wave radar point cloud branch. The infrared image branch extracts image features through a YOLO network and outputs the guardrail detection box, pose key points, edge lines, and image pose confidence. The 4D millimeter-wave radar point cloud branch encodes the point cloud into radar BEV pseudo-image features through a PointPillars network. Specifically, the process of constructing the dual-branch joint detection network and performing multimodal feature extraction is as follows: S21: Construct a dual-branch joint detection network; the image branch uses a YOLO network, which includes a backbone feature extraction layer, a feature pyramid fusion layer, a target detection head, and pose key points. Multi-level features of the infrared image are extracted through the backbone network and the feature pyramid fusion layer. Then through the target detection head Obtain the detection bounding box, class label, and detection confidence score of the side panel, i.e., the image detection result. The attitude key point set of the guardrail is extracted using attitude key point extraction. ;in, For the current infrared image, For YOLO Layer image features, This represents the YOLO backbone and feature fusion operator, and the image detection results. Includes target bounding box, category, and detection confidence, as well as a set of image pose keypoints. This includes at least two types of attitude structures among the key points of the hinged end of the side guard plate, key points of the free end, edge points of the plate surface, and centerline parameters of the plate surface. For the target detection head mapping function, This is the attitude key point mapping function; The radar branch employs a PointPillars network, dividing the point cloud into cylinders using a BEV grid. Point feature vectors are constructed for points within each cylinder, comprising at least three of the following: 3D coordinates, reflection intensity, radial velocity, offset relative to the cylinder center, and offset relative to the centroid of the point set. Cylinder features are extracted via a shared multilayer perceptron and a max-pooling layer. To achieve multimodal feature extraction of radar BEV pseudo-images; S22: Multimodal feature alignment in BEV space; To achieve unified fusion of image and radar features in BEV space, image BEV features are obtained by projecting or sampling YOLO intermediate layer image features onto the BEV mesh using camera intrinsic parameters, radar and infrared camera extrinsic parameters, and preset ground or sidewall working surface height ranges. Specifically: Image features Extrinsic parameter matrix from radar coordinate system to infrared camera coordinate system and camera projection function Converted into image BEV features within the radar BEV space. ,in The center point of the BEV element in the coordinate system. The coordinates of the column center are For the first Image BEV features of each BEV unit, Sample represents the fusion weights for image features at different scales, and it denotes the bilinear sampling operation. For each point pair in the radar point cloud Construct feature vectors ,in, Let the coordinates of the point be in three-dimensional space. Radial velocity, This refers to either reflection intensity or signal-to-noise ratio. The coordinates of the centroid of the point set within the cylinder; Radar BEV pseudo-image features are obtained through MLP and max pooling. ,in For the first Radar BEV pseudo-image features of a BEV unit; In this technical solution, a dual-branch joint detection network comprising a YOLO image branch and a PointPillars radar branch is constructed to achieve parallel extraction and BEV spatial alignment of infrared image semantic information and 4D millimeter-wave radar spatial geometric information. The image branch, through multi-level feature extraction of the YOLO network, target detection head, and attitude key points, simultaneously outputs the detection bounding box, category label, key points of the hinged and free ends, and multi-dimensional attitude structure of the guard plate, as well as the edge lines of the plate surface. The radar branch, through the PointPillars network, divides the point cloud into cylinders according to the BEV grid, constructs a comprehensive point feature vector containing three-dimensional coordinates, radial velocity, reflection intensity, and relative offset for the points within the cylinders, and extracts cylinder features through shared MLP and max pooling to generate radar BEV pseudo-image features. By using camera intrinsic parameters, radar and infrared camera extrinsic parameters, and preset height range, image features are projected or sampled onto the BEV grid, achieving precise alignment of the two types of features within a unified BEV space. This provides a spatially consistent feature basis for subsequent cross-modal fusion and joint detection, effectively overcoming the limitations of insufficient perception capability of a single sensor under conditions of underground obstruction, dust, and low illumination.

[0030] Step 3: Perform BEV spatial feature fusion and output joint detection results; fuse image BEV features and radar BEV pseudo-image features in BEV space, and output the side guard plate target detection results, two-dimensional attitude skeleton and fusion confidence based on the fused features; As a preferred option, the process of fusing BEV spatial features and outputting joint detection results is as follows: S31: Perform BEV spatial feature fusion. The BEV spatial fusion can employ any of the following: convolutional fusion, attention-weighted fusion, or gated fusion. When attention-weighted fusion is used, the fusion weights of the image BEV features and the radar BEV pseudo-image features are jointly predicted by their local features. Specifically, the image BEV features... and radar BEV pseudo-image features Attention-weighted fusion is performed to obtain fused BEV features. ,in and These are the fusion weights for the image branch and the radar branch, respectively. Both are dimensionless scalars, predicted jointly by convolution and softmax operations from the local features of both branches, satisfying the following conditions: ; This indicates a channel-wise weighted operation; S32: Based on integrated BEV features The detection head outputs the target detection results of the guardrail, the two-dimensional attitude skeleton, and the fusion confidence score, and the first... Complete test output of individual protective board ,in For the first The center coordinates of the protective side panel on the BEV plane, in meters; , , These are the length, width, and height dimensions of the side panel, in meters (m). The angle of the side guard plate on the BEV plane is expressed in rad. The two-dimensional posture skeleton of the side guard plate is composed of the key points connecting the hinged end, the free end, and the edge of the plate. The confidence level for fusion detection is set to [0,1]. This is a mapping function for the fusion detection head; In this technical solution, attention-weighted fusion of image BEV features and radar BEV pseudo-image features is performed within a unified BEV space. The fusion weights are adaptively predicted from the local features of both through convolution and Softmax operations, achieving dynamic complementary enhancement of visual semantic information and radar spatial geometric information in the feature dimension. The fused BEV features are output synchronously by the detection head, including the target center coordinates, length, width, and height dimensions of the guard plate, the BEV orientation angle, a two-dimensional attitude skeleton composed of key points at the hinged and free ends and the edge lines of the plate, and the fusion detection confidence score. This provides highly reliable, multi-dimensional, and directly quantifiable prior information for subsequent attitude skeleton constraint matching, structured completion, and joint attitude optimization.

[0031] Step 4: Project the radar point cloud onto the infrared image plane; Based on the pre-calibrated extrinsic parameters from the radar coordinate system to the infrared camera coordinate system and the intrinsic parameters of the infrared camera, project the radar point cloud onto the infrared image plane to establish the spatial correspondence between radar points and image pixels. As a preferred method, the process of projecting radar point clouds onto the infrared image plane is as follows: For the There are 1 radar point, whose homogeneous coordinates in the radar coordinate system are: According to the extrinsic parameter matrix Calculate its homogeneous coordinates in the infrared camera coordinate system. The projected pixel coordinates are calculated based on the infrared camera's intrinsic parameters and the pinhole imaging model, as shown in the following formula: ; ; In the formula, , These are the x and y coordinates of the projected pixels, respectively. , This refers to the focal length of the infrared camera in the horizontal and vertical directions of the image, expressed in pixels. , These are the x and y coordinates of the principal point of the infrared camera on the image plane, in pixels. The depth of the radar point in the camera coordinate system, in meters; Only when And projected pixel coordinates If the radar point is within the effective image range, it will be retained and proceed to step 5 for attitude constraint matching calculation and determination of the actual radar point's affiliation; otherwise, the radar point will not participate in subsequent processing. In this technical solution, based on the pre-calibrated radar-infrared camera extrinsic parameter matrix and infrared camera intrinsic parameters, a 4D millimeter-wave radar point cloud is precisely projected onto the infrared image plane through homogeneous coordinate transformation and a pinhole imaging model, establishing a precise geometric mapping relationship between three-dimensional spatial points and two-dimensional image pixels. This is achieved by setting depth constraints. The effective field-of-view constraints (projected coordinates within the image range) are used to screen radar points for physical rationality, effectively eliminating invalid projections behind the sensor and outside the field of view. This ensures the spatial validity and data reliability of radar points entering the subsequent attitude constraint matching process, providing an accurate spatial correspondence basis for cross-modal data association.

[0032] Step 5: Match radar point cloud based on attitude skeleton constraint; using the attitude skeleton of the fused detection results and the target region of the image, perform attitude constraint matching on the radar points to obtain the set of measured radar points corresponding to each guard plate. As a preferred option, the process of matching radar point clouds based on attitude skeleton constraints is as follows: S51: Regarding the first Individual care board and the first The attitude constraint matching score is determined based on several radar points, including whether the projection point is located in the target area of ​​the guardrail image, the distance from the projection point to the attitude skeleton, the consistency of the projection point along the direction from the hinge end to the free end, the consistency of the projection point with the historical spatial position, and the fusion detection confidence. As shown in the following formula: ; In the formula, The indicative function is the pixel coordinates of the radar point projection. Located in the Image target area of ​​individual protective board If the condition is met, the value is 1; otherwise, the value is 0. Projecting pixels to the pose skeleton The normalized distance is dimensionless; The width parameter of the distance from the Gaussian kernel is dimensionless; The consistency score of the projected pixels along the direction from the hinge end to the free end is given, with a value range of [0,1]. The consistency score between the projected pixel and its historical spatial location is given, with a value range of [0,1]. To determine the confidence level of the fusion detection; , , , , These are the non-negative weight coefficients for each sub-item, satisfying... ; S52: Attitude Constraint-Based Matching Get the Set of measured radar points for individual protective panels As shown in the following formula: ; In the formula, To preset the matching score threshold, when multiple side panel targets overlap, the same radar point will only be assigned to the target with the highest matching score; In this technical solution, a multi-dimensional attitude constraint matching sub-model is constructed, incorporating five factors: fusion projection area, skeleton distance, directional consistency, historical consistency, and fusion detection confidence. The two-dimensional attitude skeleton of the side panel is used as a priori constraints to filter and assign radar point clouds. In the matching sub-model, the indicative function constraint ensures that the radar point projection lies within the target area of ​​the side panel image. The Gaussian kernel function for skeleton distance prioritizes radar points closer to the attitude skeleton. The directional consistency score filters out outliers with inconsistent directions from the hinge end to the free end, and the historical consistency score enhances the coherence of inter-frame matching. Simultaneously, the unique assignment constraint (the same radar point only belongs to the side panel target with the highest matching score) effectively solves the point cloud assignment ambiguity problem in overlapping areas of multiple side panel targets, significantly reducing the interference of background structures such as roof beams, columns, and coal walls on the side panel point cloud, providing high-quality, low-noise measured point cloud input for subsequent attitude estimation.

[0033] Step 6: Structured completion of missing radar points; When the set of measured radar points does not meet the preset integrity conditions, a set of structured completion points is generated using the two-dimensional attitude skeleton, the guard plate size model, the set of measured radar points, and the historical attitude state. Since 4D millimeter-wave radar typically only obtains a small number of discrete echoes in downhole siding scenarios, especially at the free ends, edges, or inclined surfaces of the siding, this invention does not employ simple interpolation for point completion. Instead, it uses structured completion with attitude prior constraints. This structured completion consists of three parts: template deformation completion, temporal memory completion, and graph structure completion. A completion confidence level is assigned to each completion point. Specifically, the process of structured completion of missing radar points is as follows: S61: When The number of points in the middle is less than the minimum number of measured points. If the measured points do not cover the hinged end, free end, or key areas of the plate surface, the comprehensive completion operator is invoked. Using the two-dimensional attitude skeleton, the side plate size model, the measured radar point set, and the attitude state at the previous moment as constraints, a structured completion point set is generated for missing edge points, free endpoints, corner points, plate surface mesh points, or equivalent geometric constraints. As shown in the following formula: ; In the formula, This indicates the comprehensive completion operator. It is a two-dimensional attitude skeleton. This is a model of the side guard plate dimensions, including the actual length of the side guard plate. ,width and thickness , This represents the attitude state at the previous moment. For the first The historical posture memory queue of the individual protective board; S62: Perform template deformation completion; The template deformation completion sub-operator uses a standard-sized model of the side guard plate as a template. It projects and constrains the hinged and free ends of the template onto a two-dimensional attitude skeleton, and uses measured radar points as depth anchor points to solve for the template's scale, rotation, and translation corrections in three-dimensional space, thereby generating completion points. Specifically, in template deformation completion, based on the actual dimensions of the side guard plate (such as the actual length of the side guard plate)... ,width and thickness Establish a standard template point set Standard template point set This includes sampling points at the hinge end, free end, long side, short side, and plate surface mesh. By solving for the rigid body transformation and small deformation parameters of the template, the template is projected to fit the image's attitude skeleton, and the template's three-dimensional surface fits the measured radar points, as shown in the following formula: ; In the formula, For template rotation matrix, The template translation vector, For small deformations of the template point, Represents the projection function. This represents the error between the template projection contour and the image pose skeleton. This indicates the anchoring error between the template surface and the measured radar point. , , These are the non-negative weighting coefficients for each error term; Set of completion points obtained by template deformation completion Satisfy the following formula: The corresponding area is Missing ; In the formula, For template sampling points, This refers to the small-amplitude deformation corresponding to the template point; Based on this process, even with few radar points, the free end, edge, and candidate completion points on the plate surface can be recovered using the size of the guard plate and image attitude lines. S63: Complete the chronological memory; The temporal memory completion sub-operator establishes an attitude memory queue based on the attitude states of the guardrail across multiple consecutive frames, and selects completion candidate points or completion candidate edges from reliable historical frames based on the consistency between the current frame's prior attitude and the historical 3D attitude. Specifically, in temporal memory completion, the system maintains the most recent Frame-high reliability pose results and corresponding point clouds are used to form a historical pose memory queue. When the point cloud in the current frame is missing, the point clouds in the reliable historical frames are mapped to the current frame according to the pose transformations of adjacent frames, and then filtered according to the pose skeleton of the current image. The completed point set is obtained by time-series memory completion. Satisfy the following formula: ; In the formula, Indicates the attitude state A defined three-dimensional rigid body transformation matrix. The current pose is predicted from the pose of the previous time step. For the first time in history The pose state of the frame. For the first time in history The set of measured radar points for the frame. To assess the reliability of pose in historical frames, The historical frame credibility threshold. The threshold for skeleton projection distance; Based on this method, the continuity of the guardrail's attitude change can be used to compensate for short-term occlusion and instantaneous echo loss; S64: Perform graph structure completion; The graph structure completion sub-operator uses hinged ends, free ends, edge points, measured radar points, and template sampling points as graph nodes. It establishes graph edges based on image skeleton adjacency relationships, 3D spatial adjacency relationships, and template topological relationships, and updates the 3D position and completion confidence of missing nodes through graph attention or message passing. Specifically, through... Layered graph message passing aggregates the spatial location, skeletal structure, and node type information of domain nodes in each layer to update node features. , final layer features Input a multilayer perceptron and output the 3D position offset of the missing node. and completion confidence ; in, For the first Nodes in a layered graph structure eigenvectors, For nodes The set of neighboring nodes, The attention weights are determined by the image skeleton distance, 3D distance, and node type. For activation function, , Given two learnable weight matrices, For the predicted position offset, To complete the confidence level, the value range is [0,1]; All missing nodes in the graph structure are offset by the predicted position. Corrected 3D coordinates and As the completion point, and corresponding completion confidence level. Relationships form the set of points for completing the graph structure. As shown in the following formula; Missing node ; Graph structure completion can utilize the edge continuity of the side panel and the topological relationship of the panel surface to reduce the drift of completion points at free ends and edges; S65: The comprehensive set of completion points is obtained by merging the three types of completion results. As shown in the following formula: ; In the formula, This represents the operations of deduplication, spatial consistency filtering, and confidence fusion. For any completion point Its completion confidence level satisfy: ; In the formula, To compensate for the error between the points projected onto the image and the pose skeleton, To compensate for the consistency error between the depth anchor point and the measured radar point, To compensate for the consistency error between the point and historical attitude predictions, For template deformation residuals, , , , These are non-negative weighting coefficients. Represents the amplitude limiting function; S66: Calculate the completion ratio and average completion confidence ,in It is a very small positive number, used to prevent the denominator from being zero; The larger the value, the more the current result relies on completion. The larger the value, the higher the overall credibility of the completion point; In this technical solution, to address the issue of echo loss at the free end, edge, and inclined surface of the downhole sidewall plate of 4D millimeter-wave radar, a structured completion mechanism with attitude prior constraints is adopted. This mechanism uses three sub-operators—template deformation completion, temporal memory completion, and graph structure completion—to collaboratively generate structured completion points for the missing regions. Template deformation completion uses a standard-sized model of the sidewall plate as a template, projects and constrains the template onto a two-dimensional attitude skeleton, and uses measured radar points as depth anchors to solve for optimal rigid body transformation and small-amplitude deformation. Temporal memory completion uses point clouds from historical high-confidence frames to map to the current frame according to attitude transformation, compensating for short-term occlusion and instantaneous echo loss. Graph structure completion aggregates the spatial location, skeleton structure, and node type information of neighboring nodes through graph message passing, predicting the three-dimensional position offset and completion confidence of missing nodes. After deduplication, spatial consistency filtering, and confidence fusion of the three types of completion results, a multi-dimensional completion confidence score is assigned to each completion point, which is jointly determined by image skeleton consistency, depth anchor point consistency, historical pose consistency, and template deformation residual. This enables subsequent joint pose optimization to utilize completion information differently according to the completion reliability, and to maintain the continuity and interpretability of pose output even under the condition of sparse or locally missing measured point clouds.

[0034] Step 7: Establish a joint attitude optimization objective and solve for the three-dimensional attitude; use the two-dimensional attitude skeleton, the set of measured radar points, the set of complete points, and the side guard plate size model to establish a joint attitude optimization objective and solve for the three-dimensional attitude state of the side guard plate. As a preferred option, the process of establishing a joint attitude optimization objective and solving for the 3D attitude is as follows: S71: with Indicates the attitude state and size model Define the point cloud matching error for the determined 3D surface of the side panel. As shown in the following formula: ; In the formula, Point To three-dimensional surface The shortest Euclidean distance, in meters; the error weights for the completed points in the point cloud matching error calculation are determined by the completion confidence level. The error weight is determined and is not higher than that of the measured radar points. High-confidence completion points participate in the matching constraints with higher weights, while low-confidence completion points automatically reduce their influence. S72: Define completion consistency error ; Complete consistency error Used to constrain low-confidence completion points from dominating the final pose; S73: A joint pose optimization objective is established based on image pose error, point cloud matching error, completion consistency error, geometric constraint error, and temporal continuity error. The final 3D pose state is obtained by minimizing the joint optimization objective. As shown in the following formula: ; In the formula, The constrained 3D pose projection is consistent with the image key points and pose skeleton. Constrain the dimensions of the side guard plate, hinge axis, and range of mechanical movement. Constrain the pose continuity of adjacent frames , , , , These are the non-negative weighting coefficients for each error term; In this technical solution, a differentiated constraint strategy driven by completion confidence is constructed to achieve reasonable utilization and risk control of measured radar points and structured completion points. In point cloud matching error, completion points participate in matching constraints with completion confidence as weight. The contribution of high-confidence completion points to attitude constraints is close to that of measured points, while the influence of low-confidence completion points is automatically reduced. At the same time, completion consistency error is introduced, which applies additional constraints to low-confidence completion points with complementary confidence as weight to prevent them from dominating the final attitude. Based on this, image attitude error, point cloud matching error, completion consistency error, geometric constraint error, and temporal continuity error are uniformly incorporated into a joint optimization objective. By minimizing this objective, the three-dimensional spatial position, unfolding angle, and yaw angle of the sidewall are solved simultaneously. This allows the attitude estimation results to fully utilize multi-source constraint information while effectively suppressing the uncertainty caused by sparse point clouds and local missing information, significantly improving the accuracy and robustness of sidewall attitude estimation by 4D millimeter-wave radar under downhole obstruction and echo loss conditions.

[0035] Step 8: Attitude result credibility assessment and anomaly judgment; Calculate the attitude result credibility based on multi-dimensional quality indicators, and output the three-dimensional attitude, spatial occupancy and abnormal attitude judgment results of the guardrail. As a preferred option, the process for attitude result reliability assessment and anomaly detection is as follows: S81: Order For optimal posture Normalized residuals at the location; define radar coverage score for measured points. Optimize residual scoring Rating in sync with time ,in To optimize the residual normalization scale, dimensionless; The time difference between the image frame and the point cloud frame is expressed in seconds. The maximum permissible time difference, in seconds; S82: Define the confidence level of attitude results ,in Image pose confidence. To determine the confidence level of BEV fusion detection, To complete the proportional penalty index, it is dimensionless; the reliability of the attitude results. The value increases with the increase of image pose confidence, BEV fusion confidence, radar measured point coverage, average confidence of completed points, joint optimization residual score and time synchronization score, and decreases with the increase of completion ratio. The higher the value, the more reliable the pose estimation result. S83: The system determines the reliability of the attitude results. Output the following results: when When the reliability exceeds the preset reliability threshold, output the first... The three-dimensional pose of the side guard plate (including the three-dimensional coordinates of the hinged end, the three-dimensional coordinates of the free end, the plate surface normal vector, the unfolding angle, and the yaw angle), its spatial occupancy (the oriented cuboid bounding box determined by the side guard plate size model and its three-dimensional pose), and the reliability of the pose results. ;when When the reliability is below the preset reliability threshold, the system outputs a low-reliability attitude or a state pending verification, and marks the corresponding depth range of the guard plate. At the same time, the system judges whether there is an abnormal attitude or cutting interference risk based on the guard plate's unfolding angle, sway angle, free end spatial position, and spatial occupancy. When the free end of the guard plate intrudes into the preset safe area, the system outputs a cutting interference warning and the corresponding risk level.

[0036] This technical solution utilizes a multi-dimensional attitude result reliability evaluation system that integrates seven factors: image attitude confidence, BEV fusion confidence, radar measured point coverage, optimized residual score, time synchronization score, completion ratio penalty, and average completion confidence. This system achieves a comprehensive quantitative evaluation of the reliability of the side guard plate attitude estimation results. When the reliability exceeds a preset threshold, the system outputs three-dimensional attitude parameters, including the three-dimensional coordinates of the hinged and free ends, the plate surface normal vector, the unfolding angle, and the yaw angle, as well as the spatial occupancy determined by the size model. When the reliability is insufficient, the system outputs a state pending verification and marks the corresponding depth interval. Simultaneously, based on the side guard plate unfolding angle, yaw angle, and the spatial position of the free end, the system assesses abnormal attitudes and cutting interference risks. When the free end intrudes into a preset safety zone, the system outputs an early warning message, providing a complete decision-making chain from attitude estimation to reliability assessment and safety early warning for side guard plate status perception and cutting interference prevention in coal mining faces.

[0037] Example 2:

[0038] This embodiment provides a side panel attitude estimation system based on 4D millimeter-wave radar and infrared images, used to implement a side panel attitude estimation method based on 4D millimeter-wave radar and infrared images, including: The infrared image acquisition module is used to continuously acquire infrared image frames of the target area of ​​the side protection plate through an infrared camera installed at the rocker arm end of the coal mining machine or the top of the hydraulic support, and transmit the image frames to the dual-branch joint detection module and the time synchronization and frame matching module. The 4D millimeter-wave radar point cloud acquisition module is used to continuously acquire radar point cloud frames containing three-dimensional spatial coordinates, radial velocity and reflection intensity through a 4D millimeter-wave radar installed on a stable base adjacent to the infrared camera, and transmit the point cloud frames to the dual-branch joint detection module and the time synchronization and frame matching module. The time synchronization and frame matching module is used to select matching frame pairs from the infrared image frame buffer and the radar point cloud frame buffer based on the timestamp or system reception time. When the infrared camera and radar do not have a unified hardware clock, the system reception time of the data arriving at the same data processing device is used as a common time axis, and the frame with the smallest absolute time difference is selected as the matching frame. When the minimum time difference is greater than the preset maximum allowable time difference, the attitude estimation is terminated and the data desynchronization status is output. The sensor calibration module is used to obtain the intrinsic parameters (focal length, etc.) of the infrared camera through offline joint calibration or on-site calibration. , ) and principal point coordinates ( , ), the extrinsic parameter matrix from the radar coordinate system to the infrared camera coordinate system This provides spatial transformation parameters for image BEV feature projection and radar point cloud projection; The dual-branch joint detection module is used to construct a dual-branch joint detection network that includes an image branch and a radar branch. It processes the time-synchronized infrared image frames and radar point cloud frames in parallel, fuses the two types of features in the BEV space, and outputs the target detection results of the guardrail, a two-dimensional attitude skeleton, and the fusion confidence score. The dual-branch joint detection module includes a YOLO image branch, a PointPillars radar branch, and a BEV fusion detection unit. The YOLO image branch includes a backbone feature extraction layer, a feature pyramid fusion layer, a target detection head, and a pose key point head, used to extract multi-level image features from infrared images. Output the detection bounding box, category label, detection confidence of the guardrail, and a set of image pose key points containing at least two types of pose structures among the hinge end key points, free end key points, plate edge points and plate centerline parameters. The PointPillars radar branch is used to divide the radar point cloud into pillars according to the BEV grid. For points within each pillar, a point feature vector is constructed containing at least three of the following: three-dimensional coordinates, reflection intensity, radial velocity, offset relative to the pillar center, and offset relative to the centroid of the point set. Pillar features are extracted using a shared multilayer perceptron and a max-pooling layer to generate radar BEV pseudo-image features. ; The BEV fusion detection unit is used to convert the image features output from the image branch into image BEV features through the external parameter matrix and the camera projection function. and with radar BEV pseudo-image features Attention-weighted fusion is performed to obtain fused BEV features. Then, the fusion detection head outputs the target center coordinates, length, width, and height dimensions of the guardrail, the BEV orientation angle, and the two-dimensional attitude skeleton. Confidence of fusion detection ; The point cloud projection module, based on the radar-infrared extrinsic parameter matrix and infrared camera intrinsic parameters provided by the sensor calibration module, projects the radar point cloud onto the infrared image plane through homogeneous coordinate transformation and a pinhole imaging model, establishing a spatial mapping relationship between radar points and image pixels; this module only retains the depth. Furthermore, the radar points whose projected pixel coordinates are located within the effective image range are transmitted to the attitude constraint point cloud matching module. The attitude constraint point cloud matching module uses a two-dimensional attitude skeleton, image target region, and fused detection confidence as prior constraints to filter and assign projected radar points. This module calculates the attitude constraint matching score of each radar point relative to each side panel target. The matching score is determined by weighting five factors: whether the projection point is located in the target region of the image, the normalized distance from the projection point to the attitude skeleton, the consistency of the projection point along the direction from the hinge end to the free end, the consistency of the projection point with the historical spatial position, and the confidence of the fusion detection. Based on the matching score, the set of measured radar points corresponding to each guard plate is selected and output. The same radar point is assigned to the guardrail target with the highest matching score only through the unique attribution constraint; The structured point cloud completion module is used to generate a structured completion point set containing missing edge points, free endpoints, corner points, and plate surface mesh points when the measured radar point set does not meet the preset integrity conditions. This is done by using the two-dimensional attitude skeleton, the guard plate size model, the measured radar point set, and the attitude state at the previous moment as constraints. ; The structured point cloud completion module includes a template deformation completion unit, a temporal memory completion unit, a graph structure completion unit, and a completion confidence calculation unit. The template deformation completion unit is used to project and constrain the template onto a two-dimensional attitude skeleton using a standard-sized model of the side panel as a template and measured radar points as depth anchor points. It solves for the optimal rigid body transformation and small-amplitude deformation parameters of the template in three-dimensional space, extracts completion points corresponding to missing areas from the transformed template, and generates a template deformation completion point set. ; The temporal memory completion unit is used to maintain the most recent... Frame-wise high-confidence pose results and corresponding historical pose memory queues of point clouds When the point cloud in the current frame is missing, the point cloud in the historical reliable frames is mapped to the current frame according to the pose transformation of adjacent frames, and then filtered according to the pose skeleton of the current image to generate a temporal memory completion point set. ; The graph completion unit uses hinged ends, free ends, edge points, measured radar points, and template sampling points as graph nodes. It establishes graph edges based on image skeleton adjacency relationships, 3D spatial adjacency relationships, and template topological relationships. Through multi-layer graph message passing, it aggregates the spatial location, skeleton structure, and node type information of neighboring nodes, predicts the 3D position offset and completion confidence of missing nodes, and generates a graph completion point set. ; The completion confidence calculation unit is used to process the three types of completion results through deduplication, spatial consistency filtering, and confidence fusion to form a comprehensive completion point set. Each completion point in Calculate the confidence level of completion The confidence level is obtained by weighting and summing four factors—image skeleton consistency error, depth anchor point consistency error, historical pose consistency error, and template deformation residual—using an exponential decay function and then truncating them with a clipping function; this unit also calculates the completion ratio. and average completion confidence This is used to quantify the dependence of the current pose estimation result on the completion algorithm and the overall reliability of the completion points; The joint pose estimation module is used to utilize the two-dimensional pose skeleton. Set of measured radar points Complete point set and side panel size model Establish a joint attitude optimization objective, and solve for the three-dimensional attitude state of the side panel by minimizing this objective. The joint pose optimization objective includes: constraining the image pose error after 3D pose projection to be consistent with image key points and pose skeleton, in order to supplement the confidence level. The point cloud matching error involving weighted completion points, with complementary confidence. Constraining the completion consistency error of low-confidence completion points, the geometric constraint error of the side panel size and mechanical motion range, and the temporal continuity error of the attitude continuity of adjacent frames; The confidence assessment and anomaly detection module is used to assess the confidence level of the image pose. BEV fusion confidence Radar measured point coverage score Optimize residual scoring Time-synchronized scoring Compensation ratio penalty and average completion confidence The reliability of attitude results calculated using seven factors when When the reliability exceeds the preset threshold, the module outputs the first... The three-dimensional attitude parameters of the side guard plate (including the three-dimensional coordinates of the hinged end, the three-dimensional coordinates of the free end, the plate surface normal vector, the unfolding angle, and the yaw angle), the directional cuboid space occupancy determined by the side guard plate size model and three-dimensional attitude, and the reliability of the attitude results. ; when When the reliability is below the preset reliability threshold, the module outputs a low-confidence attitude or a state to be reviewed, and marks the corresponding depth range of the guard plate. Meanwhile, the module determines whether there is an abnormal posture or cutting interference risk based on the unfolding angle, sway angle, free end spatial position and spatial occupation of the side guard plate; when the free end of the side guard plate intrudes into the preset safe area, it outputs a cutting interference warning and the corresponding risk level. This system deeply integrates infrared visual semantics with radar spatial geometry in BEV space, enabling stable estimation and anomaly warning of the sidewall attitude under complex underground working conditions, and providing a complete system-level solution for intelligent perception of fully mechanized mining faces.

[0039] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. A method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images, characterized in that, Includes the following steps: Step 1: Synchronously acquire multi-source data and perform time matching; synchronously acquire infrared image frames and 4D millimeter-wave radar point cloud frames in the target operation area, and perform data matching based on timestamps or reception times; Step 2: Construct a dual-branch joint detection network and perform multimodal feature extraction; Construct a dual-branch joint detection network containing image branch and radar branch, and perform feature extraction and position modeling on infrared image frames and radar point cloud frames respectively; Step 3: Perform BEV spatial feature fusion and output joint detection results; fuse image BEV features and radar BEV pseudo-image features in BEV space, and output the side guard plate target detection results, two-dimensional attitude skeleton and fusion confidence based on the fused features; Step 4: Project the radar point cloud onto the infrared image plane; Based on the pre-calibrated extrinsic parameters from the radar coordinate system to the infrared camera coordinate system and the intrinsic parameters of the infrared camera, the radar point cloud is projected onto the infrared image plane to establish the spatial correspondence between radar points and image pixels. Step 5: Match radar point cloud based on attitude skeleton constraint; using the attitude skeleton of the fused detection results and the target region of the image, perform attitude constraint matching on the radar points to obtain the set of measured radar points corresponding to each guard plate. Step 6: Structured completion of missing radar points; When the set of measured radar points does not meet the preset integrity conditions, a set of structured completion points is generated using the two-dimensional attitude skeleton, the guard plate size model, the set of measured radar points, and the historical attitude state. Step 7: Establish a joint attitude optimization objective and solve for the three-dimensional attitude; use the two-dimensional attitude skeleton, the set of measured radar points, the set of complete points, and the side guard plate size model to establish a joint attitude optimization objective and solve for the three-dimensional attitude state of the side guard plate. Step 8: Attitude result credibility assessment and anomaly judgment; calculate the attitude result credibility based on multi-dimensional quality indicators, and output the three-dimensional attitude, spatial occupancy and abnormal attitude judgment results of the guardrail.

2. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 1, the process of synchronously acquiring multi-source data and performing time matching is as follows: S11: Install an infrared camera at the rocker arm end of the coal mining machine or on the top of the hydraulic support, and at the same time, install a 4D millimeter-wave radar on the stable base adjacent to the infrared camera. S12: Infrared image frames and radar point cloud frames within the target operation area are acquired using an infrared camera and a 4D millimeter-wave radar, respectively. S13: Select matching infrared image frames and radar point cloud frames from the cache based on the timestamp or system reception time; when the infrared camera and radar do not have a unified hardware clock, use the system reception time of the data arriving at the same data processing device as the common time axis, and select the frame with the smallest absolute time difference as the matching frame; when the minimum time difference is greater than the preset maximum allowable time difference, terminate the current attitude estimation and output the data desynchronization status.

3. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 2, the process of constructing the dual-branch joint detection network and extracting multimodal features is as follows: S21: Construct a dual-branch joint detection network; the image branch uses a YOLO network, and multi-level features of the infrared image are extracted through the backbone network and the feature pyramid fusion layer. Then through the target detection head Obtain the detection bounding box, class label, and detection confidence score of the side panel, i.e., the image detection result. The attitude key point set of the guardrail is extracted using attitude key point extraction. ;in, For the current infrared image, For YOLO Layer image features, This represents the YOLO backbone and feature fusion operator, and the image detection results. Includes target bounding box, category, and detection confidence, as well as a set of image pose keypoints. This includes at least two types of attitude structures among the key points of the hinged end of the side guard plate, key points of the free end, edge points of the plate surface, and centerline parameters of the plate surface. For the target detection head mapping function, This is the attitude key point mapping function; The radar branch employs a PointPillars network, dividing the point cloud into cylinders using a BEV grid. Point feature vectors are constructed for points within each cylinder, comprising at least three of the following: 3D coordinates, reflection intensity, radial velocity, offset relative to the cylinder center, and offset relative to the centroid of the point set. Cylinder features are extracted via a shared multilayer perceptron and a max-pooling layer. To achieve multimodal feature extraction of radar BEV pseudo-images; S22: Multimodal features aligned in BEV space; Image features Extrinsic parameter matrix from radar coordinate system to infrared camera coordinate system and camera projection function Converted into image BEV features within the radar BEV space. ,in The center point of the BEV element in the coordinate system. The coordinates of the column center are For the first Image BEV features of each BEV unit, Sample represents the fusion weights for image features at different scales, and it denotes the bilinear sampling operation. For each point pair in the radar point cloud Construct feature vectors ,in, Let the coordinates of the point be in three-dimensional space. Radial velocity, This refers to either reflection intensity or signal-to-noise ratio. The coordinates of the centroid of the point set within the cylinder; Radar BEV pseudo-image features are obtained through MLP and max pooling. ,in For the first Radar BEV pseudo-image features of a BEV cell.

4. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 3, the process of fusing BEV spatial features and outputting the joint detection results is as follows: S31: Extract BEV features from the image and radar BEV pseudo-image features Attention-weighted fusion is performed to obtain fused BEV features. ,in and These are the fusion weights for the image branch and the radar branch, respectively; This indicates a channel-wise weighted operation; S32: Based on integrated BEV features The detection head outputs the target detection results of the guardrail, the two-dimensional attitude skeleton, and the fusion confidence score, and the first... Complete test output of individual protective board ,in For the first The center coordinates of the protective side panel on the BEV plane, in meters; , , These are the length, width, and height dimensions of the side panel, in meters (m). The angle of the side guard plate on the BEV plane is expressed in rad. The two-dimensional posture skeleton of the side guard plate is composed of the key points connecting the hinged end, the free end, and the edge of the plate. The confidence level for fusion detection is set to [0,1]. This is the mapping function for the fusion detection head.

5. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 4, the process of projecting the radar point cloud onto the infrared image plane is as follows: For the There are 1 radar point, whose homogeneous coordinates in the radar coordinate system are: According to the extrinsic parameter matrix Calculate its homogeneous coordinates in the infrared camera coordinate system. The projected pixel coordinates are calculated based on the infrared camera's intrinsic parameters and the pinhole imaging model, as shown in the following formula: ； ； In the formula, , These are the x and y coordinates of the projected pixels, respectively. , This refers to the focal length of the infrared camera in the horizontal and vertical directions of the image. , These are the x and y coordinates of the principal point of the infrared camera on the image plane. The depth of the radar point in the camera coordinate system; Only when And projected pixel coordinates If the radar point is within the effective image range, it is retained and proceeds to step 5 for attitude constraint matching calculation and determination of the actual radar point's affiliation; otherwise, the radar point is not involved in subsequent processing.

6. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 5, the process of matching radar point clouds based on attitude skeleton constraints is as follows: S51: Regarding the first Individual care board and the first The attitude constraint matching score is determined based on several radar points, including whether the projection point is located in the target area of ​​the guardrail image, the distance from the projection point to the attitude skeleton, the consistency of the projection point along the direction from the hinge end to the free end, the consistency of the projection point with the historical spatial position, and the fusion detection confidence. As shown in the following formula: ； In the formula, The indicative function is the pixel coordinates of the radar point projection. Located in the Image target area of ​​individual protective board If the condition is met, the value is 1; otherwise, the value is 0. Projecting pixels to the pose skeleton Normalized distance; This is the width parameter relative to the Gaussian kernel; The consistency score of the projected pixels along the direction from the hinged end to the free end is given. Scoring is given for the consistency between the projected pixels and their historical spatial locations; To determine the confidence level of the fusion detection; , , , , These are the non-negative weight coefficients for each sub-item, satisfying... ; S52: Attitude Constraint-Based Matching Get the Set of measured radar points for individual protective panels ,in To preset the matching score threshold, when multiple sidewall targets overlap, the same radar point is only assigned to the target with the highest matching score.

7. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 6, the process of structurally completing missing radar points is as follows: S61: When The number of points in the middle is less than the minimum number of measured points. If the measured points do not cover the hinged end, free end, or key areas of the plate surface, the comprehensive completion operator is invoked. Using the two-dimensional attitude skeleton, the side plate size model, the measured radar point set, and the attitude state at the previous moment as constraints, a structured completion point set is generated for missing edge points, free endpoints, corner points, plate surface mesh points, or equivalent geometric constraints. ,in This indicates the comprehensive completion operator. It is a two-dimensional attitude skeleton. This is a model of the side guard plate dimensions, including the actual length of the side guard plate. ,width and thickness , This represents the attitude state at the previous moment. For the first The historical posture memory queue of the individual protective board; S62: Perform template deformation completion; In template deformation completion, a standard template point set is established based on the actual dimensions of the side panel. Standard template point set This includes sampling points at the hinge end, free end, long side, short side, and plate surface mesh. By solving for the rigid body transformation and small deformation parameters of the template, the template is projected to fit the image's attitude skeleton, and the template's three-dimensional surface fits the measured radar points, as shown in the following formula: ； In the formula, For template rotation matrix, The template translation vector, For small deformations of the template point, Represents the projection function. This represents the error between the template projection contour and the image pose skeleton. This indicates the anchoring error between the template surface and the measured radar point. , , These are the non-negative weighting coefficients for each error term; Set of completion points obtained by template deformation completion As shown in the following formula: The corresponding area is Missing ; In the formula, For template sampling points, This refers to the small-amplitude deformation corresponding to the template point; S63: Complete the chronological memory; In temporal memory completion, the system maintains the most recent... Frame-high reliability pose results and corresponding point clouds are used to form a historical pose memory queue. When the point cloud in the current frame is missing, the point clouds in the reliable historical frames are mapped to the current frame according to the pose transformations of adjacent frames, and then filtered according to the pose skeleton of the current image. The completed point set is obtained by time-series memory completion. Satisfy the following formula: ； In the formula, Indicates the attitude state A defined three-dimensional rigid body transformation matrix. The current pose is predicted from the pose of the previous time step. For the first time in history The pose state of the frame. For the first time in history The set of measured radar points for the frame. To assess the reliability of pose in historical frames, The historical frame credibility threshold. The threshold for skeleton projection distance; S64: Perform graph structure completion; pass Layered graph message passing aggregates the spatial location, skeletal structure, and node type information of domain nodes in each layer to update node features. , final layer features Input a multilayer perceptron and output the 3D position offset of the missing node. and completion confidence ; in, For the first Nodes in a layered graph structure eigenvectors, For nodes The set of neighboring nodes, The attention weights are determined by the image skeleton distance, 3D distance, and node type. For activation function, , Given two learnable weight matrices, For the predicted position offset, To complete the confidence level, the value range is [0,1]; All missing nodes in the graph structure are offset by the predicted position. Corrected 3D coordinates and As the completion point, and corresponding completion confidence level. Relationships form the set of points for completing the graph structure. As shown in the following formula; Missing node ; S65: The comprehensive set of completion points is obtained by merging the three types of completion results. As shown in the following formula: ； In the formula, This represents the operations of deduplication, spatial consistency filtering, and confidence fusion. For any completion point Its completion confidence level As shown in the following formula: ； In the formula, To compensate for the error between the points projected onto the image and the pose skeleton, To compensate for the consistency error between the depth anchor point and the measured radar point, To compensate for the consistency error between the point and historical attitude predictions, For template deformation residuals, , , , These are non-negative weighting coefficients. Represents the amplitude limiting function; S66: Calculate the completion ratio and average completion confidence ,in It is a very small positive number, used to prevent the denominator from being zero.

8. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 1, characterized in that, In step 7, the process of establishing a joint attitude optimization objective and solving for the 3D attitude is as follows: S71: with Indicates the attitude state and size model Define the point cloud matching error for the determined 3D surface of the side panel. As shown in the following formula: ； In the formula, Point To three-dimensional surface The shortest Euclidean distance; S72: Define completion consistency error ; S73: A joint pose optimization objective is established based on image pose error, point cloud matching error, completion consistency error, geometric constraint error, and temporal continuity error. The final 3D pose state is obtained by minimizing the joint optimization objective. As shown in the following formula: ； In the formula, The constrained 3D pose projection is consistent with the image key points and pose skeleton. Constrain the dimensions of the side guard plate, hinge axis, and range of mechanical movement. Constrain the pose continuity of adjacent frames , , , , These are the non-negative weighting coefficients for each error term.

9. The method for estimating the attitude of a side panel based on 4D millimeter-wave radar and infrared images according to claim 8, characterized in that, In step 8, the process of attitude result reliability assessment and anomaly detection is as follows: S81: Order For optimal posture Normalized residuals at the location; Define radar coverage score Optimize residual scoring Rating in sync with time ,in To optimize the residual normalization scale; This represents the time difference between the image frame and the point cloud frame. This is the maximum permissible time difference; S82: Define the confidence level of attitude results ,in Image pose confidence. To determine the confidence level of BEV fusion detection, To supplement the proportional penalty index; S83: The system determines the reliability of the attitude results. Output the following results: when When the reliability exceeds the preset reliability threshold, output the first... The three-dimensional pose, spatial positioning, and reliability of the pose results of the personal protective equipment board. ;when When the reliability is below the preset reliability threshold, the system outputs a low-reliability attitude or a state pending verification, and marks the corresponding depth range of the guard plate. At the same time, the system judges whether there is an abnormal attitude or cutting interference risk based on the guard plate's unfolding angle, sway angle, free end spatial position, and spatial occupancy. When the free end of the guard plate intrudes into the preset safe area, the system outputs a cutting interference warning and the corresponding risk level.

10. A side panel attitude estimation system based on 4D millimeter-wave radar and infrared images, used to implement the side panel attitude estimation method based on 4D millimeter-wave radar and infrared images as described in any one of claims 1 to 9, characterized in that, include: The infrared image acquisition module is used to continuously acquire infrared image frames of the target area of ​​the protective panel and transmit the image frames to the dual-branch joint detection module and the time synchronization and frame matching module. The 4D millimeter-wave radar point cloud acquisition module is used to continuously acquire radar point cloud frames containing three-dimensional spatial coordinates, radial velocity, and reflection intensity, and transmit the point cloud frames to the dual-branch joint detection module and the time synchronization and frame matching module. The time synchronization and frame matching module is used to select matching frame pairs from the infrared image frame buffer and the radar point cloud frame buffer according to the timestamp or the system reception time. The sensor calibration module is used to provide spatial transformation parameters for image BEV feature projection and radar point cloud projection through offline joint calibration or on-site calibration. The dual-branch joint detection module is used to construct a dual-branch joint detection network that includes an image branch and a radar branch. It processes the time-synchronized infrared image frames and radar point cloud frames in parallel, fuses the two types of features in the BEV space, and outputs the target detection results of the guardrail, a two-dimensional attitude skeleton, and the fusion confidence score. The point cloud projection module is used to project the radar point cloud onto the infrared image plane based on the radar-infrared extrinsic matrix and infrared camera intrinsic parameters provided by the sensor calibration module, through homogeneous coordinate transformation and pinhole imaging model, and to establish the spatial mapping relationship between radar points and image pixels. The attitude constraint point cloud matching module is used to filter and determine the affiliation of the projected radar points by using the two-dimensional attitude skeleton, the image target region and the fusion detection confidence as prior constraints. The structured point cloud completion module is used to generate a structured completion point set containing missing edge points, free endpoints, corner points, and plate surface grid points when the measured radar point set does not meet the preset integrity conditions, with the two-dimensional attitude skeleton, the guard plate size model, the measured radar point set, and the attitude state at the previous moment as constraints. The joint attitude estimation module is used to establish a joint attitude optimization objective using a two-dimensional attitude skeleton, a set of measured radar points, a set of complete points, and a side plate size model. The three-dimensional attitude state of the side plate is solved by minimizing this objective. The credibility evaluation and anomaly detection module is used to calculate the credibility of the attitude results. It detects whether there is an abnormal posture or risk of cutting interference; when the free end of the guard plate intrudes into the preset safe area, it outputs a cutting interference warning and the corresponding risk level.

Images