High-place operation risk real-time judgment method based on image and point cloud bimodal data

Abstract

The invention discloses a high-altitude operation risk real-time judgment method based on image and point cloud bimodal data, and belongs to the crossing field of computer vision, artificial intelligence and engineering construction safety, and the method comprises the following steps: S1, obtaining a bimodal data stream through a laser radar and an RGB camera; s2, fitting an external reference registration matrix by acquiring an internal reference matrix of the camera to obtain a calibration parameter set; s3, optimizing the cue word, and driving the SAM-2 segmentation model to obtain a two-dimensional mask; s4, performing reverse projection on the two-dimensional mask to obtain a color point cloud instance set; s5, calculating the three-dimensional geometric centroid to obtain the three-dimensional geometric centroid and a three-dimensional semantic topology scene graph; and S6, obtaining a plurality of security rule compliance results by executing the interpretable rule engine. By the adoption of the method, the defects of an existing high-place operation risk judgment technology in the aspects of complex environment adaptability, multi-modal fusion efficiency, three-dimensional space perception and risk judgment interpretability are overcome.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and artificial intelligence and engineering construction safety, and in particular to a high-altitude operation risk real-time determination method based on image and point cloud dual-modal data. BACKGROUND

[0002] The high-altitude operation safety supervision technology has gradually developed towards automatic monitoring from the traditional safety personnel manual patrol and monitoring video viewing mode, forming two main technical paths: one is a single-modal computer vision monitoring scheme based on an RGB camera, which realizes automatic analysis of the operation scene relying on video stream data; the other is a multi-modal monitoring scheme trying to fuse multi-sensor data such as laser radar and image, and some schemes also introduce unmanned aerial vehicles, AI image recognition and other technologies to assist supervision.

[0003] However, the existing single-modal RGB monitoring scheme is seriously affected by light, weather, shielding and viewing angle, has insufficient robustness, high false positive rate and false negative rate, and lacks depth information, so it cannot determine the three-dimensional geometric position relationship of specific objects; the multi-sensor fusion scheme is either limited by algorithm efficiency, focuses on offline analysis and event backtracking in the back end, and is difficult to meet the real-time streaming determination demand, or only stays at the simple data level or feature level splicing, and fails to fully tap the deep complementarity of point cloud and image in geometry and semantics, so the timeliness and accuracy are not good; the more core limitation is that the existing system relies on static and two-dimensional manual rule knowledge base, lacks dynamic perception and automatic topology modeling capability of the three-dimensional space of the construction scene, cannot capture the spatial interaction relationship of key objects in the dynamic construction scene, and is difficult to identify potential spatial interaction risks, so the automatic supervision level is at a preliminary stage. SUMMARY

[0004] The purpose of the present application is to provide a high-altitude operation risk real-time determination method based on image and point cloud dual-modal data, which solves the above technical problems.

[0005] To achieve the above purpose, the present application provides a high-altitude operation risk real-time determination method based on image and point cloud dual-modal data, comprising the following steps: S1, based on the basic data demand of high-altitude operation risk real-time monitoring, coupling laser radar and RGB camera through high-rigidity mechanical structure, hardware-level synchronous triggering mechanism, obtaining dual-modal data stream composed of three-dimensional point cloud and two-dimensional image aligned by milliseconds; S2, based on the dual-modal original data collected in step S1, obtaining a calibration parameter set supporting high-precision fusion through laboratory calibration to obtain a camera intrinsic matrix and on-site data fitting to obtain an extrinsic registration matrix; S3, based on the two-dimensional RGB image collected in step S1, dynamically optimizing the prompt word through the uncertainty perception adaptation module UPA, and inputting the optimized prompt word into The open vocabulary detection model generates a two-dimensional detection box of the key safety element, and uses the detection box as spatiotemporal prompt information to drive the SAM-2 deep segmentation model to obtain a high-robustness two-dimensional mask containing key safety element information labels; S4, based on the calibration parameter set in step S2, reversely projecting the two-dimensional mask image in step S3, matching the projected mask pixels with the three-dimensional point cloud in step S1, and obtaining a color point cloud instance set with semantic labels; S5, based on the color point cloud instance set obtained in step S4, sequentially passing through the statistical outlier removal SOR preprocessing, three-dimensional geometric centroid calculation, and ground plane fitting to obtain a three-dimensional geometric centroid of the key safety element information label and a three-dimensional semantic topological scene graph constructed based on the centroid position; S6, based on the three-dimensional semantic topological scene graph, the three-dimensional geometric centroid, and the fitted ground plane constructed in step S5, performing deep geometric analysis and multi-factor Boolean weighting model quantification through an interpretable rule engine to obtain a multi-item safety rule compliance result and a real-time risk score.

[0006] Preferably, the dual-mode data synchronous collection step in step S1 comprises: Based on the spatial position stability and data fusion accuracy requirements of the laser radar and the RGB camera, the laser radar and the RGB camera are spatially locked and coupled through a high-rigidity mechanical structure to obtain a constant device relative position relationship and a solidified external parameter matrix; Based on the strong time sequence consistency requirement of the dual-mode data, an edge computing device is used as a master unit to control the laser radar and the RGB camera to collect at the same frame rate using a hardware synchronization signal to obtain a three-dimensional point cloud and a two-dimensional RGB image dual-mode data stream with minimized timestamp alignment error.

[0007] Preferably, the specific steps of step S2 comprise: S21, using an RGB camera to shoot a calibration board in a suitable laboratory environment to obtain an internal parameter matrix of the RGB camera; S22, based on the dual-mode data stream in step S1, applying an automatic pixel-level calibration algorithm to extract edge features and fit an external parameter registration matrix to obtain an external parameter registration matrix supporting accurate fusion of three-dimensional point cloud and RGB image; S23, based on the external parameter registration matrix in step S22 and the internal parameter matrix in step S21, the complete calibration parameter set supporting high-precision fusion is obtained by summarizing and arranging.

[0008] Preferably, in step S3, based on the low latency requirement of real-time on-site judgment, the high-performance GPU of the edge computing device is used to accelerate the open vocabulary target detection model, subsequent image segmentation model inference, and three-dimensional geometric fitting algorithm.

[0009] Preferably, step S3 includes the following specific steps: S31. Based on the distribution characteristics of the detection confidence of key safety elements under the current environmental noise, the initial prompt word set is dynamically corrected and replaced by the UPA module to obtain adaptive optimized prompt words that are suitable for complex environments. The optimization algorithm formula is as follows: ; in, These are the optimized adaptive suggestion words; For the first word in the initial prompt set One prompt word; This is a pre-defined list of candidate synonyms or an expanded vocabulary. for The detection model is based on prompt words. For the target The semantic confidence score of the output candidate bounding boxes; Based on prompt words The expected detection confidence; Based on prompt words The variance of the detection confidence level; Risk aversion coefficient; S32. Based on the two-dimensional RGB image acquired in step S1 and the optimized prompt words obtained in step S31, firstly through... The model performs cross-modal feature fusion and semantic alignment to locate the target in the image space and output the corresponding bounding box. Then, the bounding box is used as guiding information to input the SAM-2 model for pixel-level instance segmentation, accurately identifying and distinguishing key security elements, and obtaining a highly robust two-dimensional color mask with instance ID.

[0010] Preferably, the key safety elements in step S32 include construction workers, personal protective equipment (PPE), ropes, anchor points, and guardrails.

[0011] Preferably, step S4 includes the following specific steps: S41. Based on the intrinsic and extrinsic registration matrices obtained in step S2, the two-dimensional color mask pixels from step S3 are mapped using an inverse projection algorithm. Mapping to three-dimensional space establishes a spatial correspondence between mask pixels and three-dimensional point clouds; S42, based on the three-dimensional point cloud data collected in step S1, the mask pixels obtained by step S41 are precisely aligned with the three-dimensional point cloud of the corresponding frame through the same frame timestamp matching mechanism, and a specific object color point cloud instance set with semantic labels is obtained.

[0012] Preferably, the specific steps of step S5 include: S51, based on the noise purification requirements of the color point cloud instance set, the specific object point cloud instance is preprocessed through the SOR algorithm to remove isolated noise points generated by measurement noise and incorrect projection, and pure point cloud instance data is obtained; S52, based on the point cloud instance data of step S51, the three-dimensional geometric centroid of each key safety element is calculated through the inverse distance square weighted centroid fitting algorithm, and the calculation formula is: ; Among them, , , , is the weight of the point in the point cloud, and , is the distance from the point in the point cloud to the sensor; , , is the three-dimensional coordinate of the point in the point cloud; is the three-dimensional geometric centroid of the key safety element; S53, the ground point set is robustly fitted by using the RANSAC algorithm to obtain the ground plane equation , wherein, , , , is the plane equation parameter; S54, based on the three-dimensional geometric centroid of each key object obtained in step S52 and the ground plane obtained in step S53, the object-geometry relationship mapping is established to construct a three-dimensional semantic topological scene graph , wherein the vertex set formula is: ; Among them, is the construction personnel, is the safety belt, is the hanging rope, is the hanging point, is the adjacent edge, is the protective fence; The edge set formula is: ; in, Represents a geometric relationship that is higher than; Represents the geometric relationship to which it is attached; Represents the geometric relationship of proximity.

[0013] Preferably, step S6 includes the following specific steps: S61. Based on the requirements for determining the status of high-altitude operations, the centroid of the construction worker is calculated using the distance calculation formula from a point to a plane. to ground plane vertical distance To verify whether construction workers are performing high-altitude operations, the calculation formula is as follows: ; And when At that time, it was determined to be a high-altitude operation. This is the threshold for the starting height of high-altitude operations; S62. Based on the compliance requirements for hanging point height, by comparing the center of gravity of construction personnel... With the center of mass of the hanging point The Z-axis coordinate is used to verify the compliance of the hanging point height. The determination formula is as follows: ; in, The Z-axis coordinate of the centroid of the hanging point; The Z-axis coordinate of the centroid of the construction personnel; The minimum difference threshold for the hanging point height compliance; S63. Based on the compliance requirements for safety rope length, calculate the set of construction worker points using the nearest point search algorithm. and neighboring point set Shortest Euclidean distance and the length of the safety rope A comparison is performed to verify the compliance of the rope length; the comparison formula is as follows: ; in, The safety factor is the rope length. This is the shortest Euclidean distance; S64. Based on the compliance requirements of work posture, the point cloud of construction personnel is determined using the PCA algorithm. attitude normal vector The point cloud of the safety rope was fitted using the RANSAC algorithm. Obtain its direction vector Calculate the angle between the two. And verify whether it meets the safety threshold. ; S65, quantifying the demand based on the overall risk, quantifying the compliance results of steps S61-S64 through a multi-factor Boolean weighted model to obtain a real-time risk score , the scoring formula is: ; Wherein, 、 、 is a Boolean indicator function, which takes the value of 1 when the condition in the bracket is met, otherwise 0; 、 、 is a preset risk weight; is in the state of high-altitude operation; is that the construction personnel are too close to the adjacent side, is the adjacent side safety distance threshold; is the personnel-hanging rope-hanging point core safety element ternary group topology verification unqualified.

[0014] Preferably, the high-altitude operation risk real-time determination system based on image and point cloud dual-mode data comprises: a high-precision coupled acquisition module for synchronously acquiring dual-mode data, the high-precision coupled acquisition module comprising a fixedly coupled laser radar and an RGB camera; an edge computing processing unit for performing all computing tasks in real time on site, adopting , the edge computing processing unit internally deploying an UPA module for dynamically self-adapting correction of initial prompt words, an open vocabulary detection module for generating a target detection frame, and an adaptive semantic segmentation module for image segmentation; a three-dimensional high-precision mapping module, the three-dimensional high-precision mapping module being connected with the edge computing processing unit to utilize a pre-calibrated intrinsic parameter matrix and an extrinsic parameter matrix to accurately project a two-dimensional mask to a three-dimensional point cloud; a real-time scene topology reconstruction module, the real-time scene topology reconstruction module being connected with the three-dimensional high-precision mapping module; an interpretable rule determination module, the interpretable rule determination module being connected with the real-time scene topology reconstruction module.

[0015] Therefore, the high-altitude operation risk real-time determination method and system based on image and point cloud dual-mode data have the beneficial effects that: 1. By introducing the uncertainty perception adaptation UPA module, the dynamic self-adaptive adjustment of the image recognition prompt word is realized, effectively solving the pain points of insufficient recognition accuracy of traditional recognition models in poor environments such as weak light, severe occlusion and high image noise, and innovatively adopting the "UPA semantic optimization + The cascade architecture of "spatial positioning + SAM-2 fine segmentation", With its open vocabulary detection capability, it makes up for the short board of the SAM-2 model lacking text understanding capability, converts abstract semantic labels into precise geometric bounding boxes, and guides the segmentation edge to be high-precision, significantly improves the detection rate and semantic classification correctness of "hanging points" and "safety belts" and other small targets in complex semantic scenes, and can stably output high-quality semantic segmentation results in complex and variable construction site environments, providing reliable two-dimensional foundation support for subsequent three-dimensional data fusion and risk judgment, and having stronger environmental adaptability and recognition robustness than existing solutions.

[0016] 2, relying on the powerful computing power of edge computing equipment, the object recognition segmentation, three-dimensional geometric analysis and risk scoring whole process are systematically optimized, and an end-to-end efficient judgment architecture is constructed, which completely gets rid of the dependence on artificial patrol and offline analysis, realizes real-time intelligent judgment of high-altitude operation compliance, not only significantly reduces the delay and subjective error risk caused by human monitoring, but also quickly responds to the dynamic changes of the construction site, timely captures potential safety hazards, effectively meets the core needs of modern engineering management for safety supervision "all-weather, full coverage, low delay", and greatly improves the efficiency and reliability of construction site safety supervision.

[0017] 3, by accurately projecting the two-dimensional segmentation mask to the three-dimensional point cloud, the accurate quantification and automatic three-dimensional topological modeling of the three-dimensional geometric position relationship of the key objects in the scene are realized, compared with the traditional risk judgment method based on static two-dimensional rules, the present application takes the geometric consistency formula as the core, constructs the "person-hanging rope-hanging point" ternary topological verification mechanism, clearly defines the compliance judgment basis of each safety rule, so that the risk score is no longer dependent on fuzzy empirical indicators, but has clear geometric logic support, not only makes the risk assessment result more persuasive and explainable, but also accurately identifies potential spatial interaction risks, greatly improves the scientificity and accuracy of high-altitude operation risk assessment.

[0018] The technical solutions of the present application will be further described in detail below with the help of the drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 The flowchart of the high-altitude operation risk real-time judgment method based on image and point cloud dual-mode data provided by the present application.

[0020] Figure 2 The system architecture diagram of the high-altitude operation risk real-time judgment method based on image and point cloud dual-mode data provided by the present application.

[0021] Figure 3The algorithm flowchart of the high-altitude operation risk real-time judgment method based on image and point cloud dual-modal data provided by the application is shown in the figure. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the embodiments of the application will be further described in detail below with reference to the drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the embodiments of the application and not to limit the embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor belong to the scope of protection of the present application. The examples of the embodiments are shown in the drawings, wherein the same or similar reference numerals represent the same or similar elements or elements with the same or similar functions throughout.

[0023] It should be noted that the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or server including a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0024] The embodiments of the application will be described in detail below with reference to the drawings.

[0025] The existing high-altitude operation safety supervision technology has gradually developed from the traditional safety officer manual patrol and monitoring video viewing mode to the single-modal computer vision monitoring based on RGB camera and the automatic monitoring scheme of multi-sensor data fusion, and the improvement of computer vision image recognition technology and edge computing device performance provides technical support for real-time detection, but there are still significant defects: the single-modal scheme is greatly affected by light, weather, shielding and viewing angle, has insufficient robustness and lacks depth information, and cannot determine the three-dimensional geometric position relationship; the multi-modal scheme focuses on offline analysis in the back end, which is difficult to meet the real-time demand, or only stays at the simple data level / feature level splicing, and fails to fully play the complementary nature of data, with poor timeliness and accuracy; the core limitation is that the existing system relies on static two-dimensional manual rule knowledge base, lacks dynamic perception and automatic topology modeling capability for three-dimensional space of building scene, and is difficult to capture the spatial interaction relationship of key objects, cannot effectively identify potential spatial interaction risks, and has low level of automatic supervision.

[0026] The application is designed based on the above analysis, referring to the figure Figures 1-3 The high-altitude operation risk real-time judgment method based on image and point cloud dual-modal data comprises the following steps: S1, based on the real-time monitoring of the high-altitude operation risk, the basic data requirement is obtained by coupling the laser radar and the RGB camera through the high-rigidity mechanical structure, the hardware-level synchronous triggering mechanism, the dual-mode data stream composed of the millisecond-level time alignment three-dimensional point cloud and the two-dimensional image; The dual-mode data synchronous acquisition step in step S1 includes: Based on the spatial position stability and data fusion accuracy requirement of the laser radar and the RGB camera, the laser radar and the RGB camera are spatially locked and coupled through the high-rigidity mechanical structure, so as to obtain the constant device relative position relationship and the fixed external parameter matrix. Based on the strong time sequence consistency requirement of the dual-mode data, the edge computing device is used as the main control unit, the laser radar and the RGB camera are collected at the same frame rate by using the hardware synchronization signal, and the dual-mode data stream of the three-dimensional point cloud and the two-dimensional RGB image with minimized timestamp alignment error is obtained.

[0027] S2, based on the dual-mode raw data collected in step S1, the camera internal parameter matrix is obtained through laboratory calibration, the external parameter registration matrix is fitted through on-site data collection, and the calibration parameter set supporting high-precision fusion is obtained. The specific steps of step S2 include: S21, the internal parameter matrix of the RGB camera is obtained by using the RGB camera to shoot the calibration board in a suitable laboratory environment with appropriate illumination; S22, based on the dual-mode data stream of step S1, the automatic pixel-level calibration algorithm is applied to extract the edge features and fit the external parameter registration matrix, so as to obtain the external parameter registration matrix supporting the accurate fusion of the three-dimensional point cloud and the RGB image; S23, based on the external parameter registration matrix of step S22 and the internal parameter matrix of step S21, the complete calibration parameter set supporting high-precision fusion is obtained by summarizing and arranging.

[0028] S3, based on the two-dimensional RGB image collected in step S1, the prompt word is dynamically optimized by the uncertainty perception adaptation module UPA, and the optimized prompt word is input The two-dimensional detection frame of the key safety element is generated by the open vocabulary detection model, the detection frame is used as the spatio-temporal prompt information to drive the SAM-2 deep segmentation model, and the high-robustness two-dimensional mask containing the key safety element information identifier is obtained; In step S3, based on the low-delay requirement of real-time judgment on site, the open vocabulary target detection model, the subsequent image segmentation model inference and the three-dimensional geometric fitting algorithm are accelerated by the high-performance GPU of the edge computing device; The specific steps of step S3 include: S31. Based on the distribution characteristics of the detection confidence of key safety elements under the current environmental noise, the initial prompt word set is dynamically corrected and replaced by the UPA module to obtain adaptive optimized prompt words that are suitable for complex environments. The optimization algorithm formula is as follows: ; in, These are the optimized adaptive suggestion words; For the first word in the initial prompt set One prompt word; This is a pre-defined list of candidate synonyms or an expanded vocabulary. for The detection model is based on prompt words. For the target The semantic confidence score of the output candidate bounding boxes; Based on prompt words The expected detection confidence; Based on prompt words The variance of the detection confidence level; This is the risk aversion coefficient, used to balance expected returns and uncertainty, ensuring that while maintaining high detection accuracy, robustness to different environmental factors is enhanced. S32. Based on the two-dimensional RGB image acquired in step S1 and the optimized prompt words obtained in step S31, firstly through... The model performs cross-modal feature fusion and semantic alignment, locates targets in image space, and outputs corresponding bounding boxes. These bounding boxes are then used as guiding information to input into the SAM-2 model for pixel-level instance segmentation, accurately identifying and distinguishing key security elements, resulting in a highly robust two-dimensional color mask with instance IDs. This is achieved by introducing... This solves the problem that SAM-2 cannot directly understand natural language instructions; the UPA module outputs adaptive prompts. go through The bidirectional cross-modal attention mechanism can accurately locate local areas in large backgrounds or complex occlusion environments through text descriptions, ropes, or hanging points. The generated detection boxes effectively reduce the search space of SAM-2 and significantly reduce the segmentation false alarm rate. The key safety elements in step S32 include construction workers, personal protective equipment (PPE), ropes, anchor points, and guardrails.

[0029] S4. Based on the calibration parameter set of step S2, the two-dimensional mask image of step S3 is reverse-projected so that the mask pixels obtained by projection are matched with the three-dimensional point cloud of step S1 in the same frame, and a set of color point cloud instances with semantic labels is obtained. The specific steps of step S4 include: S41, based on the internal parameter matrix and the external parameter registration matrix obtained in step S2, mapping the two-dimensional color mask pixels in step S3 to a three-dimensional space through an inverse projection algorithm to establish a spatial correspondence between the mask pixels and the three-dimensional point cloud; S42, based on the three-dimensional point cloud data collected in step S1, precisely aligning the mask pixels obtained through projection in step S41 with the three-dimensional point cloud of the corresponding frame through a same frame timestamp matching mechanism to obtain a specific object color point cloud instance set with semantic labels.

[0030] S5, based on the color point cloud instance set obtained in step S4, sequentially performing statistical outlier removal SOR preprocessing, three-dimensional geometric centroid calculation, and ground plane fitting to obtain a three-dimensional geometric centroid of a key safety element information identifier and a three-dimensional semantic topological scene graph constructed based on the centroid position; The specific steps of step S5 include: S51, based on the noise purification requirements of the color point cloud instance set, preprocessing the specific object point cloud instance through an SOR algorithm to remove isolated noise points generated by measurement noise and incorrect projection to obtain pure point cloud instance data; S52, based on the point cloud instance data in step S51, calculating the three-dimensional geometric centroid of each key safety element through an inverse distance square weighted centroid fitting algorithm, and the calculation formula is: ; wherein, , , , is the weight of the point in the point cloud, and , is the distance from the point in the point cloud to the sensor; , , is the three-dimensional coordinates of the point in the point cloud; is the three-dimensional geometric centroid of the th key safety element; S53, robustly fitting a ground point set to obtain a ground plane equation wherein, , , , are plane equation parameters; S54, based on the three-dimensional geometric centroid of each key object obtained in step S52 and the ground plane obtained in step S53, constructing a three-dimensional semantic topological scene graph by establishing an “object-geometry relationship” mapping ​wherein the vertex set formula is: ; wherein, is the construction worker, is the safety belt, is the hanging rope, is the hanging point, is the adjacent edge, is the guardrail; the edge set formula is: ; wherein, represents a geometric relationship higher than; represents a geometric relationship attached to; represents a geometric relationship adjacent to.

[0031] S6, based on the three-dimensional semantic topological scene graph, the three-dimensional geometric centroid and the fitted ground plane constructed in step S5, a deep geometric analysis is performed through an interpretable rule engine and a multi-factor Boolean weighted model quantization, to obtain a plurality of safety rule compliance results and a real-time risk score; The specific steps of step S6 include: S61, based on the high-altitude operation state determination requirement, the vertical distance of the construction worker centroid to the ground plane is calculated through the point-to-plane distance calculation formula to verify whether the construction worker is performing high-altitude operation, and the calculation formula is: ; ; and when , it is determined that it is a high-altitude operation state, is the high-altitude operation starting height threshold; S62, based on the hanging point height compliance requirement, the Z-axis coordinates of the construction worker centroid and the hanging point centroid are compared to verify the hanging point height compliance, and the determination formula is: ; wherein, is the Z-axis coordinate of the hanging point centroid; is the Z-axis coordinate of the construction worker centroid; is the minimum difference threshold of the hanging point height compliance; S63, based on the safety rope length compliance requirement, the shortest Euclidean distance between the construction worker point set and the adjacent edge point set is calculated through the nearest point search algorithm, and compared with the safety rope length The comparison is made to verify the rope length compliance, and the comparison formula is: ; wherein, is the rope length safety factor; is the shortest Euclidean distance, and ; S64, based on the work posture compliance requirement, the posture normal vector of the point cloud of the construction personnel is determined by the PCA algorithm , the direction vector of the safety rope point cloud is obtained by the RANSAC algorithm , the included angle between the two is calculated and whether it meets the safety threshold is verified , the formula is: ; ; wherein, is the included angle between the posture normal vector of the construction personnel and the direction vector of the safety rope; is the maximum allowed included angle threshold of the posture compliance, and when , the posture compliance is determined; S65, based on the overall risk quantification requirement, the compliance results of steps S61-S64 are quantitatively scored by a multi-factor Boolean weighted model, to obtain a real-time risk score , the scoring formula is: ; wherein, , , is a Boolean indicator function, which takes a value of 1 when the condition in the bracket is met, and 0 otherwise; , , is a preset risk weight; is in the state of high-altitude work; is that the construction personnel is too close to the adjacent edge, is the adjacent edge safety distance threshold; is the personnel-hanging rope-hanging point core safety element ternary group topology verification unqualified.

[0032] The high-altitude work risk real-time determination system based on image and point cloud dual-mode data, as shown in Figure 2 , comprises: a high-precision coupled acquisition module for synchronously acquiring dual-mode data, the high-precision coupled acquisition module comprising a fixedly coupled laser radar and an RGB camera, to ensure that the relative position relationship between the sensors is stable; the laser radar is selected from ​The lidar, with its non-repeating scanning characteristics and high point cloud density, ensures comprehensive coverage and high-precision acquisition of three-dimensional spatial information; the RGB camera is selected... This series of industrial cameras features high frame rates and global shutter capabilities, effectively avoiding motion blur and ensuring high-quality image capture. It also supports... Precise synchronization; The lidar and RGB camera are fixed to the same base by a high-rigidity mechanical bracket to ensure that the extrinsic registration matrix between the two is constant. Edge computing processing units are used to perform all computing tasks in real time on-site. Its integrated high-performance GPU and dedicated deep learning accelerator can meet the stringent real-time processing requirements of open-vocabulary object detection models, subsequent image segmentation model inference, and 3D geometric fitting algorithms; the UPA module for dynamically adaptively correcting the initial prompt words and the module for generating object detection boxes... An open vocabulary detection module and an adaptive semantic segmentation module for image segmentation; A three-dimensional high-precision mapping module is connected to the edge computing processing unit to accurately project a two-dimensional mask onto a three-dimensional point cloud using pre-calibrated intrinsic and extrinsic parameter matrices. A real-time scene topology reconstruction module is connected to the three-dimensional high-precision mapping module. It uses a weighted centroid fitting algorithm to extract the centroids of key objects in real time and establishes a three-dimensional semantic topology map based on the ground plane fitted by RANSAC. An interpretable rule determination module is connected to the real-time scene topology reconstruction module. The interpretable rule determination module includes a deep geometry analysis engine, which performs real-time compliance verification of risks in the scene based on algorithms such as vertical distance calculation, PCA pose fitting and shortest distance search, and outputs a quantitative risk score. The acquired point cloud data and color image data are transmitted via Gigabit Ethernet and USB 3.2 high-speed interfaces, respectively. To perform subsequent real-time processing; Hardware-level synchronous triggering of sensor recording: utilizing The camera's external trigger interface and The PTP / PPS signal or a custom sync line, in Under the master control, high-precision hardware synchronous triggering of dual-modal sensors with the same high frame rate is achieved to ensure point cloud frame rate. With RGB image frames Strong temporal consistency.

[0033] Example: Data processing flow based on the real-time assessment method for compliance risks in high-altitude operations, see [link / reference]. Figure 3As shown, it includes, in sequence: 1. Information is collected through a dual-modal data acquisition module to obtain synchronized RGB images. and point cloud data ; 2. The data enters the UPA module, which dynamically optimizes the initial prompt words based on the confidence distribution and outputs the corrected prompt words. ; 3. The target detection and semantic segmentation cascade processing module receives... and Generate a two-dimensional color instance mask ; Specifically, the process for generating a 2D color instance mask is as follows: 3.1 Open vocabulary target detection: Adaptive prompt words corrected by UPA are used. Compared with the current RGB image Simultaneously input The model retrieves image features through a cross-modal attention mechanism and outputs a set of bounding boxes containing key features. and its corresponding semantic confidence; 3.2 Prompt for driver segmentation: [The following text appears to be a separate, unrelated section:] ... Output bounding box SpatialPrompts, as spatial geometric cues, are fed into the cue encoder of the SAM-2 model. Based on the bounding box cues, SAM-2 then... Feature decoding is performed to generate a two-dimensional color instance mask with pixel-level precision. And assign a unique semantic ID to the mask based on the category label of the detection box.

[0034] 4. Through the 3D mapping and point cloud coloring module, Precise reverse projection to Forming a colored semantic point cloud Specifically: Using high-precision extrinsic matrix and intrinsic parameter matrix The two-dimensional semantic mask generated by SAM-2 The pixels in the image are precisely reverse-projected onto... 3D point cloud collected In the process, a set of 3D point cloud instances of specific objects with color and instance semantic labels is generated. The formula is: ; 5. Geometric feature extraction module Preprocessing is performed and a weighted centroid fitting algorithm is applied to extract the object's centroid. Meanwhile, the ground plane was obtained through RANSAC fitting. .based on and And constructed a 3D scene map. Preprocessing includes: projecting the object point cloud instance set into SAM-2. Perform statistical outlier removal or radius filtering to remove isolated noise points caused by measurement noise or incorrect projection; 6. The interpretable rule engine performs deep geometric analysis, calculates various compliance indicators, and performs verification based on three-dimensional geometric relationships, including: Verification of high-altitude work status: for construction personnel center of mass and ground plane Calculate the perpendicular distance based on the formula for the distance from a point to a plane. The judgment rule is : ; Verification of compliance of hanging point height: ; Safety rope length compliance verification: for construction workers' points and neighboring edge set The nearest point search algorithm is used to quickly calculate the shortest Euclidean distance between two point sets. The judgment rule is : ; Work posture compliance verification: Extract the main posture direction of the construction worker and the normal vector of the safety rope, calculate the angle between the two, and the judgment rule is as follows. The specific steps are as follows: For construction workers Point cloud examples Principal Component Analysis (PCA) algorithm is used to determine its main geometric orientation, specifically the orientation of the first principal component. Approximate to the main posture direction of a person, it is usually defined as being in relation to The perpendicular vectors with opposite directions are obtained by fitting the local point cloud using the least squares method; For the safety rope normal vector Point clouds of safety ropes can be fitted using RANSAC. Given a three-dimensional straight line, then take the direction vector of that line; calculate and The angle between : ; 7. The risk quantification assessment module uses a Boolean weighted model to calculate a quantitative risk score. The output module provides real-time alerts and reports, specifically: The overall risk score is calculated using a multi-factor Boolean weighted model. The model transforms geometric non-compliance events into Boolean metrics and assigns them preset weights. The risk score is calculated using the following formula for quantitative analysis: ; This verification comprehensively assessed the compliance of the attachment point height and the posture angle. When the geometric topological relationship between core safety elements does not meet the preset safety specifications, such as the attachment point being lower than the person's waist or the posture angle... This risk is activated and takes a value of and multiplied by weight Weight : These are risk weights preset by managers based on safety regulations and experience, reflecting the severity of each risk; The compliance assessment module outputs quantitative risk scores in real time. and according to Does the value exceed the total risk threshold? Decide whether to trigger an alarm.

[0035] This embodiment selects LiDAR and Industrial cameras and edge computing devices use The module uses SAM-2 for image segmentation and RANSAC for point cloud fitting. The choice of hardware and algorithms is not fixed; those skilled in the art can adjust or change the model and parameters according to on-site needs, as long as it still achieves the desired result. Figure 1 The methods and processes shown are all within the protection scope of this invention.

[0036] In summary, this invention specifically addresses the shortcomings of existing high-altitude operation safety supervision technologies. These include the insufficient robustness and lack of depth information in single-modal recognition schemes under complex environments such as low light and severe occlusion, the poor real-time performance of multi-modal fusion schemes, and the failure to fully exploit the deep complementarity of data. Furthermore, it addresses the core limitations of existing systems, such as reliance on static two-dimensional rules, lack of automated three-dimensional topology modeling capabilities, inability to accurately capture the spatial interaction relationships of key objects, and weak interpretability of risk assessment. Ultimately, this invention achieves highly robust recognition of key objects in complex scenarios, high-precision real-time fusion of dual-modal data, dynamic perception and modeling of three-dimensional spatial relationships, and more accurate and interpretable risk assessment.

[0037] It should be pointed out finally that the above examples are only used to illustrate the technical solutions of the present application but not to limit it, and although the present application has been described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solutions of the present application can still be modified or replaced equivalently, and these modifications or equivalent replacements should not make the modified technical solutions deviate from the spirit and scope of the technical solutions of the present application.

Claims

1. A high-altitude work risk real-time determination method based on image and point cloud bimodal data, characterized in that: The method comprises the following steps: S1, based on the real-time monitoring of the high-risk working risk, the laser radar and the RGB camera are coupled through a high-rigidity mechanical structure, a hardware-level synchronous triggering mechanism is used, and a double-mode data stream composed of a three-dimensional point cloud and a two-dimensional image aligned in milliseconds is obtained; S2, based on the double-mode original data collected in step S1, a camera intrinsic matrix is obtained through laboratory calibration, and a registration matrix is fitted through on-site data collection, so that a calibration parameter set supporting high-precision fusion is obtained; S3, based on the two-dimensional RGB image collected in step S1, dynamically optimizing the prompt word through the uncertainty perception adaptation module UPA, and inputting the optimized prompt word into The open vocabulary detection model generates a two-dimensional detection box of the key safety element, and drives the SAM-2 deep segmentation model by using the detection box as spatiotemporal prompt information to obtain a high-robustness two-dimensional mask containing key safety element information identification. S4, based on the calibration parameter set in step S2, the two-dimensional mask image in step S3 is reversely projected, the mask pixels obtained through projection are matched with the three-dimensional point cloud in step S1, and a color point cloud instance set with semantic labels is obtained; S5, based on the color point cloud instance set obtained in step S4, the three-dimensional geometric centroid and the three-dimensional semantic topological scene graph constructed based on the centroid position are obtained through SOR preprocessing, three-dimensional geometric centroid calculation and ground plane fitting in sequence; S6, based on the three-dimensional semantic topological scene graph, the three-dimensional geometric centroid and the fitted ground plane constructed in step S5, a deep geometric analysis and a multi-factor Boolean weighted model quantization are performed through an interpretable rule engine, and a multi-item safety rule compliance result and a real-time risk score are obtained.

2. The method of claim 1, wherein the method comprises: The double-mode data synchronous collection step in step S1 comprises: Based on the spatial position stability and data fusion accuracy requirement of the laser radar and the RGB camera, the laser radar and the RGB camera are spatially locked and coupled through a high-rigidity mechanical structure, so that a constant device relative position relationship and a fixed registration matrix are obtained; Based on the strong time sequence consistency requirement of the double-mode data, an edge computing device is used as a master unit, a hardware synchronization signal is used to control the laser radar and the RGB camera to collect at the same frame rate, and a three-dimensional point cloud and a two-dimensional RGB image double-mode data stream with minimized timestamp alignment error are obtained.

3. The method of claim 2, wherein the method further comprises: The specific steps of step S2 comprise: S21, a calibration board is photographed by using the RGB camera in a laboratory environment with suitable illumination, and an intrinsic matrix of the RGB camera is obtained; S22, based on the double-mode data stream in step S1, an automatic pixel-level calibration algorithm is applied to extract edge features and fit a registration matrix, so that a registration matrix supporting accurate fusion of the three-dimensional point cloud and the RGB image is obtained; S23, based on the registration matrix in step S22 and the intrinsic matrix in step S21, a complete calibration parameter set supporting high-precision fusion is obtained through summarization and arrangement.

4. The method of claim 3, wherein the method further comprises: In step S3, based on the low-delay requirement of on-site real-time determination, a high-performance GPU of an edge computing device is used to accelerate the open vocabulary target detection model, the subsequent image segmentation model inference and the three-dimensional geometric fitting algorithm.

5. The method of claim 4, wherein the method further comprises: The specific steps of step S3 comprise: S31, based on the detection confidence distribution characteristics of the key safety elements under the current environmental noise, an optimization strategy of maximizing the expected confidence and minimizing the confidence variance is performed through a UPA module, the initial prompt word set is dynamically modified and replaced, and an adaptive optimization prompt word suitable for complex environments is obtained, and the optimization algorithm formula is: ； in, These are the optimized adaptive suggestion words; For the first word in the initial prompt set One prompt word; This is a pre-defined list of candidate synonyms or an expanded vocabulary. for The detection model is based on prompt words. For the target The semantic confidence score of the output candidate bounding boxes; Based on prompt words The expected detection confidence; Based on prompt words The variance of the detection confidence level; Risk aversion coefficient; S32、based on the two-dimensional RGB image collected in step S1 and the optimized prompt word obtained in step S31, first pass through the model to perform cross-modal feature fusion and semantic alignment, locate the target in the image space and output the corresponding bounding box; then input the bounding box as guide information into the SAM-2 model to perform pixel-level instance segmentation, accurately identify and distinguish key safety elements, and obtain a high-robustness two-dimensional color mask with instance ID.

6. The method of claim 5, wherein the method further comprises: The key safety elements in step S32 include construction personnel, personal protective equipment PPE, hanging ropes, hanging points, and protective fences.

7. The method of claim 6, wherein the method further comprises: The specific steps of step S4 include: S41, based on the internal parameter matrix obtained in step S2 and the external parameter registration matrix, mapping the two-dimensional color mask pixels in step S3 to a three-dimensional space through a reverse projection algorithm to establish a spatial correspondence between the mask pixels and the three-dimensional point cloud; mapping to a three-dimensional space through a reverse projection algorithm to establish a spatial correspondence between the mask pixels and the three-dimensional point cloud; S42, based on the three-dimensional point cloud data collected in step S1, the mask pixels obtained by the projection in step S41 are precisely aligned with the three-dimensional point cloud of the corresponding frame through a same frame timestamp matching mechanism, to obtain a specific object color point cloud instance set with semantic labels.

8. The method of claim 7, wherein the method further comprises: The specific steps of step S5 include: S51, based on the noise purification requirement of the color point cloud instance set, the specific object point cloud instance is preprocessed through an SOR algorithm to remove isolated noise points generated by measurement noise and incorrect projection, to obtain pure point cloud instance data; S52, based on the point cloud instance data in step S51, the three-dimensional geometric centroid of each key safety element is calculated through an inverse distance square weighted centroid fitting algorithm, and the calculation formula is: ； wherein , , , is a weight of a point in the point cloud, and , is a distance of a point in the point cloud to the sensor; , , is a three-dimensional coordinate of a point in the point cloud; is a three-dimensional geometric centroid of the th critical safety element; S53, robustly fitting the ground point set using RANSAC algorithm obtaining the ground plane equation wherein, , , , are the plane equation parameters; S54, based on the key object three-dimensional geometric centroid obtained in step S52 and the ground plane obtained in step S53, a three-dimensional semantic topological scene graph is constructed by establishing an object-geometry relationship mapping wherein the vertex set formula is: ； wherein, for a construction worker, for a safety belt, for a hanging rope, for a hanging point, for an adjacent side, for a guardrail; The edge set formula is: ； wherein represents a geometric relationship above; represents a geometric relationship attached to; represents a geometric relationship adjacent.

9. The method of claim 8, wherein the method further comprises: The specific steps of step S6 include: S61, based on the aerial work state determination requirements, through the point to the plane distance calculation formula, the construction personnel centroid to the ground plane vertical distance , to verify whether the construction personnel is working, the calculation formula is: ； and when a high work state is determined, is a high work start height threshold value; S62, based on the hanging point height compliance requirements, by comparing the construction personnel mass center and the hanging point mass center Z-axis coordinates to verify the hanging point height compliance, the determination formula is: ； wherein, is the Z-axis coordinate of the hook point centroid; is the Z-axis coordinate of the worker centroid; is the hook height compliance minimum difference threshold value; S63, based on the safety rope length compliance requirements, the nearest point search algorithm is used to calculate the construction personnel point set The shortest Euclidean distance of the adjacent edge point set And the safety rope length Comparison to verify the compliance of the rope length, the comparison formula is:​ ； wherein, is a rope length safety factor; is the shortest Euclidean distance; S64, determine the pose normal vector of the worker point cloud based on the job pose compliance requirements by principal component analysis (PCA) algorithm fit the safety rope point cloud by RANSAC algorithm get its direction vector calculate the included angle between the two and verify whether it meets the safety threshold ; S65, quantifying the demand based on the overall risk, quantifying the compliance results of steps S61-S64 through a multi-factor Boolean weighted model to obtain a real-time risk score The scoring formula is: ； wherein, , , is a Boolean indicator function, taking the value 1 when the condition in the brackets is satisfied, and 0 otherwise; , , is a preset risk weight; is in a high-altitude operation state; is that the construction personnel are too close to the adjacent side, is a safety distance threshold of the adjacent side; is that the personnel-hanging rope-hanging point core safety element ternary group topology verification is unqualified.

10. The method of claim 9, wherein the method further comprises: The real-time judgment system based on the above method includes: A high-precision coupled acquisition module for synchronously acquiring bimodal data, the high-precision coupled acquisition module including a fixedly coupled laser radar and an RGB camera; An edge computing processing unit for performing all computing tasks in real time on site adopts An UPA module for dynamically self-adapting correction of initial prompt words, an Open vocabulary detection module for generating target detection frame and adaptive semantic segmentation module for image segmentation are internally deployed in the edge computing processing unit. A three-dimensional high-precision mapping module connected with the edge computing processing unit to accurately project a two-dimensional mask to a three-dimensional point cloud by using a pre-calibrated intrinsic matrix and an extrinsic matrix; A real-time scene topology reconstruction module connected with the three-dimensional high-precision mapping module; An interpretable rule judgment module connected with the real-time scene topology reconstruction module.

Images