Construction safety early warning system and method based on multi-modal behavior reasoning

Abstract

The application discloses a construction safety early warning system and method based on multi-modal behavior reasoning, relates to the field of building construction safety monitoring, and comprises the following steps: acquiring construction personnel video, double-hook state, hoisting equipment operation and operation area structure data of a working area near an edge, determining operation state and hoisting movement state; identifying local action and its time sequence in double-hook hooking and combining state change to deduce hooking state evolution sequence; predicting future hoisting movement range and operation state, generating candidate disposal actions meeting human action accessibility conditions, deducing post-disposal state of the candidate disposal actions, and screening safe disposal actions supporting stability, at least one hook being effectively connected and human body and hoisting space being non-intersected; determining the time when the safe disposal action set is first changed from non-empty rotation to empty rotation as an action commitment point, and determining an early warning time point according to the action commitment point. Thus, prospective safety early warning for hooking behavior and hoisting risk evolution is realized.

Description

Technical Field

[0001] This application relates to the field of construction safety monitoring technology, and in particular to a construction safety early warning system and method based on multimodal behavioral reasoning. Background Technology

[0002] In high-rise building construction, steel structure installation, and curtain wall construction, workers frequently need to adjust their positions near edges, cross structural members, or move between different work locations, and adjust the connection status of safety protection equipment accordingly. Meanwhile, lifting equipment such as tower cranes may be transporting and installing components nearby, and the position and range of motion of the load will continuously change with equipment operation and load fluctuations. As a result, the work behavior of construction workers, their safety protection status, and the operating status of construction equipment may interact and change within a short period of time, making the safety risks at the construction site dynamic and complex.

[0003] Currently, some construction safety monitoring systems typically collect on-site video, personnel location information, or sensor information from safety protection equipment to identify whether personnel are wearing safety protective equipment, whether they have entered hazardous areas, whether safety protection equipment is in the prescribed state, and whether predefined abnormal behaviors have occurred. Based on the identification results, corresponding early warning information is output. This type of technology can help detect relatively obvious violations or abnormal states and has been used for safety management in construction scenarios such as working at heights, working near edges, and hoisting operations.

[0004] However, current monitoring results typically focus on the status of personnel, equipment, or the working environment at a given moment or within a short timeframe, making it difficult to fully reflect the changes and interactions of different states during continuous operations. Construction sites may also experience visual obstruction, changes in observation angles, asynchronous data from different sources, and delays or missing sensor information, further affecting the consistency of monitoring results. In situations where personnel actions and equipment status change rapidly, or when multiple risk factors overlap, monitoring systems are prone to delayed warnings, false alarms, or missed warnings, and their ability to provide safety warnings for complex and dynamic construction operations remains somewhat limited. Summary of the Invention

[0005] This application provides a construction safety early warning method, system, storage medium, computer program product, and electronic device based on multimodal behavioral reasoning, which aims to at least solve the problem of untimely and inaccurate safety risk early warning in complex dynamic construction operation scenarios in the prior art.

[0006] In a first aspect, embodiments of this application provide a construction safety early warning method based on multimodal behavioral reasoning. The method includes: acquiring multimodal construction data of target construction personnel within a target edge work area, and determining the work status and hoisting motion status of the target construction personnel based on the multimodal construction data; the multimodal construction data includes construction personnel video data, double hook status data, hoisting equipment operation data, and work area structure data; the work status includes human posture, support status, and effective protective connection status of each hook, wherein the effective protective connection status is used to characterize whether the hook is connected to an anchor point that meets the fall protection bearing conditions; identifying multiple local actions and their occurrence sequence during the double hook replacement process based on the multimodal construction data, determining the work status changes corresponding to each local action, and performing state deduction for the subsequent replacement process based on the occurrence sequence and the work status changes to generate a replacement state evolution sequence; and predicting the hoisting motion status in various future times within a preset future time domain. The movement range of the moment is determined, and the predicted operation state corresponding to each future moment is determined according to the evolution sequence of the changing state; for each predicted operation state, multiple candidate disposal actions that meet the preset human movement reachability conditions are generated in combination with the operation area structure data; the corresponding candidate disposal actions are deduced with each predicted operation state as the starting state to obtain the post-disposal operation state of each candidate disposal action; the candidate disposal actions in the post-disposal operation state that meet the preset stability conditions of the support state, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-disposal operation state does not intersect with the hoisting movement range of the corresponding future moment are formed into a set of safe disposal actions corresponding to each future moment; the future moment when the set of safe disposal actions first changes from a non-empty state to an empty state is determined as the action commitment point, the warning triggering time earlier than the action commitment point is determined according to the action commitment point, and construction safety warning information is generated at the warning triggering time.

[0007] Secondly, embodiments of this application provide a construction safety early warning system based on multimodal behavioral reasoning. The system includes: a multimodal state perception unit, used to acquire multimodal construction data of target construction personnel within a target edge work area, and determine the work status and hoisting motion status of the target construction personnel based on the multimodal construction data; the multimodal construction data includes construction personnel video data, double hook status data, hoisting equipment operation data, and work area structure data; the work status includes human posture, support status, and effective protective connection status of each hook, the effective protective connection status being used to characterize whether the hook is connected to an anchor point that meets the fall protection bearing conditions; a hook replacement state deduction unit, used to identify multiple local actions and their occurrence sequence during the double hook replacement process based on the multimodal construction data, determine the work status changes corresponding to each local action, and perform state deduction for the subsequent hook replacement process based on the occurrence sequence and the work status changes, generating a hook replacement state evolution sequence; and a candidate disposal generation unit, used to predict the hoisting in a preset future time based on the hoisting motion status. The movement range at each future moment within the domain is determined, and the predicted operation state corresponding to each future moment is determined according to the evolution sequence of the swapping state. For each predicted operation state, multiple candidate disposal actions that meet the preset human movement reachability conditions are generated in combination with the operation area structure data. A safety disposal screening unit is used to deduce the corresponding candidate disposal actions with each predicted operation state as the starting state to obtain the post-disposal operation state of each candidate disposal action. Candidate disposal actions in the post-disposal operation state that meet the preset stability conditions, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-disposal operation state does not intersect with the hoisting movement range of the corresponding future moment are formed into a set of safety disposal actions corresponding to each future moment. A commitment point early warning unit is used to determine the future moment when the set of safety disposal actions first changes from a non-empty state to an empty state as the action commitment point, determine the early warning trigger time earlier than the action commitment point according to the action commitment point, and generate construction safety early warning information at the early warning trigger time.

[0008] Thirdly, an electronic device is provided, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the construction safety early warning method based on multimodal behavioral reasoning according to any embodiment of the present application.

[0009] Fourthly, embodiments of this application provide a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the construction safety early warning method based on multimodal behavioral reasoning according to any embodiment of this application.

[0010] Fifthly, embodiments of this application provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the steps of the construction safety early warning method based on multimodal behavioral reasoning according to any embodiment of this application.

[0011] The construction safety early warning method and system based on multimodal behavioral reasoning provided in this application can produce at least the following technical effects: By transforming sequential local actions during the double-hook swapping process into corresponding changes in operational status, and extrapolating subsequent swapping processes according to the sequence of actions, a continuously changing sequence of swapping status evolution can be obtained as the operation progresses. Based on this, the predicted operational status at each future moment is correlated with the range of hoisting motion. This ensures that human posture, protective connection status, support status, and hoisting motion are no longer independent monitoring results, but rather participate in risk analysis according to a unified future timescale. This provides a more complete reflection of the safety status evolution process under the combined effects of continuous personnel actions and dynamic hoisting operation.

[0012] Based on the prediction of future operational states, the risk is further evaluated from the perspective of whether personnel still have safe and feasible actions. Specifically, by combining the on-site structural conditions to determine the candidate actions that are actually accessible to the human body, and by extrapolating the operational state after performing different actions, the candidate actions are considered safe only if they can maintain stable support, maintain at least one effective protective connection, and avoid entering the corresponding hoisting range. Therefore, the set of safe actions formed at each future moment can characterize the actual remaining safe action space of construction personnel in the corresponding state; when this set becomes empty for the first time, it indicates that according to the current state evolution trend, personnel have reached the critical point where their safe action capability is about to be lost. Based on this, an early warning can be triggered, allowing necessary time margin for actual risk avoidance and disposal.

[0013] This technical solution expands the scope of construction safety early warning from the future hazardous state itself to the changes in the safe action space of personnel during the evolution of the hazard, and characterizes the time boundary of the disappearance of this safe action space through action commitment points. Therefore, early warnings can be issued when the hazardous outcome has not yet materialized but feasible safety measures are about to be exhausted. This makes the timing of the warning constrained by both the future risk evolution and the actual handling capabilities, thus forming a predictive safety early warning mechanism that more directly corresponds to the actual risk avoidance opportunities of construction personnel. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 A flowchart illustrating an example of a construction safety early warning method based on multimodal behavioral reasoning according to an embodiment of this application is shown; Figure 2 A schematic diagram illustrating an example scenario of safety warning during edge replacement construction according to an embodiment of this application is shown. Figure 3 This paper shows an example operation flowchart of constructing a set of safety handling actions in the method according to an embodiment of the present application; Figure 4 A flowchart illustrating an example of determining the warning triggering time driven by the action commitment point in the method according to an embodiment of this application is shown. Figure 5 A schematic diagram illustrating the operational mechanism of an example construction safety early warning method based on multimodal behavioral reasoning according to an embodiment of this application is shown. Figure 6 A comparative experimental simulation diagram showing the distribution of early warning lead time for different early warning methods under the condition of action commitment point alignment is presented. Figure 7 A comparative simulation diagram of the false negative rate response of different methods under the combined disturbance of visual occlusion and load swing angle is shown. Figure 8 A comparative simulation diagram illustrating the performance of different methods in multidimensional engineering deployment is shown. Figure 9 A structural block diagram of an example of a construction safety early warning system based on multimodal behavioral reasoning according to an embodiment of this application is shown. Detailed Implementation

[0016] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0017] It should be noted that some current systems typically use target detection technology to identify construction workers and their safety belts, hooks, and other protective equipment to determine whether workers are wearing appropriate protective gear or whether hooks are in a closed state. For example, hook detection systems based on YOLO series target detection algorithms can extract the appearance features of hooks when the target is small, easily obscured, or in complex work backgrounds. These systems mainly form detection results based on the target category and local morphology in the image. However, it is often difficult to make a complete judgment on the connection and usage status of protective equipment during specific work processes based solely on appearance detection results.

[0018] To identify continuous construction activities, some systems currently employ CNN-LSTM temporal analysis techniques. These techniques input image features extracted by a CNN (Convolutional Neural Network) into an LSTM (Long Short-Term Memory) network to determine the corresponding construction activity category for video segments. Other systems use ST-GCN (Spatial-Temporal Graph Convolutional Network) to process human skeleton sequences and combine this with target detection results to identify interactions between personnel and construction machinery or materials. While these systems can utilize temporal information or changes in human joints within continuous images, their output typically represents pre-defined action or interaction categories. For similar-looking, short-duration, or incomplete work states, the recognition results are easily affected by the observation duration and the classification method.

[0019] To improve the accuracy of target occlusion and spatial location judgment biases under single-view conditions, construction violation identification systems based on multi-view images and 3D scene understanding have been developed. For example, the Safe-Construct system utilizes multi-view spatial information and scene-level personnel-object relationships to identify the 3D participation relationships between construction workers, construction equipment, and surrounding components. Such systems can provide the location, category, and spatial relationships of personnel and objects; however, their results typically focus on the spatial state at a specific moment or within a given scene. For changes in the state of each object over time during continuous operations, further analysis using other temporal information is still necessary.

[0020] Regarding proactive early warning, some systems currently track the location of construction workers and predict their subsequent movement trajectories, comparing the predicted locations with pre-defined danger zones. Other systems determine the affected area as the load's position changes based on its detection location and projection onto the work surface. While these systems can reflect the movement trends of personnel or loads to some extent, they typically represent workers and loads as location points, movement trajectories, or spatial envelopes, respectively, and make judgments based on the geometric relationship between the corresponding objects and the danger zone. In construction scenarios where personnel behavior and hoisting operations change synchronously, this type of geometric relationship still has limitations in representing continuous operation states.

[0021] It should be understood that the above description of the relevant technologies is intended only to help the public better understand the inventive spirit and motivation of this application, and is not intended to limit this application. Furthermore, the technical solutions described in the above-mentioned relevant technologies are not prior art, and may also be undisclosed technical solutions, such as those under research or in the laboratory stage.

[0022] The technical solutions in this application, including the collection, storage, use, processing, transmission, provision, and disclosure of users' personal information, comply with relevant laws and regulations and do not violate public order and good morals.

[0023] Figure 1 A flowchart illustrating an example of a construction safety early warning method based on multimodal behavioral reasoning according to an embodiment of this application is shown.

[0024] Regarding the execution subject of the method in this application embodiment, it can be applied to high-rise building edge installation, steel structure connection, curtain wall component placement, and other work scenarios that require construction personnel to adjust the connection status of safety belt hooks between adjacent anchor points. The target edge work area refers to the local construction area where the target construction personnel are currently located and may move, change hooks, or be swept by the hoist within a preset future time domain. The method takes the target construction personnel as the processing object, places the construction personnel's actions, safety protection connections, and hoisting movements under the same construction event time axis for simulation, and determines the early warning timing based on whether future executable safety disposal actions still exist.

[0025] The method described in this application can be executed by electronic equipment in a construction safety early warning system. This system may include electronic equipment and video acquisition equipment, a dual-hook status acquisition component, a hoisting equipment control system, and an early warning output device communicatively connected to the electronic equipment. The electronic equipment may be a field controller, an edge computing device, a server, or a computing system composed of a field controller and a server. The term "system" as used below refers to a construction safety early warning system in which the electronic equipment performs corresponding data acquisition, status deduction, action filtering, and early warning generation processing.

[0026] like Figure 1 As shown, in step S110, multimodal construction data of the target construction personnel within the target edge work area is acquired, and the working status and hoisting movement status of the target construction personnel are determined based on the multimodal construction data. The multimodal construction data includes construction personnel video data, double hook status data, hoisting equipment operation data, and work area structure data; the working status includes human posture, support status, and effective protective connection status of each hook, wherein the effective protective connection status is used to characterize whether the hook is connected to an anchor point that meets the fall protection bearing conditions.

[0027] The video data of the construction workers can be acquired by at least two cameras with overlapping fields of view, positioned around the target's perimeter work area. These cameras may include fixed wide-angle cameras or cameras mounted near the tower or hook. Based on images acquired by different cameras, the three-dimensional human joints, human envelope, hand positions, foot contact positions, and center of gravity of the target construction worker are determined, and the hook position, rope extension path, anchor point position, and spatial position of the hoisting load are identified. In the event of partial obstruction, a short-term state of the corresponding object can be maintained based on images from the unobstructed viewpoint and the continuous human state before and after the obstruction.

[0028] The dual-hook status data may include the door lock opening / closing signals corresponding to each hook, anchor point identification information, and rope tension change information. Anchor point identification information is used to match the currently connected construction component with pre-registered qualified anchor point information. Tension change information reflects whether a test pull occurred after the hook was closed and whether the rope underwent a force response. The hoisting equipment operation data may include the hoisting equipment's slewing status, luffing status, lifting status, operating speed, control commands, braking status, and communication delay. The work area structural data may be derived from a building information model or a site scan model, recording the location and attributes of edges, floors, steel beams, holes, obstacles, support surfaces, and anchor points.

[0029] Multimodal construction data from different sources can have different sampling frequencies. The system associates image frames, hook status events, hoisting equipment status, and work area structural data with a unified construction event clock based on the timestamps carried by various data types. For data collected before the current processing time, the time difference between the data collection time and the current processing time is retained, enabling subsequent state predictions to identify the update level of various data types. When the data time difference exceeds the corresponding allowable range, the participation of the corresponding observation in the state determination process can be reduced, or the spatial range determined by the corresponding state can be expanded.

[0030] Human posture is used to characterize the target construction worker's joint position, torso orientation, and body space at the current moment. Support status characterizes the contact position of the target construction worker's feet, the position of the center of gravity relative to the support area, and whether the body can maintain stability. The support area can be determined based on the contact relationship between the feet and the floor slab, steel beam, or other standing structures. When the center of gravity is within the support area, it indicates that the current posture has a corresponding support foundation; when the center of gravity moves towards the boundary of the support area or the supporting feet shift, the system updates the support status accordingly.

[0031] For any hook, the system determines its effective protective connection status based on the visual connection between the hook and the anchor point, the door lock status, the anchor point's identity, the anchor point's load-bearing capacity, the change in tether tension, and the tether geometry. The hook appearing closed in the image is not directly used as evidence of an effective protective connection. Only when the anchor point connected to the hook meets the fall protection load-bearing conditions, and the hook's closed state, tension change, and connection geometry collectively support the connection between the hook and the anchor point, is the corresponding hook identified as having an effective protective connection. The resulting human posture, support status, effective protective connection status of each hook, and load movement status together constitute the state input for the current processing moment.

[0032] In step S120, multiple local actions and their occurrence sequence in the double hook replacement process are identified based on the multimodal construction data, the operation state changes corresponding to each local action are determined, and the subsequent replacement process is deduced based on the occurrence sequence and the operation state changes to generate a replacement state evolution sequence.

[0033] The partial actions during the double-hook replacement process can include reaching towards the new anchor point, closing the new hook, testing and confirming the pull, releasing the original hook, shifting the supporting foot, and straddling the torso. These partial actions are not independent, complete behavioral categories, but rather localized operations that can alter the body's posture, support state, or effective protective connection status. The system identifies reaching towards the new anchor point based on changes in the distance between the hand and the new anchor point, identifies the closing of the new hook based on changes in the door lock status, identifies testing and confirming the pull based on changes in tension after the hook closes, identifies the release of the original hook based on the door lock status and tension changes, and identifies the shifting of the supporting foot and straddling the torso based on changes in the foot contact position, the body's center of gravity, and the torso position.

[0034] For partially completed actions, the system determines the time interval of occurrence and the operational status before and after the action. For partially completed actions, the system predicts the possible time interval of the corresponding action based on the current human movement trend, the spatial relationship between the hands and feet and the anchor point or support surface, changes in the hook status, and the sequence of partially completed actions. After a partially completed action, the system updates the operational status according to the actual effect of that action. For example, after the original hook is released, the system updates the effective protective connection status of the original hook; after the supporting foot is transferred, the system updates the contact position of both feet, the support area, and the human center of gravity; after the torso crosses over, the system updates the human posture and the human spatial range.

[0035] The remounting state evolution sequence consists of multiple predicted operational states arranged chronologically, with each predicted operational state corresponding to a future moment or time interval within a preset future time domain. The remounting state evolution sequence records at least the human posture, support state, and effective protective connection status of each hook at the corresponding future moment. For state transitions where the new hook has not yet formed an effective protective connection while the original hook has been released, where no hook is in an effective protective connection state when the support leg is transferred, or other state transitions that do not conform to the preset remounting sequence, the system retains the impact of this state transition on subsequent human posture, support state, and hook state, rather than correcting it to a normal remounting state.

[0036] In step S130, the movement range of the hoist in each future time in the preset future time domain is predicted according to the hoist movement state, and the predicted operation state corresponding to each future time is determined according to the changeover state evolution sequence; for each predicted operation state, multiple candidate disposal actions that meet the preset human movement reachability conditions are generated in combination with the operation area structure data.

[0037] The system extrapolates the predicted position and attitude of the hoist at various future moments within a preset future time domain based on the hoist's current position, speed, slewing state, luffing state, lifting state, sling length, swing angle, and sway trend. This preset future time domain covers the time required to complete the current hoisting operation, personnel actions, or changes in the hoisting equipment's operating status. During prediction, the actual geometric dimensions of the hoist are applied to the predicted position and attitude at each future moment, and the space occupied by the hoist is expanded based on hoist position errors, sway changes, data time differences, and control execution delays to obtain the hoist's range of motion at each future moment. Therefore, this range of motion is not a single trajectory of the hoist's center point, but rather a spatial range encompassing the hoist's geometric dimensions and prediction uncertainties.

[0038] The system correlates each predicted operational state in the loading and unloading state evolution sequence with the movement range of the load at the same future time according to the time correspondence. Each predicted operational state includes the target construction personnel's body posture, support status, and the effective protective connection status of each hook at the corresponding future time. Through this correlation, the system can compare the target construction personnel's body space, protective status, and the space that the load may occupy at the same future time, avoiding direct comparison of personnel status and load status at different times.

[0039] Candidate actions can include maintaining the current posture, stopping the crossing, returning to the previous support surface, tangentially shifting laterally along the edge, re-attaching the safety harness hook, and waiting for the load to pass. For each predicted work state, the system performs an initial screening of candidate actions based on the target worker's joint range of motion, body movement speed, current support position, accessibility between the hand and candidate anchor points, rope length, and edges and obstacles in the work area. The system identifies actions that can be performed within a standing area, do not cross fixed obstacles, and conform to the human joint range of motion as candidate actions that meet the preset human movement accessibility conditions.

[0040] In step S140, candidate disposal actions are derived from each of the predicted operation states as the starting state to obtain the post-disposal operation state of each candidate disposal action; the candidate disposal actions in the post-disposal operation state that satisfy the preset stability conditions, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-disposal operation state does not intersect with the hoisting movement range of the corresponding future time are formed into a set of safety disposal actions corresponding to each future time.

[0041] For a candidate action, the system starts with the predicted operational state at the corresponding future moment and performs a forward simulation based on the target worker's body movement direction, hand and foot position changes, support surface transformation, and hook status changes when performing the action. During the simulation, the system updates the human joint points, center of gravity, foot contact positions, support area, hook connection status, rope path, and human spatial range. After the simulation is complete, a post-action operational state is generated, which is associated one-to-one with the candidate action and its corresponding future moment.

[0042] The preset stability conditions can be determined based on the position of the person's center of gravity relative to the support area after the procedure. For example, if the projection of the person's center of gravity after the procedure is within the support area, or if a preset stability margin is maintained between the person's center of gravity and the boundary of the support area after the procedure, the support state after the procedure is determined to meet the preset stability conditions. For candidate procedures that require the transfer of support legs, multiple intermediate working states can also be determined according to the sequence of action execution, and it can be determined whether the person's center of gravity and the support area in each intermediate working state continuously meet the corresponding stability conditions, avoiding the need to judge the support stability of the entire procedure solely based on the static standing position after the procedure is completed.

[0043] The system further checks whether at least one hook is in a valid protective connection state in the post-treatment operation status, and determines the post-treatment human body space range based on the human body joint points and corresponding human body dimensions at the time the candidate treatment action is completed. The system performs a spatial intersection judgment between the post-treatment human body space range and the hoisting movement range corresponding to the future time of the candidate treatment action. If the post-treatment human body space range and the hoisting movement range do not intersect, it indicates that the human body did not enter the hoisting space predicted for the corresponding future time after the candidate treatment action was completed.

[0044] For candidate actions involving continuous body movement, when continuous spatial verification of the action execution process is required, the system can divide the candidate actions into multiple discrete moments according to the action execution sequence, determine the human body spatial range at each discrete moment, and perform spatial intersection judgment between each human body spatial range and the hoisting motion range corresponding to the time. This allows the human body motion state and the hoisting motion state to be compared under the same time reference.

[0045] Candidate actions that simultaneously meet the following conditions—accessibility to human movement, support meeting preset stability requirements, at least one hook in an effective protective connection state, and the human space range not intersecting with the hoisting movement range—are added to the corresponding future time-initiated set of safe action actions. Candidate actions that fail to meet any of these conditions are excluded from the corresponding safe action action set, with the reason for exclusion recorded as inaccessibility, unstable support, failed protective connection, or intersection with the hoisting movement range. Thus, each future time-initiated ...

[0046] In step S150, the future moment when the set of safety action actions first changes from a non-empty state to an empty state is determined as the action commitment point. Based on the action commitment point, an early warning triggering time earlier than the action commitment point is determined, and construction safety early warning information is generated at the early warning triggering time.

[0047] The system checks the set of safety response actions corresponding to each future moment within a preset future time domain in chronological order. When the set of safety response actions is non-empty, it indicates that at least one candidate action satisfies the conditions of action reachability, support stability, protective connection, and spatial collision avoidance. When the set of safety response actions is empty, it indicates that no candidate action can simultaneously satisfy the above conditions under the corresponding predicted operational state. The system defines the future moment when the set of safety response actions first changes from non-empty to empty as the action commitment point. The action commitment point represents the time boundary at which all candidate safety response methods first disappear after continuing the current swapping process.

[0048] The warning trigger time is set before the action commitment point. The system can determine the warning protection time to be retained before the action commitment point based on the time required for the target construction personnel to complete the safety action, the time required for the personnel to receive and identify the warning information, the system data processing time, and the warning information transmission time. The system then determines the warning trigger time based on the action commitment point and the warning protection time. The warning trigger time is earlier than the action commitment point. When the time calculated backward from the warning protection time is earlier than the current time, the system immediately triggers the warning at the current time; since the action commitment point is a future time after the current time, the current time is still earlier than the action commitment point.

[0049] Construction safety early warning information is generated based on candidate actions from the set of safety response actions at the time the warning is triggered. This information may include at least one of the following: stopping the current hook-up action, retreating to the next support position, moving laterally along the edge, re-hooking the hook, waiting for the load to pass, and requesting the hoisting equipment to adjust its operating status. When multiple safety response actions exist, a target action can be determined according to preset action selection rules, based on the estimated completion time of each candidate action, the support stability margin, and the spatial separation distance relative to the load's movement range. The warning content is then generated based on this target action. Construction safety early warning information can be output via a wrist-worn terminal worn by the target construction personnel, on-site audio-visual devices, or the hoisting equipment control interface.

[0050] In one example, after the target construction worker attaches the new hook to the new anchor point, no tension change corresponding to the test pull has yet occurred, and the original hook remains in an effective protective connection state. Simultaneously, the load moves towards the target's adjacent work area. The system determines the effective connection probability and immediate load-bearing probability of the new hook based on the door lock status, anchor point identity and load-bearing capacity, tension change, and tether geometry. When the test pull micro-tension waveform is missing, the probability of the new hook being in an effective standby state is limited, and its effective protective connection state is determined based on the comparison between the effective connection probability and a preset connection probability threshold. The system predicts that the target construction worker may subsequently release the original hook and move the support leg based on the current action trend. If the state projection indicates that the effective connection probability of the new hook after the original hook is released still does not reach the preset connection probability threshold, and the retreat action is restricted by the tether path, the human space corresponding to the lateral movement intersects with the load's movement range, and the re-attaching action cannot be completed in the corresponding future time, then the set of safety handling actions for the corresponding future time is empty. The system determines the future moment when the set first changes from a non-empty state to an empty state as the action commitment point, and outputs construction safety warning information before the action commitment point, indicating whether to maintain the original hook connection or return to the current support position.

[0051] In this embodiment, construction safety early warning information is determined by the future work status, the lifting range, and the set of safety response actions. Before the target construction worker completes a dangerous action or enters the lifting range, the system outputs early warning information corresponding to executable candidate actions based on whether the set of safety response actions is about to disappear.

[0052] Figure 2 This diagram illustrates an example scenario of safety warning during edge replacement construction according to an embodiment of the present application. Figure 2 The safety margin values ​​and the current speed ratio of the hoisting equipment shown are the interface display results under this application example. The specific values ​​are determined based on the current multimodal construction data and the corresponding state deduction results.

[0053] like Figure 2 As shown in the example, in one application scenario, a high-rise building is undergoing the hoisting and placement of curtain wall unit panels. The target construction workers are located on the steel beams at the edge of the floor and need to complete the double hook replacement between adjacent anchor points and move to the curtain wall installation position. The hoisting equipment pulls the curtain wall unit panels along the building facade to move them to the target edge work area. During the movement and swinging process, the curtain wall unit panels may sweep across the space that the target construction workers will subsequently need to pass through.

[0054] The system synchronously acquires multi-view video data of construction workers, double hook status data, hoisting equipment operation data, and structural data of the edge-operating area. Multi-view video data of construction workers is used to determine the three-dimensional human posture, center of gravity, hand and foot positions, and spatial range of the human body; double hook status data is used to determine whether the hooks form an effective protective connection with anchor points that meet fall-prevention load-bearing conditions; hoisting equipment operation data is used to determine the position, direction of movement, speed of movement, and swaying state of the curtain wall unit panels; and structural data of the edge-operating area is used to determine the spatial positions of steel beams, edges, anchor points, support areas, and obstacles.

[0055] At the current moment, the system has determined that the original hook and the first anchor point have formed an effective protective connection. Although the new hook has been attached to the second anchor point, no force change corresponding to the test pull confirmation has yet occurred. The target construction worker is currently supported by both feet on the steel beam and has a tendency to move towards the curtain wall installation position by shifting the support feet and crossing over the body. At the same time, the hoist is approaching the area where the target construction worker may subsequently pass.

[0056] Based on the identified actions of reaching towards the new anchor point, closing the new hook, and the current movement trend of the human body, the system predicts subsequent possible actions such as trial pull confirmation, release of the original hook, transfer of the supporting foot, and torso crossing. The system forms a sequence of hook-changing state evolution according to the timing of each local action and the resulting changes in human posture, support status, and effective protective connection status, and determines the predicted operational state for each future moment.

[0057] The system predicts the lifting range that the curtain wall unit panels may occupy at the same future time based on their current position, direction of movement, speed of movement, geometric dimensions, and swaying state. The predicted operational status at each future time is associated with the corresponding lifting range, so that the human body status, protective connection status, and lifting movement status participate in the judgment of the handling action under the same time reference.

[0058] For each predicted operational state, the system combines the reach of human movement, the steel beam support area, anchor point location, rope path, and obstacle location to generate candidate actions such as stopping the release of the original hook, returning to the current support position, completing a test pull confirmation of the new hook, moving laterally along the steel beam, and waiting for the load to pass. The system then performs counterfactual analysis on each candidate action, starting from each predicted operational state, to obtain the human posture, support status, hook connection status, and human spatial range after each action is executed.

[0059] The system combines candidate actions that meet preset stability conditions, have at least one hook in an effective protective connection, and whose human space and the corresponding time-bound lifting movement range do not intersect, into a set of safe actions for the future timeframe. In the future phase, before the original hook is released, stopping the hook replacement and returning to the current support position can still maintain stable support and an effective protective connection, and the human space and the lifting movement range are separated; therefore, the corresponding set of safe actions is not empty.

[0060] As the predicted hook-up process continues, if the target construction personnel release the original hook and move the support leg, but the new hook has not yet formed an effective protective connection, the retreat action will be limited by body position and rope path. The human space corresponding to the lateral movement action will enter the lifting load's movement range, and the re-hooking action cannot be completed in the corresponding future moment. At this time, none of the candidate actions can simultaneously meet the conditions of support stability, effective protective connection, and lifting load avoidance, and the set of safe actions for the corresponding future moment becomes empty.

[0061] The system determines the future moment when the set of safety response actions first changes from a non-empty state to an empty state as the action commitment point, according to the chronological order of future moments. It also determines the warning trigger moment earlier than the action commitment point based on the personnel's reaction time after receiving the warning, the execution time of the response action, the system's calculation time, and the information transmission time.

[0062] At the moment the warning is triggered, the system outputs a prompt to the terminal worn by the target construction personnel, instructing them to maintain the original hook connection, stop the support leg transfer, and return to the current support position. When adjustments to the hoisting motion state are required, the system can also output a deceleration prompt to the hoisting equipment control terminal and output corresponding construction safety warning information through on-site alarm equipment. After the target construction personnel or hoisting equipment complete the corresponding actions, the system updates the hook replacement state evolution sequence, hoisting motion range, safety action set, action commitment point, and warning trigger time based on newly acquired multimodal construction data.

[0063] The above application example uses the existence of future safety procedures as the basis for early warning judgment, enabling the system to output construction safety early warning information corresponding to candidate procedures that can still be performed at that time before the target construction personnel complete the dangerous reloading action and the human body enters the range of the hoisting movement.

[0064] Regarding the implementation details of determining the working status of target construction personnel based on multimodal construction data in the method of this application embodiment, in some examples of this application embodiment, for each hook, the visual hooking relationship, door lock status, anchor point identity and load-bearing capacity, test pull micro-tension waveform and rope geometry are extracted, and the candidate connection status of the hook is divided into unconnected state, pseudo-joined state, effective slack state and effective standby state; wherein, the unconnected state indicates that the hook is separated from the anchor point, the pseudo-joined state indicates that the hook is in a hooked or closed state but has not formed a connection that meets the fall protection load conditions, the effective slack state indicates that the hook is connected to the anchor point that meets the fall protection load conditions and the rope is slack, and the effective standby state indicates that the hook forms a force path that can immediately bear the fall protection load.

[0065] The visual connection relationship is determined based on the hook position, anchor point position, hook opening direction, and rope extension direction; the door lock status is output by the hook door lock sensor; the anchor point identity and load-bearing capacity are obtained by matching the anchor point identity reading result with preset qualified anchor point information; the test pull micro-tension waveform is formed by the rope tension data within a preset observation period after the hook is closed; the rope geometry is determined based on the safety belt connection point, rope path, hook position, and anchor position. All types of data are linked to the same processing time according to the construction event clock, so that observations corresponding to the same hook and the same processing time jointly participate in the connection status judgment.

[0066] For example, respectively using , , and The terms "unconnected state," "pseudo-connected state," "effectively relaxed state," and "effectively ready-to-load state" are used to indicate these states. For each candidate connection state, corresponding observation features are pre-set. These pre-set observation features can be calibrated based on confirmed hook connection samples: The unconnected state corresponds to the hook being separated from the anchor point and the tether not forming a force path pointing towards the anchor point; the pseudo-connected state corresponds to the hook appearing hooked or closed, but the anchor point conditions, the test pull micro-tension waveform, or the tether geometry do not collectively support an effective connection; the effective relaxed state corresponds to the hook being connected to a qualified anchor point and the tether not yet forming a tension state capable of immediately bearing load; the effective ready-to-load state corresponds to the door lock being closed, the anchor point being qualified, the test pull micro-tension waveform matching the pre-set waveform characteristics, and the tether forming a fall-prevention force path.

[0067] For any candidate connectivity state, the normalized deviations between various observations and their corresponding preset observation features are weighted and fused to obtain the state energy used to characterize the degree of inconsistency between the observation results and the candidate connectivity state: Equation (1) in, For the first The hook at the current moment In candidate connection state The state energy at that time , ; For the observation category index, categories 1 to 4 correspond to visual attachment relationship, door lock status, trial pull micro-tension waveform, and rope geometry, respectively; For the first Class observation relative to candidate connection states The normalized state deviation, and ; For the first The non-negative weights corresponding to the observations, and Each weight can be preset based on the sensor data calibration results or the reliability of historical observations. The state energy is obtained by weighting the normalized state deviations, without attached units of measurement, and increases with the weighting results of various normalized state deviations.

[0068] For visual hook-up relationships and door lock status, the normalized state deviation can be determined based on whether the observed results are consistent with the preset state of the corresponding candidate connection status. For the test pull micro-tension waveform, the normalized state deviation can be determined based on the differences between the measured waveform and the preset test pull waveform in terms of duration, peak value change, and fall trend. For the tether geometry, the normalized state deviation can be determined based on the difference between the observed tether length and the path length from the safety belt connection point to the anchor point, as well as the tether extension direction. All deviations are normalized using the allowable variation range of the corresponding observed data in the calibration sample. When hook obstruction occurs or any type of sensor data exceeds the allowable update time, the corresponding observation category uses the pre-calibrated normalized state deviation and non-negative weights for data loss situations, with the sum of the weights for each observation category remaining constant at one.

[0069] The unconnected state and the pseudo-connected state are determined as feasible connection states, and the effective relaxation state and the effective standby state are added to the feasible connection states only when the anchor point identity belongs to the preset qualified anchor point set and the bearing capacity meets the fall protection bearing condition.

[0070] The preset set of qualified anchor points is established based on the anchor point identifiers, anchor point locations, and anchor point categories in the work area structure data. The anchor point bearing capacity is obtained from the anchor point attributes in the work area structure data or from on-site registration data associated with the anchor point identity. Based on the anchor point identity and bearing capacity, the first anchor point is determined according to the following formula. The hook at the current moment The set of feasible connection states:

[0071] Equation (2) in, For the first A set of feasible connection states for each hook; For the first The identity of the anchor point corresponding to each hook; For a pre-defined set of qualified anchor points; This refers to the load-bearing capacity of the corresponding anchor point; This refers to the lower limit of the bearing capacity corresponding to the fall prevention bearing conditions. When the anchor point's identity does not match the preset qualified anchor point set, or when the anchor point's bearing capacity is lower than the lower limit of the bearing capacity, the feasible connection state set consists of an unconnected state and a pseudo-joined state. Door lock closure or visual hooking results do not change the constraints of this state set.

[0072] The various observations are compared with the preset observation features corresponding to each of the feasible connection states to obtain the observation bias, and the comprehensive state bias is determined based on the observation bias. The state probability of each of the feasible connection states is determined based on the comprehensive state bias, so that the state probability corresponding to the feasible connection state with the smaller the comprehensive state bias is, the sum of the state probabilities of each of the feasible connection states is one, and the state probability of the candidate connection states other than the feasible connection states is zero.

[0073] The state energy serves as a numerical representation of the comprehensive state bias. For candidate connection states belonging to the set of feasible connection states, the state probability is determined through exponential transformation and normalization.

[0074] Equation (3) in, For the first The hook is in the candidate connection state. The state probability; To traverse the set of feasible connection states State index; It is an exponential function. The state probability corresponding to candidate connection states outside the feasible connection state set is zero. The feasible connection state set includes at least unconnected states and pseudo-joined states to form the non-empty state set required for state probability normalization; the effective relaxation state and the effective reserve state are pre-constrained by the anchor point identity and bearing capacity, and other observation results do not change the pre-constraints.

[0075] Furthermore, the sum of the probabilities of the effective relaxation state and the effective standby state is determined as the effective connection probability, the probability of the effective standby state is determined as the probability of immediate carrying, and when the effective connection probability reaches a preset connection probability threshold, the corresponding hook is determined to be in the effective protection connection state.

[0076] The probability of effective connection and the probability of immediate carrying are determined by the following formula: Equation (4) in, For the first The probability of a valid connection to a hook. For the first The probability that a hook can immediately bear weight; and These represent the state probabilities corresponding to the effective slack state and the effective standby state, respectively. The preset connection probability threshold can be calibrated based on confirmed effective connection samples and ineffective connection samples. The effective connection probability is used to determine whether the hook is in an effective protective connection state, and the immediate load-bearing probability is used to distinguish between the effective standby state where a fall-prevention force path has been formed and the effective slack state where the rope is in a slack state, and participates in the state deduction of the subsequent double hook replacement process.

[0077] In one example, the new hook is already closed at the anchor point that meets the fall protection load-bearing conditions, but a micro-tension waveform consistent with the preset test pull waveform characteristics has not yet been detected. The hook can obtain the state probability corresponding to the effective relaxation state, while the state probability corresponding to the effective standby state is limited by the deviation of the test pull micro-tension waveform. When the effective connection probability reaches the preset connection probability threshold, the hook can be determined to be in an effective protective connection state; when the immediate load-bearing probability has not yet reached the preset load-bearing probability threshold used in the subsequent hook replacement process, the hook will not participate in the original hook release or support foot transfer as a hook that has completed load-bearing confirmation. Thus, the hook connection establishment degree and immediate load-bearing capacity are respectively retained in the corresponding probabilities, avoiding directly equating the door lock closed state with a force path that can immediately bear the fall protection load.

[0078] Regarding the implementation details of generating the hook-changing state evolution sequence in the method of this application embodiment, in some examples of this application embodiment, based on the spatial relationship between the human hand, foot, and center of gravity and the hook, anchor point, and support area, and combined with the door lock state and the test pull micro-tension waveform, the local actions of reaching out to move to the new anchor point, closing the new hook, test pull confirmation, releasing the original hook, transferring the support foot, and the torso crossing are identified, the occurrence time interval of the local actions that have occurred is determined, and the occurrence time interval and completion time of the incomplete local actions are predicted.

[0079] Each local action is identified using recognition conditions corresponding to its state change. When the hand continuously moves towards the new anchor point and the distance between the hand and the new anchor point decreases, the corresponding human action is identified as reaching towards the new anchor point; when the door lock state changes from open to closed, the corresponding hook action is identified as the new hook closing; when a micro-tension waveform that conforms to the preset test pull change characteristics appears after the new hook closes, the corresponding operation is identified as test pull confirmation; when the door lock state of the original hook changes from closed to open and the tension corresponding to the original hook decreases, the corresponding operation is identified as the original hook releasing. The support foot transfer is identified based on the foot contact position and the change in the position of the human body's center of gravity relative to the support area, and the torso crossing is identified based on the spatial relationship between the human torso and the target edge or component boundary. The distance change, test pull change characteristics, and spatial relationship can be determined based on the confirmed local action samples and the structural data of the working area.

[0080] In practical implementation, the changes in human posture, door lock status, and test-tension waveforms can be time-aligned according to the construction event clock. The starting point of the action occurrence time interval is the moment when each observation first meets the corresponding local action recognition condition, and the ending point is the moment when the corresponding state change reaches the action completion condition. For local actions that have not yet been completed, their predicted occurrence time interval and completion time are determined based on the current local action state and the sequential relationship between the identified local actions.

[0081] The prediction result for incomplete local actions can be determined based on the current human movement state, the remaining distance from the hand to the target anchor point, the remaining distance from the foot to the target support position, and the historical completion time of similar local actions. The historical completion time can be statistically analyzed based on confirmed swapping action samples and combined with the current movement rate of the target construction personnel to determine the expected duration of the corresponding action. For local actions that have started but not yet completed, the starting point of the determined action occurrence time interval is used as the starting point of the corresponding predicted occurrence time interval. The predicted completion time is determined based on the remaining distance, current movement rate, and expected duration, and the end point of the predicted occurrence time interval is determined based on the uncertainty range of the action completion time. For local actions that have not yet started, the earliest time when the expected start condition is met is determined as the starting point of the predicted occurrence time interval, and the latest time when the expected completion condition is met is determined as the end point of the predicted occurrence time interval. The corresponding predicted completion time is determined within the predicted occurrence time interval.

[0082] Based on the changes in human posture, support state, effective connection probability, and immediate load-bearing probability before and after each local action, an action state transition record is generated.

[0083] Each action state transition record includes at least the local action category, the time interval of occurrence, the work status before the action execution, and the work status after the action execution. For new hook closure and test pull confirmation, the action state transition record mainly records the changes in the effective connection probability and the probability of immediate load bearing of the new hook; for original hook release, it mainly records the changes in the effective protective connection status of the original hook; for support foot transfer and torso crossing, it mainly records the changes in support status, human posture, and human spatial range.

[0084] The pre-action working state is taken from the most recently determined working state before the corresponding local action occurs. The post-action working state is jointly determined by the pre-action working state and the state change relationship of the corresponding local action. After the new hook closes, the effective connection probability and immediate load-bearing probability of the new hook are re-determined based on the updated door lock state and visual hooking relationship, while retaining the support state before the action is executed. After test pull confirmation, the test pull micro-tension waveform is added to the hook connection state judgment, and the corresponding probability is re-determined. After the original hook is released, the effective protective connection state of the original hook is updated based on the door lock opening state and tension reduction result. The support foot transfer changes the foot contact position, support area, and center of gravity of the human body, while the torso crossing changes the human body posture and human body space range. Both retain the hook connection observation before the action is executed until a local action that changes the hook connection state occurs.

[0085] The occurrence sequence is determined based on the occurrence time interval and completion time of each local action. The action sequence constraint is determined when the new hook closure and test pull confirmation precede the release of the original hook, and when the support foot transfers, the probability of at least one hook being able to bear load immediately reaches a preset bearing probability threshold. Based on the action sequence constraint, the state transition records of each action are divided into normal state transition and abnormal state transition.

[0086] Action timing constraints are used to determine whether the changes in protective connections and supports corresponding to local actions meet the requirements for continuous protection with dual hooks. Actions where the original hook release occurs before the new hook completes the test pull confirmation, or where the probability of immediate load bearing of both hooks does not reach the preset load bearing probability threshold when the support foot is transferred, are all retained as abnormal state transfers and are not corrected to the normal hook replacement sequence.

[0087] When the time intervals of local actions do not overlap, the order in which action state transition records are called is determined according to the completion time of each local action. When the predicted time intervals of two or more incomplete local actions overlap, the allowed order of completion is determined based on the predicted time intervals of each local action, and state deduction branches are formed accordingly. Each state deduction branch calls action state transition records according to the corresponding order of completion, and determines the normal and abnormal state transitions within the branch based on the action timing constraints. Local actions such as early release of the original hook, early transfer of the support leg, or other actions that do not meet the action timing constraints continue to update the work status according to the corresponding state change relationship.

[0088] Then, taking the operation state as the starting state, each of the action state transition records is called sequentially according to the occurrence sequence to update the operation state in the subsequent reloading process, and the operation state changes caused by the abnormal state transition are retained to generate the reloading state evolution sequence.

[0089] The remounting state evolution sequence arranges the operational state after the action is executed according to the predicted completion time of each local action, so that both normal and abnormal remounting processes can form corresponding predicted operational states, avoiding future dangerous states caused by premature release of the original hook or premature transfer of the support foot due to filtering abnormal actions.

[0090] The state evolution sequence for hook replacement is assigned branch identifiers according to state deduction branches. Each state deduction branch forms a sub-sequence of states arranged according to the predicted completion time. Each predicted operation state records at least the corresponding future time, human posture, support state, human spatial range, and the effective connection probability, immediate load-bearing probability, and effective protective connection state of each hook. When multiple state deduction branches correspond to the same future time, the corresponding predicted operation states are saved according to the branch identifiers. When no new state transition occurs between the completion times of two adjacent actions within the same state deduction branch, the state components in the previous predicted operation state that have not been changed by the currently executed action are inherited.

[0091] Subsequent processing generates candidate actions for each state projection branch and the corresponding predicted operational state at future moments. Abnormal state transitions causing a decrease in the probability of effective connections, changes in support areas, or alterations in human space allow these actions to continue participating in the safety action screening process, without being overlooked due to the adoption of the normal reloading sequence instead of the actual predicted sequence.

[0092] Regarding the implementation details of predicting the motion range of the hoist at each future moment within a preset future time domain in the method of the embodiments of this application, in some examples of the embodiments of this application, a normal operation branch for maintaining the current operating state of the hoisting equipment and a deceleration operation branch for performing deceleration operation are set. The nominal pose, position prediction uncertainty, and motion rate of the hoist at each future moment under each operation branch are predicted according to the hoisting equipment motion state and the hoisting equipment operation data; wherein, the nominal pose of the deceleration operation branch is determined according to the allowable deceleration of the hoisting equipment, the braking establishment time, and the remaining swing state of the hoisting equipment after braking.

[0093] The hoisting motion status may include the current position of the hoisting equipment, direction of motion, swing angle, and rate of change of swing angle. The hoisting equipment operation data may include slewing status, luffing status, lifting status, operating speed, braking status, and sling length. All types of data are converted to the spatial coordinate system corresponding to the structural data of the work area, enabling comparison of the nominal hoisting posture, hoisting geometry, and human spatial range within the same spatial coordinate system.

[0094] The normal operation branch continues to simulate the hoisting motion according to the current slewing, luffing, and lifting states of the hoisting equipment. The deceleration operation branch adds allowable deceleration and braking establishment processes to the hoisting motion model, and continues to simulate the remaining swing of the hoisting after braking is established, so that the deceleration operation branch covers the motion processes corresponding to deceleration command transmission, braking mechanism response, decrease in motion rate, and attenuation of hoisting swing.

[0095] For the normal operation branch, the hoisting position can be updated hourly based on the current operating command, and the hoisting attitude can be updated based on the sling length, swing angle, and swing angle change rate. For the deceleration operation branch, the operating state before the actual deceleration operation is simulated during the braking establishment time; after braking establishment, the movement rates in the slewing direction, luffing direction, and hoisting direction are corrected based on the allowable deceleration, and the remaining swing after braking is simulated based on the swing angle and swing angle change rate at the moment of braking establishment. The hoisting position and attitude obtained at each future time point together constitute the nominal pose under the corresponding operation branch.

[0096] The uncertainty in position prediction can be determined based on the observation error of the current hoisting position and the state propagation relationship of the hoisting motion model. Taking the position observation error at the current moment as the initial uncertainty, the motion state error of the hoisting equipment, the swing angle observation error, and the disturbance error in the prediction process are propagated to each future moment to obtain the position prediction covariance, which characterizes the degree of dispersion of the predicted position in different spatial directions.

[0097] The maximum data time difference is determined based on the data time difference of various multimodal construction data relative to the current time. The remaining execution delay of the current running instruction is determined as the branch control delay of the normal operation branch, and the communication and execution delay of the deceleration control instruction is determined as the branch control delay of the deceleration operation branch. After various multimodal construction data are correlated according to the construction event clock, the non-negative time difference between the timestamp of the latest valid data and the current time is calculated, and the maximum value among the time differences is determined as the maximum data time difference. The remaining execution delay of the current running command can be determined based on the start time of the current command, the predetermined duration, and the current time; when the current running command has been fully applied to the hoisting equipment, the corresponding remaining execution delay is set to zero. The communication and execution delay of the deceleration control command can be determined based on the control command transmission time, the hoisting equipment controller processing time, and the braking mechanism response time, and can be calibrated using the on-site response records of the hoisting equipment.

[0098] The nominal occupancy range of the hoist is determined based on the hoist geometry and the nominal pose, and the uncertainty of the nominal occupancy range is expanded based on the position prediction uncertainty.

[0099] The lifting geometry can be obtained from lifting mission data or lifting 3D reconstruction results. The lifting geometry formed based on the lifting geometry is translated and rotated according to the nominal pose under the corresponding operational branch to obtain the nominal occupancy range of the lifting at the corresponding future time.

[0100] At the current moment As the starting point for prediction, Relative to the current time The future time offset, and ,in, This is to preset the duration of the future time domain. (Using...) Indicates the running branch, This indicates a normal operating branch. This indicates the deceleration branch. The error ellipsoid is determined based on the position prediction covariance and the preset confidence coefficient.

[0101] Equation (5) in, For running branches Next Future Moment The corresponding error ellipsoid; To predict the covariance of the hoisting position at a corresponding future time; The confidence coefficient is determined based on a preset confidence level; This is a three-dimensional spatial offset vector relative to the nominal predicted position; for Transpose of; It is a third-order identity matrix; The covariance stabilizer is positive, and its unit of measurement is the same as that of the diagonal elements of the position prediction covariance. This covariance stabilizer ensures that inverse matrix operations can still be performed even when the position prediction covariance is a singular matrix. The error ellipsoid is used to expand the uncertainty of the nominal occupancy range of the load, ensuring that the expanded range covers the directional position prediction deviation under a preset confidence level.

[0102] Based on the displacement formed within a time period determined by the maximum data time difference and the branch control delay of the corresponding operating branch, the nominal occupied range of the load after uncertainty expansion is extended by time delay, and the range after time delay expansion is perturbed by combining the sling elasticity, environmental disturbance and residual swing state, so as to obtain the range of motion of the load under each operating branch at each future time.

[0103] The future movement range of the load under each operating branch is determined by the following formula: Equation (6) in, For running branches Lowering load in the future The range of motion; This refers to the nominal pose of the load under the corresponding running branch; The lifting geometry is determined based on the lifting geometry dimensions; The nominal occupancy range of the hoisting after the nominal pose transformation; For Minkowski sum operations; Centered on the origin of the spatial coordinate system, with a radius of... Spatial expansion region; For running branches Future time offset The expansion distance below.

[0104] The expansion distance is determined according to the following formula: Equation (7) in, This is a preset foundation expansion amount set based on the measurement error of the lifting geometry and the error of the lifting positioning foundation; For running branches The following is used to determine the motion rate of the time delay expansion; This represents the maximum time difference between various types of multimodal construction data and the current moment. For running branches Branch control delay, where, The remaining execution delay for the currently running instruction. For the communication and execution delay of deceleration control commands; This represents the non-negative additional expansion corresponding to the prediction bias of sling elasticity, environmental disturbance, and residual oscillation.

[0105] In the When the motion rate changes within a defined time delay interval, Take the upper bound of the predicted motion rate within the time delay interval; if the motion rate remains constant, take the future time. The corresponding predicted motion rate. Therefore, The additional displacement range that represents the combined effect of data staleness and branch control delay.

[0106] Nominal change in the remaining swing of the load enters The nominal pose represented; Only the predicted deviation of the actual residual swing relative to the nominal change is taken into account, and determined in conjunction with the allowable elastic variation range of the sling under the corresponding load and the environmental disturbance boundary. Thus, the nominal swing trajectory and the swing prediction deviation are respectively included in the nominal occupied range and the disturbance expansion range, avoiding repeated expansion of the same residual swing amount.

[0107] Furthermore, the range of motion under the normal operation branch is taken as the range of motion of the hoisting equipment at each future moment, and the range of motion under the deceleration operation branch is used to determine whether the hoisting equipment can maintain or restore safe handling conditions after performing deceleration operation.

[0108] Subsequent processing compares the movement ranges under the two operational branches with the post-treatment human space range corresponding to the same future time and the same predicted operational state. If the set of safety actions in the normal operation branch is not empty, it indicates that a corresponding safety action window still exists while the hoisting equipment maintains its current operational state. If both the normal operation branch and the deceleration operation branch have empty safety action sets, it indicates that deceleration can maintain or restore safe handling conditions. If both operation branches have empty safety action sets, the corresponding future state is determined to be unrecoverable. Therefore, the movement range of the normal operation branch is used to determine the action commitment point, and the movement range of the deceleration operation branch is used to evaluate whether deceleration intervention can extend or restore the safe action window.

[0109] Figure 3 A flowchart illustrating an example of constructing a set of safety handling actions in a method according to an embodiment of this application is shown.

[0110] In step S310, for each candidate disposal action at a future moment, the human body space range after disposal, the effective connection probability of each hook, and the probability of immediate load bearing are determined according to the post-disposal work status, and the support stability margin is determined according to the directional distance between the projection of the human body's center of gravity and the boundary of the support area.

[0111] For any candidate action, the predicted working state corresponding to that action is used as the starting state. State deduction is performed based on the range of motion of the human joints, the direction of movement of the candidate action, changes in foot contact position, and changes in hook state. The post-action working state obtained from the state deduction includes at least the post-action human posture, post-action support state, post-action human center of gravity, post-action human spatial range, and the effective connection probability and immediate load-bearing probability of each hook.

[0112] For hook states that change due to candidate handling actions, predict the door lock state, micro-tension characteristics of the test pull, and the geometric state of the tether after the action is executed based on the corresponding action state transition record. Combined with the anchor point identity and bearing capacity, redetermine the effective connection probability and the probability of immediate bearing according to the hook connection state judgment method. For hook states that do not change due to candidate handling actions, maintain the effective connection probability and the probability of immediate bearing in the corresponding predicted operation state.

[0113] The support area is determined based on the contact positions between the feet and the floor slab, steel beam, or other standing structures after treatment. When both feet are on the same support surface, the support area is formed based on the contact range of the feet; when the feet are on adjacent structures, the support area is determined based on the two contact ranges and the load-bearing area between the two contact ranges in the structural data of the work area. The minimum directed distance from the projection of the center of gravity of the treated person to the boundary of the support area is used as the support stability margin. When the projection of the center of gravity of the person is inside the support area, the directed distance is positive; when it is outside the support area, the directed distance is negative; when it is at the boundary of the support area, the directed distance is zero.

[0114] The post-treatment human body spatial range is formed into a three-dimensional human body envelope based on the post-treatment human body joints and the target construction worker's body dimensions. When the candidate treatment action involves a continuous movement process, the intermediate human body posture corresponding to each future moment is determined, and the post-treatment human body spatial range for the corresponding future moment is formed; when the candidate treatment action is completed, the corresponding human body spatial range is formed using the action completion posture. Subsequent processing compares the human body spatial range at each future moment with the hoisting movement range at the same future moment, ensuring that the action execution path is covered by multiple human body spatial ranges arranged in time.

[0115] In step S320, the predicted fall direction is determined based on the post-treatment work status and the structural data of the work area, and the protective contribution is determined based on the geometric relationship between the hook, anchor point and tether relative to the predicted fall direction and the immediate load-bearing probability of the corresponding hook.

[0116] The location of the edge, the direction outside the edge, and the potential fall space below the current support surface are determined from the structural data of the work area. Based on the position of the person's center of gravity relative to the boundary of the support area after the incident, the target boundary that the person might cross when losing support is determined. The direction from the person's center of gravity after the incident to the potential fall space outside the target boundary is determined as the predicted fall direction. When multiple potential fall boundaries exist, the protective contribution of two hooks is determined for the predicted fall direction corresponding to each potential fall boundary. The corresponding protective contributions are aggregated according to the parallel protection relationship, and the predicted fall direction with the smallest protective continuity is determined as the screening direction for candidate incident actions.

[0117] For each hook, the fall protection direction coverage factor is determined based on the hook connection position, anchor point position, tether path, available tether length, and predicted fall direction. When the anchor point position and tether force direction can limit further movement of the human body along the predicted fall direction, and the tether path remains continuous, the fall protection direction coverage factor is determined within a range greater than zero and not greater than 1, based on the angle between the force direction and the predicted fall direction and the remaining constraint distance of the tether. When the tether cannot provide constraint before the human body enters the hazardous space, the fall protection direction coverage factor is determined to be zero. When the tether path passes through sharp edges or obstacles, or when the swinging path formed after force intersects with surrounding components, the fall protection direction coverage factor is reduced based on the corresponding geometric deviation.

[0118] The fall arrest direction coverage factor and the immediate load-bearing probability of the corresponding hook are used together as the protective contribution of the hook. When the hook is connected to an anchor point that meets the fall arrest load-bearing conditions but has not yet formed an effective standby load state, its immediate load-bearing probability limits the corresponding protective contribution; when the hook is in an effective standby load state but its force path deviates from the predicted fall direction, its fall arrest direction coverage factor limits the corresponding protective contribution.

[0119] In step S330, the protective contribution of each hook is aggregated according to the parallel protection relationship of forming a fall protection force path by at least one hook, so as to obtain the protection continuity characterizing the degree of continued existence of the effective fall protection force path.

[0120] For example, the protection continuity is determined according to the following formula: Equation (8) in, To execute candidate disposal actions Later in the future The continuity of protection; For the first One hook is used in the execution of candidate disposal actions. The fall protection direction coverage coefficient, and ; To execute candidate disposal actions The next The probability that each hook can be immediately loaded. This indicates the protective contribution of the corresponding hook in forming an effective fall-prevention force path along the predicted fall direction.

[0121] The product term represents the remaining degree of directional protective contribution when neither hook contributes directionally when aggregated according to the parallel protection relationship. The protection continuity is a continuity evaluation quantity determined based on the protective contributions of the two hooks, and its calculation does not presuppose that the states of the two hooks are independent. When the protective contributions of both hooks are zero, the protection continuity is zero; if the protective contribution of one hook remains unchanged, and the protective contribution of the other hook increases, the protection continuity either increases or remains unchanged. The parallel aggregation relationship does not directly add the immediate bearing probabilities of the two hooks arithmetically, but rather constrains the protective contributions of the corresponding hooks with fall arrest direction coverage coefficients, and then aggregates them according to a complementary product relationship; whether the aggregation result satisfies the protection continuity condition is determined by comparing it with the preset lower limit of protection continuity.

[0122] In step S340, for each of the aforementioned operating branches, candidate actions that simultaneously satisfy the preset human body movement attainability conditions, the support stability margin not less than the preset stability margin lower limit, the effective connection probability of at least one hook reaching the preset connection probability threshold, the protection continuity not less than the preset protection continuity lower limit, and the post-action human body space range not intersecting with the hoisting movement range of the corresponding operating branch, are combined into a set of safe handling actions for the corresponding operating branch at the corresponding future time. The set of safe handling actions for the normally operating branch is the set of safe handling actions; the set of safe handling actions for the deceleration operating branch is used to determine whether the deceleration operation can restore the safe handling conditions.

[0123] The candidate actions that meet the preset human movement reachability conditions at the corresponding future time are combined into a candidate action set. Based on the support stability margin, a set of support stability candidate actions is determined from the set of candidate actions:

[0124] Equation (9) Based on the effective connection probability of each hook, a set of effective connection candidate actions is determined from the set of support stability candidate actions: Equation (10) Based on the continuity of protection, a set of state-safe candidate actions is determined from the set of valid connection candidate actions: Equation (11) in, For the future The corresponding set of candidate actions for supporting stability; This is the set of corresponding valid candidate actions for connection. This is the corresponding set of candidate actions for state safety. To execute candidate disposal actions Subsequent support and stability margin; To preset a lower limit for stability margin; To execute candidate disposal actions The later The probability of an effective connection of each hook; Set a preset connection probability threshold; This is a preset lower limit for protection continuity. The boundary distance to be maintained is set according to the projection of the human body's center of gravity within the support area. Based on the confirmed valid and invalid connection samples, the calibration was performed. The required level of protection is set based on the continuous existence of at least one effective fall-prevention force path.

[0125] Within the set of candidate safe actions for the state, the set of safe handling actions for each operational branch is determined according to the following formula: Equation (12) in, For running branches In the future The corresponding set of safety response actions; To execute candidate disposal actions Later in the future The corresponding human body space range after treatment; This represents an empty set. When two spatial ranges share a common spatial point, the corresponding candidate action is excluded from the safety action set of that operational branch. Actions where the human body's spatial range intersects with adjacent restricted areas or fixed obstacles after the action is taken can be excluded during the candidate action generation stage, ensuring that the actions in the candidate action set meet the preset human body reachability conditions.

[0126] The normal operation branch and the deceleration operation branch use the same candidate handling actions, post-handling work status, support stability margin, effective connection probability, and protection continuity, and respectively call the hoisting motion range under the corresponding operation branch for spatial filtering. Therefore, the safety handling action sets formed by the two operation branches have the same personnel status basis, and their difference corresponds to the different hoisting motion ranges formed when the hoisting equipment maintains its current operating state and performs deceleration operations. The safety handling action set of the normal operation branch is used to determine whether the safety action window continues to exist, and the safety handling action set of the deceleration operation branch is used to determine whether the deceleration operation can maintain or restore the safety action window.

[0127] Figure 4 An example operation flowchart is shown for determining the warning triggering time driven by the action commitment point in the method according to an embodiment of this application.

[0128] In step S410, the future moment when the set of safety handling actions for the normal operation branch first changes from a non-empty state to an empty state is determined as the action commitment point.

[0129] In the discrete prediction time domain, the set of safety response actions for the normal operation branch is detected in chronological order of future moments. When the set of safety response actions for the preceding future moment is non-empty and the set of safety response actions for the following future moment is empty, the following future moment is determined as the action commitment point. The prediction step size in the discrete prediction time domain can be set according to the update cycle of the multimodal construction data, the duration of local actions, and the rate of change of the hoisting motion state.

[0130] When it is necessary to refine the time location of the action commitment point, the prediction step size can be reduced between the future time corresponding to the last non-empty safety action set and the future time corresponding to the first empty safety action set. The predicted operation status and safety action set can then be redefined, and the time of the first empty set obtained from the re-detection can be used as the action commitment point. This refinement process uses the same candidate actions and safety screening conditions.

[0131] If the set of safety action actions for the normally operating branch is empty during the initial detection, it indicates that the current state has exceeded the prediction range for the set of safety action actions to change from a non-empty state to an empty state. In this round of processing, the immediate early warning process is initiated directly, and the search for future action commitment points is no longer conducted. If the set of safety action actions remains non-empty before the end of the preset future time domain, multimodal construction data will continue to be monitored, and the state simulation will be performed again after the data is updated.

[0132] In step S420, for each safety action at each future moment and in each operating branch, three types of remaining safety time before the load sweep, protection continuity, and support stability margin reach their respective safety boundaries are determined, as well as three types of corresponding action time for completing load avoidance, restoring at least one hook to the effective standby state, and restoring stable support.

[0133] All three types of remaining safety time are calculated from the future moment currently being evaluated. The human body space range after treatment is compared with the hoisting movement range of the corresponding operating branch along the time direction. The time difference between the moment when the two space ranges first intersect and the starting point is determined as the sweeping remaining time; when the two space ranges have already intersected at the starting point, the sweeping remaining time is determined to be zero.

[0134] Based on the future changes in the immediate load-bearing probability, fall protection direction coverage, and protection continuity of each hook in the hook-up evolution sequence, the time difference between the moment when the protection continuity first drops to the preset lower limit of protection continuity and the starting point is determined as the remaining protection time. The support stability margin is determined based on the human center of gravity and support area at each future moment, and the time difference between the moment when the support stability margin first drops to the preset lower limit of stability margin and the starting point is determined as the remaining stability time. If the corresponding safety boundary has not been reached within the preset future time domain, the duration between the starting point and the preset future time domain end point can be used as the cutoff value for the corresponding remaining safety time.

[0135] The evacuation time is determined based on the distance the human body needs to move and the rate of action that can be performed to exit the expected sweeping area of ​​the hoist after executing the candidate disposal action. The protection recovery time is determined based on the continuous time required for the hands to reach the target anchor point, for the new hook to close, and for a test pull to confirm. The support recovery time is determined based on the distance the body's center of gravity needs to move, the distance the torso needs to move, and the corresponding rate of action required to restore the body's center of gravity to a position that meets the preset stability conditions. The above disposal times can be determined based on the duration of similar local actions already completed by the target construction personnel, or based on the confirmed historical action durations and their preset safety percentiles.

[0136] When at least one hook is already in an effective standby state at the corresponding future time, and the candidate action will not disrupt this effective standby state, the protection recovery time is set to zero. When the support state at the corresponding future time has met the preset stability conditions, and the candidate action will not change this support state, the support recovery time is set to zero. In other cases, the corresponding handling time is determined according to the action process required to restore the corresponding target state.

[0137] In step S430, the minimum difference between the three types of remaining safety time and the corresponding handling time is subtracted from the system response delay corresponding to system calculation, communication transmission and early warning output to obtain the recoverable time margin of the corresponding action; the maximum recoverable time margin under the same future time and running branch is determined as the irreversible time margin; when the set of safety handling actions is empty, the irreversible time margin is determined to be zero; running branches with irreversible time margin greater than zero are determined to be recoverable states, otherwise they are determined to be irreversible states.

[0138] The recoverable time margin of candidate actions is determined according to the following formula: Equation (13) in, For running branches Next candidate handling action In the future Recovery time margin; The remaining sweeping time before the human space area is dealt with after the intrusion of the hoisting movement range under the corresponding operating branch; The evacuation time required to complete the lifting and avoidance maneuver; The remaining time of protection before the protection continuity drops to the preset lower limit of protection continuity; The protection recovery time is required to restore at least one hook to an effective standby state; To support the remaining time for stability before the stability margin decreases to the preset lower limit of the stability margin; The support recovery time is required to restore stable support. This refers to the system response delay corresponding to system calculations, communication transmission, and early warning output. Each time value pertains to the same candidate action. and the same future moment Sure.

[0139] The three time differences represent the time margins corresponding to spatial avoidance, protection recovery, and support recovery, respectively. The minimum of these three time differences is taken, ensuring that the safety conditions exhausted first determine the recoverable time margin of the candidate action. The system response latency can be calibrated based on the processing records of edge computing nodes, the transmission time of communication links, and the output response time of the early warning terminal, and updated according to the actual response time recorded during system operation.

[0140] The irreversible time margin of the corresponding running branch is determined by the following formula: Equation (14) in, For running branches In the future The irreversible time margin is determined as follows: When the set of safety response actions is non-empty, the candidate action with the largest recoverable time margin is selected, and its recoverable time margin is determined as the irreversible time margin of the corresponding operating branch; when the set of safety response actions is empty, the irreversible time margin is determined to be zero. When the irreversible time margin is greater than zero, the corresponding operating branch is in a recoverable state; when the irreversible time margin is equal to or less than zero, the corresponding operating branch is in an irreversible state.

[0141] When determining the irreversible time margin, the candidate action that achieves the maximum recoverable time margin and its direction of movement are retained. If multiple candidate actions achieve the same maximum value, one of them can be selected according to a preset action priority. The selected candidate action serves as the basis for generating personnel action adjustment information.

[0142] In step S440, a positive early warning protection margin is determined based on the preset safety quantile of the personnel early warning response time, and the moment when the irreversible time margin of the normal operation branch between the current moment and the action commitment point is first not greater than the early warning protection margin is determined as the margin trigger moment.

[0143] The personnel early warning response time can be determined based on the controlled operation records. Its starting point is when the early warning terminal outputs a prompt that can be perceived by the target construction personnel, and its ending point is when a change occurs in the human posture, hand position, or foot contact position corresponding to the target's handling action. A preset safety quantile is selected based on the statistical distribution of personnel early warning response times under the corresponding operation type, and the response time corresponding to this quantile is determined as the early warning protection margin.

[0144] The aforementioned early warning protection margin is used to cover the reaction time required for target construction personnel to perceive the early warning information and begin to take action; the time required for system calculation, communication transmission, and early warning output has been included in the system response delay. These two correspond to the personnel reaction process and the system response process, respectively.

[0145] In the discrete prediction time domain, the irreversible time margin of the normal operation branch is compared with the early warning protection margin in chronological order of future moments. When the irreversible time margin of the preceding future moment is greater than the early warning protection margin, and the irreversible time margin of the following future moment is not greater than the early warning protection margin, the following future moment is determined as the margin trigger moment. When the irreversible time margin of the current moment is no longer greater than the early warning protection margin, the current moment is determined as the margin trigger moment. To refine the margin trigger moment, the prediction step size between two adjacent future moments can be reduced, and the irreversible time margin can be recalculated.

[0146] The set of safety handling actions for the normal operation branch at the action commitment point is empty, corresponding to an irreversible time margin of zero. Therefore, when the action commitment point has been determined and the early warning protection margin is positive, the margin trigger time can be determined between the current time and the action commitment point.

[0147] In step S450, the earlier time is selected from the margin trigger time and the time obtained by advancing the warning protection margin relative to the action commitment point as the candidate warning time; when the candidate warning time is earlier than the current time, the current time is determined as the warning trigger time, otherwise the candidate warning time is determined as the warning trigger time.

[0148] The warning trigger time is determined by the following formula: Equation (15) in, The time when the warning is triggered; This is the margin trigger time; For action commitment points; The warning protection margin is positive. The inner operation selects the earlier time from the margin trigger time and the time obtained from the early warning protection margin relative to the action commitment point. The candidate warning time is determined by the condition that is more likely to be met between the two trigger conditions: irreversible time margin decrease and action commitment point proximity. The outer operation uses the current time as the lower limit to avoid the warning trigger result falling into the time interval that has already passed.

[0149] If the candidate warning time is earlier than the current time, it indicates that the current available time is less than the required warning protection margin, and the system outputs an immediate warning at the current time. If the candidate warning time is between the current time and the action commitment point, the system outputs a warning at the candidate warning time. After the warning trigger time is determined, the recovery status of the two operating branches at that time, the irreversible time margin, and the candidate handling actions that achieve the maximum recoverable time margin are transmitted to the construction safety warning information generation process.

[0150] Regarding the implementation details of generating construction safety early warning information at the early warning triggering time in the method of this application embodiment, in some examples of this application embodiment, the safety handling action set and recovery status of the normal operation branch and the deceleration operation branch at the early warning triggering time are obtained, and the irreversible time margin corresponding to the operation branch in the recoverable state is obtained.

[0151] The two operational branches use the same predicted operational status, candidate actions, and safety judgment conditions at the same warning trigger time, and determine the set of safety actions using the lifting motion range under the corresponding operational branch. Therefore, the recovery status of the two operational branches corresponds to the same personnel status and future time, and their difference comes from the lifting equipment maintaining its current operating status or performing a deceleration operation, resulting in the lifting motion range.

[0152] When the set of safety response actions is not empty and the irreversible time margin is greater than zero, the corresponding operational branch is determined to be in a recoverable state. When the set of safety response actions is not empty but the irreversible time margin is equal to or less than zero, it indicates that although there are candidate response actions in the set that meet the conditions of action reachability, support stability, protection continuity, and spatial separation, the corresponding actions cannot be completed before the first safety boundary is reached, and therefore the corresponding operational branch is determined to be in an unrecoverable state. When the set of safety response actions is empty, the corresponding operational branch is also in an unrecoverable state, and the selection of target response actions from this set is stopped.

[0153] When the normal operation branch is in a recoverable state, the candidate action with the largest recoverable time margin is selected from the set of safety handling actions of the normal operation branch as the target handling action, and personnel action adjustment information is generated according to the movement direction of the target handling action.

[0154] The movement direction of candidate actions can be determined based on the spatial change in the person's center of gravity before and after the action, and converted into a leftward, rightward, or backward direction. Maintaining the current posture, stopping crossing, re-attaching, or waiting for the load to pass are written into the personnel action adjustment information as the corresponding action types. Re-attaching target actions can also be associated with the target anchor point direction.

[0155] When multiple candidate actions have the same maximum recoverable time margin, they can be compared sequentially in ascending order of action completion time, descending order of post-action support stability margin, and descending order of post-action protection continuity. The candidate action ranked first will be determined as the target action. This ranking is only used for candidate actions with the same maximum recoverable time margin and does not change the conditions for forming the set of safety action actions.

[0156] When both of the operating branches are in a recoverable state, and the irreversible time margin increased by the deceleration operating branch relative to the normal operating branch reaches a preset recovery gain threshold, a deceleration command for the hoisting equipment is generated.

[0157] The irreversible time margin added by the deceleration branch relative to the normal operation branch is determined according to the following relationship: Equation (16) in, To provide the irreversible time margin for normally operating branches at the time of warning triggering, To provide the irreversible time margin for the deceleration branch at the moment of warning triggering, A preset recovery gain threshold is set. The preset recovery gain threshold can be pre-set based on the communication and execution delay of the deceleration control command, the braking setup time of the hoisting equipment, and the on-site safety response requirements.

[0158] The irreversible time margin comparison is performed only when both operating branches are in a recoverable state. If the deceleration operating branch is in an irreversible state or the increased irreversible time margin does not reach the preset recovery gain threshold, the personnel action adjustment information generated based on the normal operating branch is retained. When the recovery gain condition is met, a hoisting equipment deceleration command is generated in addition to the personnel action adjustment information.

[0159] The hoisting equipment deceleration command may include the hoisting equipment identifier, the target deceleration state, and the command effective time. The target deceleration state is determined based on the allowable deceleration or target operating rate used when forming the deceleration operation branch, so that the control state corresponding to the output command is consistent with the predicted conditions of the hoisting movement range under the deceleration operation branch.

[0160] When the normal operation branch is in an unrecoverable state and the deceleration operation branch is in a recoverable state, the candidate action with the largest recoverable time margin is selected from the set of safety handling actions of the deceleration operation branch as the target handling action, and a forced deceleration command is generated to control the hoisting equipment to execute the deceleration operation corresponding to the deceleration operation branch and personnel action adjustment information corresponding to the target handling action.

[0161] Under this branch combination, the hoisting equipment has no action that can be completed before reaching the safety boundary while maintaining its current operating state, while the deceleration branch still retains candidate actions with a positive recoverable time margin. Therefore, the target action is determined from the set of safe actions in the deceleration branch, and personnel action adjustment information is generated according to the movement direction and action type of the target action.

[0162] The forced deceleration command uses the deceleration control conditions corresponding to the formation of the deceleration operation branch, and is directly generated when the branch combination is satisfied, without the need for a recovery gain threshold comparison. The forced deceleration command and personnel action adjustment information carry the same construction event identifier and early warning trigger time, so that the deceleration operation of the hoisting equipment and the action adjustment of the target construction personnel correspond to the same predicted operation state and hoisting movement range.

[0163] Furthermore, when both of the aforementioned operating branches are in an unrecoverable state, an emergency stop command for the hoisting equipment, an on-site audible and visual alarm command, and a manual handling prompt are generated; at least one of the personnel action adjustment information, the hoisting equipment deceleration command, the forced deceleration command, the hoisting equipment emergency stop command, the on-site audible and visual alarm command, and the manual handling prompt is written into the construction safety early warning information.

[0164] When both operating branches are in an unrecoverable state, neither maintaining the current operating state nor executing the preset deceleration operation of the hoisting equipment can generate a safe handling action with a positive recoverable time margin. An emergency stop command for the hoisting equipment is sent to the hoisting equipment control interface, an on-site audible and visual alarm command is sent to the corresponding audible and visual alarm device in the target edge work area, and a manual handling prompt is sent to the on-site safety management terminal. The manual handling prompt may include the target edge work area, the target construction personnel, the warning trigger time, and the recovery status of the two operating branches.

[0165] Construction safety early warning information is linked to personnel action adjustment information, equipment control commands, on-site audible and visual alarm commands, and manual handling prompts according to construction event identifiers, and the target recipients and output times of each piece of information are recorded. Personnel action adjustment information is sent to the prompting terminal worn by the target construction personnel or the on-site directional prompting device; hoisting equipment deceleration commands, forced deceleration commands, and hoisting equipment emergency stop commands are sent to the hoisting equipment control interface; on-site audible and visual alarm commands and manual handling prompts are sent to the on-site alarm device and the safety management terminal, respectively. Thus, the recovery status of the two operational branches is converted into construction safety early warning information that matches the corresponding candidate handling actions and hoisting equipment control status.

[0166] Figure 5 A schematic diagram illustrating the operational mechanism of an example of a construction safety early warning method based on multimodal behavioral reasoning according to an embodiment of this application is shown.

[0167] like Figure 5 As shown, the system acquires video data of construction workers, double hook status data, hoisting equipment operation data, and work area structure data of the target construction personnel within the target edge work area, and associates data from different sources with a unified construction event timeline and work area coordinate system.

[0168] Based on the video data of the construction workers, the body posture and support status of the target construction workers are determined, and the connection status of each hook is determined in conjunction with the double hook status data. Based on the time intervals and completion times of each local action during the double hook replacement process, and the resulting changes in the work status, the subsequent hook replacement process is simulated, forming a hook replacement state evolution sequence. This hook replacement state evolution sequence records the body posture, support status, and hook protection connection status at each future moment, and retains the changes in the work status caused by abnormal action sequences.

[0169] Based on the connection status of each hook, its immediate load-bearing capacity, and the geometric relationship between the hooks, anchor points, and tethers relative to the predicted fall direction, the continuity of protection corresponding to the candidate response actions is determined. The continuity of protection characterizes the degree to which at least one hook continues to form an effective fall-prevention force path along the predicted fall direction after the candidate action is performed.

[0170] The hoisting motion state is determined based on the hoisting equipment's operating data, and the hoisting motion is predicted according to the control conditions of maintaining the current operating state and performing deceleration operations. The hoisting prediction results are converted to the spatial coordinate system corresponding to the structural data of the work area, and combined with the hoisting geometry, position prediction uncertainty, data time difference, control delay, and hoisting disturbances, a dual-branch hoisting motion range corresponding to the normal operation branch and the deceleration operation branch is formed. The structural data of the work area is also used to determine the support area, qualified anchor points, edge boundaries, fixed obstacles, and areas accessible to human movement.

[0171] For each predicted operational state in the alternation state evolution sequence, candidate handling actions are generated based on the reachable area of ​​human movement, and the operational state after executing each candidate handling action is deduced. The dual-branch hoisting motion range is used as a spatial constraint in the feasible domain deduction of the handling action. During and after the execution of the candidate handling actions, the system performs spatial intersection judgment between the human spatial range and the hoisting motion range of the corresponding operating branch. At the same time, the support stability state is judged based on the positional relationship between the human center of gravity and the boundary of the support area after the handling, and the fall protection status is judged based on the connection status and protection continuity of each hook.

[0172] For both the normal operation branch and the deceleration operation branch, candidate handling actions that meet the conditions of being accessible to human movement, having stable support, effective hook connection, continuous protection, and not intersecting with the range of human space and the range of load movement are combined to form a set of safe handling actions for each future time point of the corresponding operation branch. . Figure 5 The feasible domain deduction of the disposal action shown in the figure represents the process of performing state deduction and safety screening of candidate disposal actions using the above-mentioned personnel status, work area structural constraints and hoisting space constraints.

[0173] Based on the change of the set of safety response actions in the normal operation branch over time, the future moment when the set of safety response actions first changes from a non-empty state to an empty state is determined as the action commitment point. For the safety response actions of each operation branch, the remaining safety time before the load sweep, protection continuity, and support stability margin reach the corresponding safety boundary is compared with the time required to complete the corresponding response to determine the recoverable time margin of each safety response action. The maximum recoverable time margin under the same future moment and the same operation branch is determined as the irreversible time margin. The warning trigger time is determined based on the irreversible time margin of the normally operating branch, the action commitment point, and the personnel warning response time.

[0174] At the time the warning is triggered, tiered intervention is performed based on the safety action sets, irreversible time margins, and recovery status of the normal operation branch and the deceleration operation branch. When the normal operation branch is in a recoverable state, personnel action adjustment information is generated based on the candidate action with the largest recoverable time margin. When both operation branches are in a recoverable state, and the irreversible time margin increased by the deceleration operation branch reaches a preset recovery gain threshold, a deceleration command for the hoisting equipment is further generated. When the normal operation branch is in an irreversible state while the deceleration operation branch is in a recoverable state, a forced deceleration command and personnel action adjustment information corresponding to the deceleration operation branch are generated. When both operation branches are in an irreversible state, an emergency stop command for the hoisting equipment, an on-site audible and visual alarm command, and a manual handling prompt are generated.

[0175] Figure 5 The terms "personnel action adjustment," "gradual deceleration of hoisting equipment," and "emergency stop alarm handling" correspond to three levels: personnel handling, general deceleration or forced deceleration, and emergency stop alarm and manual intervention, respectively. Thus, the evolution of the swapping status, prediction of the movement of the dual-branch hoisting equipment, screening of safe handling actions, determination of action commitment points, and gradual intervention output form a continuous construction safety early warning and processing process.

[0176] Considering the risk of personnel injury in high-rise edge-mounted load transfer and hoisting conflict scenarios, a multi-physics coupled simulation engine is used to construct EdgeLift-Transfer-Sim (a digital twin simulation experimental environment for edge-mounted load transfer) to verify the early warning performance of the method in this application under different construction conditions. The simulation experimental environment establishes a spatial model based on the geometric parameters of the floor slab, steel beams, anchor points, hoisting equipment, hoisting load, and virtual construction personnel. The hoisting motion process is determined according to the operating status of the hoisting equipment, the motion status of the slings, and external wind load, ensuring that the simulated construction personnel video data, double hook status data, hoisting equipment operating data, and work area structural data have a unified time reference and spatial coordinate system. For the hoisting load sway during the lifting and rotation of the hoisting equipment, a nonlinear sway equation incorporating wind load and random dynamic disturbances is used to deduce the hoisting load sway angle.

[0177] Equation (17) In the formula, This is the equivalent moment of inertia corresponding to the swing of the suspended load. This is the equivalent damping coefficient. For the lifting mass, It is the acceleration due to gravity. The effective length of the sling. The hoisting swing angle over continuous time. This refers to the equivalent torque acting on the swing of the load during the operation and braking process of the hoisting equipment. air density, This is the drag coefficient. To accommodate the windward area of ​​the load, The equivalent force arm corresponding to the point of application of the wind load. For wind speed, To conform to a mean of zero and a variance of The Gaussian random disturbance torque is applied. Sensor measurement noise is applied independently to the observation data generated based on the actual swing angle, load position, and hook state, and does not participate in the update of the actual dynamic state. Accordingly, the load motion range determined based on the simulation data can reflect the load geometry, continuous swing, wind load disturbance, and control response process.

[0178] The simulation experiment employed a layered combination approach to set scene factors, including 24 different floor slab or steel beam layouts, 8 different anchor point spacings, 6 different combinations of hoisting dimensions and mass, and virtual construction workers with different heights, arm spans, and movement rhythms, generating a total of 3600 simulation sequences, each 12 seconds long. Each sequence outputs two channels of RGB-D (Red-Green-Blue-Depth) video at 30 frames per second, outputs hook lock status and trial tension data at 50 Hz, outputs hoisting equipment operating status at 20 Hz, and simultaneously outputs the state truth trajectory with millimeter-level spatial coordinate resolution. The simulation sequences cover 0–2.4 s of hook visual occlusion, 0–12° of initial hoisting swing angle, 0–15% of camera frame drop rate, 0–10% of intelligent hook data packet loss rate, and 40–180 ms of hoisting equipment communication round-trip delay, used to create a composite construction state including video occlusion, missing sensor data, hoisting swing, and control delay. In addition to the simulation sequence, to determine the warning performance boundary when visual occlusion and load swing angle increase together, under the same scenario model and preset false alarm control conditions, the duration of visual occlusion was taken within the range of 0 to 10 seconds, and the amplitude of load swing angle was taken within the range of 0 to 30 degrees, forming... Figure 7 The combined disturbance scan data is shown. The false negative rate for each parameter combination is the proportion of risk events for which no warning was issued to the total number of corresponding risk events.

[0179] The experiment selected four representative methods as comparison methods, including YOLO-PPE (You Only Look Once - Personal Protective Equipment, a personal protective equipment detection method based on single-stage target detection), CNN-LSTM (Convolutional Neural Network - Long Short-Term Memory), Social-LSTM (Social Long Short-Term Memory), and 3D-HOI (Three-Dimensional Human-Object Interaction), respectively to represent the comparative processing methods based on protective equipment appearance detection, video action sequence classification, judgment of the intersection of future personnel trajectories and dangerous areas, and judgment of multi-view three-dimensional human-object interaction. The method in this application embodiment and each comparison method use the same training set and test set division, and each simulation sequence is assigned to only one of the training set or the test set.

[0180] Evaluation metrics include event-level AUPRC (Area Under the Precision-Recall Curve) and pre-commitment recall. The metrics include: early warning lead time, late alarm rate, FA / h (False Alarmsper Hour), and end-to-end latency. Pre-commitment recall is the proportion of risk events for which an alarm has been issued before the action commitment point is reached, out of the total number of risk events. Early warning lead time is the difference between the time corresponding to the action commitment point and the time of alarm output. Late alarm rate is the proportion of test sequences in which a risk event occurs, where an alarm is issued only after the action commitment point has been reached. False alarms per hour are the ratio of the number of false alarms generated during non-risk operation periods to the corresponding non-risk operation duration. End-to-end latency is the time between a set of multimodal construction data meeting processing conditions and the completion of the output of the corresponding construction safety early warning information. The state truth trajectory, event labels, and action commitment point truth values ​​determined offline based on the state truth trajectory used in the experiment are only used for evaluation metric statistics and are not necessary inputs for the operation of the method in this application embodiment.

[0181] Figure 6 A comparative experimental simulation diagram showing the distribution of early warning lead time for different early warning methods under the condition of action commitment point alignment is presented.

[0182] like Figure 6As shown, using the action commitment point determined offline based on the simulation state truth value as the time zero point, the warning lead time of static appearance detection methods, video temporal classification methods, trajectory intersection prediction methods, 3D interactive recognition methods, and the proposed method are compared. The static appearance detection method, video temporal classification method, trajectory intersection prediction method, and 3D interactive recognition method in the figure correspond to the aforementioned YOLO-PPE, CNN-LSTM, Social-LSTM, and 3D-HOI, respectively. The vertical axis represents the warning lead time relative to the action commitment point, and the horizontal axis represents different warning methods; the horizontal dashed line represents the action commitment point, marked as the "irreversible commitment point" in the figure. A positive warning lead time indicates that the warning is output before the action commitment point is reached; a negative warning lead time indicates that the warning is output after the action commitment point is reached.

[0183] The warning lead time for each method is represented by violin plots, box plots, and sample scatter plots. The violin plot represents the distribution density of the warning lead time, the box plot represents the median and interquartile range, and the sample scatter plot represents the warning lead time corresponding to each risk event sample. The warning lead times of the static appearance detection method, video temporal classification method, trajectory intersection prediction method, and 3D interactive recognition method are all partially distributed after the action commitment point, with corresponding late reporting rates of 34%, 24%, 15%, and 8%, respectively. The warning lead time of the method presented in this paper is all distributed before the action commitment point, with a corresponding late reporting rate of 0%, a median of approximately 2.3 seconds, and a smaller interquartile range and overall distribution range than the comparative methods.

[0184] Under this simulation test set, the method of this application's embodiments forms a set of safety handling actions corresponding to each future moment by deduce the evolution sequence of the swapping state and the movement range of the dual-branch hoisting. The action commitment point is determined at the moment when the set of safety handling actions for the normally operating branch first changes from a non-empty state to an empty state. Then, combined with the irreversible time margin, the warning trigger time is determined before the action commitment point. This ensures that the warning is output within the time range where executable safety handling actions still exist, reducing the occurrence of late reports after the action commitment point has arrived. The state truth values, risk event labels, and action commitment point truth values ​​used in the figure are only for experimental evaluation and are not necessary inputs for the operation of the method of this application's embodiments.

[0185] Figure 7 A schematic diagram of the comparative experimental simulation results of different methods under the combined disturbance of visual occlusion and sway angle of the load is shown.

[0186] like Figure 7As shown, the left subplot corresponds to the 3D interactive recognition method, and the right subplot corresponds to the method presented in this paper. The horizontal axis of both subplots represents the duration of visual occlusion, ranging from 0 to 10 seconds; the vertical axis represents the swing angle of the load, ranging from 0 to 30 degrees. The contour lines and color gradients represent the false alarm rate under different joint perturbation conditions, with colors from light to dark corresponding to low to high false alarm rates. The color scale on the right indicates the false alarm rate range of 0% to 100%. The two methods are compared under the same preset false alarm control conditions.

[0187] The false negative rate of the 3D interactive recognition method increases continuously with the duration of visual occlusion and the swing angle of the load, and its high false negative rate contour lines extend to the middle of the parameter domain. The method in this paper maintains a light-colored, low false negative rate distribution in most parameter domains, and the false negative rate contour lines of 20% and above are mainly located in the region where both the duration of visual occlusion and the swing angle of the load increase; when both perturbations approach the upper limit of the scanning range, the false negative rate increases further and forms a high false negative rate region in the upper right corner.

[0188] The above results demonstrate that, under this combined disturbance scanning condition, the method in this application determines the operational status by combining construction worker video data, double hook status data, hoisting equipment operation data, and operational area structural data. It also forms a set of safety response actions by combining hook-changing status deduction and prediction of the hoisting motion range including uncertainty expansion. This allows for early warning judgments to be completed even when some visual observations are obstructed, by utilizing other modal data. When extreme visual obstruction and large hoisting sway occur simultaneously, the false negative rate of the proposed method increases, reflecting the applicability limits of this method under conditions of continuous visual information loss and increased uncertainty in hoisting motion.

[0189] Figure 8 This diagram illustrates a comparative simulation of the performance of different methods in multidimensional engineering deployment. The horizontal axis represents end-to-end system latency, and the vertical axis represents the area under the event-level precision-recall curve. Different colored bubbles represent static appearance detection, video temporal classification, trajectory intersection prediction, 3D interactive recognition, and the method presented in this paper, respectively. The area of ​​the bubble represents the number of false alarms per hour for the corresponding method; a larger bubble area indicates a higher number of false alarms per hour. The bubble area legend in the diagram shows the bubble areas corresponding to 2, 6, and 10 false alarms per hour, respectively.

[0190] Figure 8The shaded area in the upper left corner represents the high-performance, low-latency deployment zone, corresponding to a performance combination where the area under the event-level precision-recall curve falls within the range defined by the shaded area, and the end-to-end system latency falls within the latency range defined by the shaded area. The bubble corresponding to the proposed method is located within this area, with an event-level precision-recall curve area of ​​approximately 0.96, an end-to-end system latency of approximately 90 milliseconds, and a false positive rate of approximately 2 times per hour. The bubbles corresponding to the comparative methods are located to the lower right or below the proposed method, and their bubble areas are all larger than those of the proposed method.

[0191] Within the scope of this simulation test and the compared methods, the proposed method simultaneously achieved the highest area under the event-level precision-recall curve, the lowest end-to-end system latency, and the fewest false alarms per hour. Therefore, its performance was not dominated by any of the comparative methods in these three evaluation dimensions. This result demonstrates that the method in this application, through multimodal operation state determination, swapping state evolution deduction, safety action screening, and pre-action commitment point warning, maintains event-level warning discrimination capability while reducing warning output latency and on-site false alarm burden.

[0192] When some multimodal construction data becomes abnormal, the system adopts a conservative approach to update relevant states and movement ranges: when a camera fails, the system expands the human space range and reduces the uncertainty of the hoist position prediction; when smart hook data is lost, the system reduces the reliability of the corresponding hook connection status observation based on the data time difference; when the hoisting equipment operation data exceeds the allowable update time, the system expands the hoist movement range of the normal operation branch according to the allowable movement rate and determines the deceleration operation branch as an unrecoverable state; when the anchor point position recorded in the structural data of the work area is consistently inconsistent with the on-site observation position, the corresponding anchor point is excluded from the reachable anchor points used in the generation of candidate disposal actions. The above anomaly handling maintains the conditions of stable support, effective protective connection, protective continuity, and spatial separation between the human body and the hoist, thus shrinking the set of safety disposal actions caused by a single data source anomaly and maintaining the safety screening conditions for candidate disposal actions.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of combined actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Secondly, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application. In the above embodiments, the descriptions of each embodiment have their own emphasis; for parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0194] Figure 9A structural block diagram of an example of a construction safety early warning system based on multimodal behavioral reasoning according to an embodiment of this application is shown.

[0195] like Figure 9 As shown, the construction safety early warning system 900 based on multimodal behavioral reasoning includes a multimodal state perception unit 910, a swapping state inference unit 920, a candidate disposal generation unit 930, a safety disposal screening unit 940, and a commitment point early warning unit 950.

[0196] The multimodal state perception unit 910 is used to acquire multimodal construction data of the target construction personnel in the target edge work area, and determine the working state and hoisting movement state of the target construction personnel based on the multimodal construction data; the multimodal construction data includes construction personnel video data, double hook state data, hoisting equipment operation data and work area structure data; the working state includes human posture, support state and effective protective connection state of each hook, and the effective protective connection state is used to characterize whether the hook is connected to the anchor point that meets the fall protection bearing conditions.

[0197] The hook-up state deduction unit 920 is used to identify multiple local actions and their occurrence sequence in the hook-up process based on the multimodal construction data, determine the operation state changes corresponding to each local action, and perform state deduction for the subsequent hook-up process based on the occurrence sequence and the operation state changes to generate a hook-up state evolution sequence.

[0198] The candidate disposal generation unit 930 is used to predict the range of motion of the hoist in each future time in a preset future time domain based on the hoist motion state, and to determine the predicted operation state corresponding to each future time based on the changeover state evolution sequence; for each predicted operation state, multiple candidate disposal actions that meet the preset human movement reachability conditions are generated in combination with the operation area structure data.

[0199] The safety handling screening unit 940 is used to deduce the corresponding candidate handling actions with each of the predicted operation states as the starting state, and obtain the post-handling operation state of each candidate handling action; and to form a set of safety handling actions corresponding to each future time by the candidate handling actions in the post-handling operation state that meet the preset stability conditions, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-handling operation state does not intersect with the hoisting movement range of the corresponding future time.

[0200] The commitment point early warning unit 950 is used to determine the future moment when the set of safety handling actions first changes from a non-empty state to an empty state as the action commitment point, determine an early warning trigger time earlier than the action commitment point based on the action commitment point, and generate construction safety early warning information at the early warning trigger time.

[0201] In some embodiments, this application provides a non-volatile computer-readable storage medium storing one or more programs including execution instructions. The execution instructions can be read and executed by electronic devices (including but not limited to computers, servers, or network devices) to perform the steps of any of the above-described construction safety early warning methods based on multimodal behavioral reasoning.

[0202] In some embodiments, this application also provides a computer program product, the computer program product including a computer program stored on a non-volatile computer-readable storage medium, the computer program including program instructions, which, when executed by a computer, cause the computer to perform the steps of any of the above-described construction safety early warning methods based on multimodal behavioral reasoning.

[0203] In some embodiments, this application also provides an electronic device, comprising: at least one processor, and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of a construction safety early warning method based on multimodal behavioral reasoning.

[0204] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0205] The electronic devices in this application can exist in various forms, including but not limited to: mobile communication devices, ultra-mobile personal computer devices, portable entertainment devices, or other airborne electronic devices with data interaction functions.

[0206] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0208] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A construction safety early warning method based on multimodal behavioral reasoning, characterized in that, The method includes: Acquire multimodal construction data of target construction personnel within the target edge work area, and determine the work status and hoisting movement status of the target construction personnel based on the multimodal construction data; the multimodal construction data includes construction personnel video data, double hook status data, hoisting equipment operation data, and work area structure data; the work status includes human posture, support status, and effective protective connection status of each hook, the effective protective connection status being used to characterize whether the hook is connected to an anchor point that meets the fall protection bearing conditions; Based on the multimodal construction data, identify multiple local actions and their occurrence sequence during the double hook replacement process, determine the corresponding changes in the work state for each local action, and perform state deduction for the subsequent replacement process based on the occurrence sequence and the changes in the work state to generate a replacement state evolution sequence. Based on the hoisting motion state, predict the range of motion of the hoisting at each future moment in the preset future time domain, and determine the predicted operation state corresponding to each future moment based on the swapping state evolution sequence; for each predicted operation state, generate multiple candidate handling actions that meet the preset human movement reachability conditions in combination with the operation area structure data. Each candidate action is derived from the predicted operation state as the starting state to obtain the post-operation state of each candidate action; the candidate actions in the post-operation state that satisfy the preset stability condition, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-operation state does not intersect with the hoisting movement range of the corresponding future time are formed into a set of safe action actions corresponding to each future time. The future moment when the set of safety action actions first changes from a non-empty state to an empty state is determined as the action commitment point. Based on the action commitment point, an early warning trigger time earlier than the action commitment point is determined, and construction safety early warning information is generated at the early warning trigger time.

2. The method according to claim 1, characterized in that, Determining the working status of the target construction personnel based on the multimodal construction data includes: For each hook, the visual hooking relationship, door lock status, anchor point identity and load-bearing capacity, test pull micro-tension waveform, and rope geometry are extracted. The candidate connection states of the hooks are divided into unconnected state, pseudo-connected state, effective slack state, and effective standby state. Among them, the unconnected state indicates that the hook is separated from the anchor point; the pseudo-connected state indicates that the hook is in a hooked or closed state but has not formed a connection that meets the fall protection load-bearing conditions; the effective slack state indicates that the hook is connected to the anchor point that meets the fall protection load-bearing conditions and the rope is slack; and the effective standby state indicates that the hook forms a force path that can immediately bear the fall protection load. The unconnected state and the pseudo-connected state are determined as feasible connection states, and the effective relaxation state and the effective standby state are added to the feasible connection states only when the anchor point identity belongs to the preset qualified anchor point set and the bearing capacity meets the fall protection bearing conditions. The various observations are compared with the preset observation features corresponding to each of the feasible connection states to obtain the observation bias, and the comprehensive state bias is determined based on the observation bias. The state probability of each of the feasible connection states is determined based on the comprehensive state bias, so that the state probability corresponding to the feasible connection state with the smaller the comprehensive state bias is larger, the sum of the state probabilities of each of the feasible connection states is one, and the state probability of the candidate connection states other than the feasible connection states is zero. The sum of the probabilities of the effective relaxation state and the effective standby state is determined as the effective connection probability. The probability of the effective standby state is determined as the probability of immediate load carrying. When the effective connection probability reaches a preset connection probability threshold, the corresponding hook is determined to be in the effective protection connection state.

3. The method according to claim 2, characterized in that, The step involves identifying multiple local actions and their timing during the double-hook replacement process based on the multimodal construction data, determining the corresponding operational state changes for each local action, and performing state deduction for the subsequent replacement process based on the timing and operational state changes to generate a replacement state evolution sequence, including: Based on the spatial relationship between the human hand, foot, and center of gravity and the hook, anchor point, and support area, and combined with the door lock status and the test pull micro-tension waveform, the local actions of reaching out to move to the new anchor point, closing the new hook, test pull confirmation, releasing the original hook, shifting the supporting foot, and straddling the torso are identified. The time interval of the local actions that have occurred is determined, and the time interval of the incomplete local actions and the completion time are predicted. Based on the changes in human posture, support state, effective connection probability, and immediate bearing probability before and after each of the aforementioned local actions, an action state transition record is generated. The occurrence sequence is determined based on the occurrence time interval and completion time of each local action. The action sequence constraint is determined when the new hook closure and test pull confirmation are earlier than the original hook release, and when the support foot is transferred, the probability of at least one hook being able to bear the load immediately reaches a preset load probability threshold. Based on the action sequence constraint, the state transition records of each action are divided into normal state transition and abnormal state transition. Starting with the operation status, each action status transition record is called sequentially according to the occurrence sequence to update the operation status during the subsequent reloading process, and the operation status changes caused by the abnormal status transition are retained to generate the reloading status evolution sequence.

4. The method according to claim 3, characterized in that, The step of predicting the range of motion of the load at various future times within a preset future time domain based on the load's motion state includes: The hoisting equipment is configured with a normal operation branch that maintains its current operating state and a deceleration operation branch that performs deceleration. Based on the hoisting motion state and the hoisting equipment operating data, the nominal pose, position prediction uncertainty, and motion rate of the hoisting at each future moment are predicted under each operation branch. The nominal pose of the deceleration operation branch is determined based on the hoisting equipment's allowable deceleration, braking establishment time, and the remaining swing state of the hoisting after braking. The maximum data time difference is determined based on the data time difference of various multimodal construction data relative to the current time. The remaining execution delay of the current running instruction is determined as the branch control delay of the normal operation branch, and the communication and execution delay of the deceleration control instruction is determined as the branch control delay of the deceleration operation branch. The nominal occupancy range of the hoist is determined based on the hoist geometry and the nominal pose, and the uncertainty of the nominal occupancy range of the hoist is expanded based on the uncertainty of the position prediction. Based on the displacement formed by the motion rate within a time period determined by the maximum data time difference and the branch control delay of the corresponding operating branch, the nominal occupied range of the load after uncertainty expansion is extended by time delay, and the range after time delay expansion is perturbed by combining the sling elasticity, environmental disturbance and residual swing state, so as to obtain the motion range of the load under each operating branch at each future time. The range of motion under the normal operation branch is taken as the range of motion of the hoisting equipment at each future moment, and the range of motion under the deceleration operation branch is used to determine whether the hoisting equipment can maintain or restore the safe handling conditions after performing the deceleration operation.

5. The method according to claim 4, characterized in that, The candidate handling actions that satisfy preset stability conditions in the post-handling operation state, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-handling operation state does not intersect with the hoisting movement range at the corresponding future time are formed into a set of safe handling actions for each future time, including: For each future moment's candidate action, the post-action work status is used to determine the post-action human body space range, the effective connection probability of each hook, and the probability of immediate load bearing. The support stability margin is determined based on the directional distance between the projection of the human body's center of gravity and the boundary of the support area. The predicted fall direction is determined based on the post-treatment work status and the structural data of the work area. The protective contribution is determined based on the geometric relationship between the hook, anchor point and rope relative to the predicted fall direction and the probability of the corresponding hook being able to bear the load immediately. The protective contribution of each hook is aggregated according to the parallel protection relationship of forming a fall protection force path by at least one hook, so as to obtain the protection continuity characterizing the degree of continued existence of the effective fall protection force path; For each of the aforementioned operating branches, candidate actions that simultaneously satisfy the preset human body movement attainability conditions, the support stability margin not less than the preset stability margin lower limit, the effective connection probability of at least one hook reaching the preset connection probability threshold, the protection continuity not less than the preset protection continuity lower limit, and the human body space range after the action does not intersect with the hoisting movement range of the corresponding operating branch, constitute the set of safe handling actions for the corresponding operating branch at the corresponding future time. Among them, the set of safe handling actions for the normal operating branch is the set of safe handling actions; the set of safe handling actions for the deceleration operating branch is used to determine whether the deceleration operation can restore the safe handling conditions.

6. The method according to claim 5, characterized in that, The future moment when the set of safety action actions first changes from a non-empty state to an empty state is determined as the action commitment point. The determination of an early warning trigger time earlier than the action commitment point based on the action commitment point includes: The future moment when the set of safety handling actions for the normally operating branch first changes from a non-empty state to an empty state is determined as the action commitment point; For each safety action at each future moment and in each operational branch, determine three types of remaining safety time before the load sweep, protection continuity, and support stability margin reach their respective safety boundaries, as well as three types of corresponding action time for completing load avoidance, restoring at least one hook to the effective standby state, and restoring stable support; The minimum difference between the remaining safety time and the corresponding handling time in the three categories is subtracted from the system response delay corresponding to system calculation, communication transmission and early warning output to obtain the recoverable time margin of the corresponding action; the maximum recoverable time margin under the same future time and running branch is determined as the irreversible time margin; when the set of safety handling actions is empty, the irreversible time margin is determined as zero; running branches with an irreversible time margin greater than zero are determined as recoverable states, otherwise they are determined as irreversible states. A positive early warning protection margin is determined based on the preset safety quantile of the personnel early warning response time. The moment when the irreversible time margin of the normal operation branch between the current moment and the action commitment point is first not greater than the early warning protection margin is determined as the margin trigger moment. The earlier time is selected as the candidate warning time from the margin trigger time and the time obtained by advancing the warning protection margin relative to the action commitment point; when the candidate warning time is earlier than the current time, the current time is determined as the warning trigger time, otherwise the candidate warning time is determined as the warning trigger time.

7. The method according to claim 6, characterized in that, The generation of construction safety early warning information at the time the early warning is triggered includes: Obtain the set of safety handling actions and recovery status of the normal operation branch and the deceleration operation branch at the time of the warning trigger, and obtain the irreversible time margin corresponding to the operation branch in the recoverable state; When the normal operation branch is in a recoverable state, the candidate action with the largest recoverable time margin is selected from the set of safety handling actions of the normal operation branch as the target handling action, and personnel action adjustment information is generated according to the movement direction of the target handling action. When both of the operating branches are in a recoverable state, and the irreversible time margin increased by the deceleration operating branch relative to the normal operating branch reaches a preset recovery gain threshold, a deceleration command for the hoisting equipment is generated. When the normal operation branch is in an unrecoverable state and the deceleration operation branch is in a recoverable state, the candidate action with the largest recoverable time margin is selected from the set of safety handling actions of the deceleration operation branch as the target handling action, and a forced deceleration command for controlling the hoisting equipment to execute the deceleration operation corresponding to the deceleration operation branch and personnel action adjustment information corresponding to the target handling action are generated. When both of the aforementioned operating branches are in an unrecoverable state, an emergency stop command for the hoisting equipment, an on-site audible and visual alarm command, and a manual handling prompt are generated; at least one of the personnel action adjustment information, the hoisting equipment deceleration command, the forced deceleration command, the hoisting equipment emergency stop command, the on-site audible and visual alarm command, and the manual handling prompt is written into the construction safety early warning information.

8. A construction safety early warning system based on multimodal behavioral reasoning, characterized in that, The system includes: A multimodal state perception unit is used to acquire multimodal construction data of target construction personnel within the target edge work area, and determine the working state and hoisting motion state of the target construction personnel based on the multimodal construction data; the multimodal construction data includes construction personnel video data, double hook status data, hoisting equipment operation data, and work area structure data; the working state includes human posture, support status, and effective protective connection status of each hook, the effective protective connection status being used to characterize whether the hook is connected to an anchor point that meets the fall protection bearing conditions; The hook-changing state deduction unit is used to identify multiple local actions and their occurrence sequence in the hook-changing process based on the multimodal construction data, determine the operation state change corresponding to each local action, and perform state deduction for the subsequent hook-changing process based on the occurrence sequence and the operation state change to generate a hook-changing state evolution sequence. The candidate disposal generation unit is used to predict the movement range of the hoist in each future time in a preset future time domain based on the hoist movement state, and to determine the predicted operation state corresponding to each future time based on the swapping state evolution sequence; for each predicted operation state, multiple candidate disposal actions that meet preset human movement reach conditions are generated in combination with the operation area structure data. The safety handling screening unit is used to deduce the corresponding candidate handling actions based on each of the predicted operation states as the starting state, and obtain the post-handling operation state of each candidate handling action; and to form a set of safety handling actions corresponding to each future time by the candidate handling actions in the post-handling operation state that meet the preset stability conditions, at least one hook is in an effective protective connection state, and the human space range corresponding to the human posture in the post-handling operation state does not intersect with the hoisting movement range of the corresponding future time. The commitment point early warning unit is used to determine the future moment when the set of safety action actions first changes from a non-empty state to an empty state as the action commitment point, determine an early warning trigger time earlier than the action commitment point based on the action commitment point, and generate construction safety early warning information at the early warning trigger time.

9. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor, characterized in that the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the steps of the method as described in any one of claims 1-7.

10. A storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps of the method as described in any one of claims 1-7.

Images