A visual recognition method and system for high-altitude operation behavior in human-machine collaboration

Abstract

This application provides a visual recognition method and system for high-altitude operation behavior based on human-machine collaboration. The method includes: acquiring multi-source streams from a high-altitude construction site; the multi-source streams include a first stream corresponding to fixed equipment and a second stream corresponding to a drone; decoding and aligning the first and second streams to obtain a unified frame sequence; based on the unified frame sequence, performing personnel detection and scene element object generation to obtain a personnel trajectory sequence and scene element objects; calculating hazard boundary relationships based on the personnel trajectory sequence and scene element objects to determine a target relationship sequence; the target relationship sequence represents the relationship sequence between people and boundaries; fusing the personnel trajectory sequence and target relationship sequence to obtain candidate events; judging the candidate events to determine the final recognition result; and visually displaying the final recognition result. This solution can improve the accuracy of identifying safety hazards in high-altitude operations.

Description

Technical Field

[0001] This application relates to the technical field of civil engineering, specifically to a visual recognition method and system for high-altitude operation behavior oriented towards human-machine collaboration. Background Technology

[0002] Working at heights is one of the high-risk types of work on construction sites, commonly seen in scaffolding operations, exterior facade construction platforms, work near edges and openings, roof work, and maintenance of equipment at heights. These work scenarios simultaneously possess both "high risk" and "high site complexity": on the one hand, the dangerous space boundaries for working at heights are clearly defined (edges, openings, cantilever platform edges, etc.), and the consequences are severe if personnel approach or cross these boundaries; on the other hand, there are numerous obstructions and interferences on site (scaffolding members, safety nets, material stacks, personnel crossing paths), and the work activities have a clear temporal sequence (e.g., actions such as approaching edges, stopping, crossing, and climbing are usually continuous segments), making it difficult for single-frame recognition or solutions relying solely on static images to operate stably. Traditional safety management relies mainly on manual inspections, which suffers from discontinuous coverage, numerous blind spots, high verification costs, and unstructured records. Summary of the Invention

[0003] This application aims to provide a visual recognition method and system for high-altitude operation behavior based on human-machine collaboration, which can improve the accuracy of identifying safety hazards in high-altitude operations.

[0004] The technical solution of this application is implemented as follows:

[0005] In a first aspect, embodiments of this application provide a visual recognition method for high-altitude work behavior oriented towards human-machine collaboration, the method comprising:

[0006] Acquire multi-source bitstreams from the high-altitude construction site; wherein, the multi-source bitstreams include a first bitstream corresponding to the fixed equipment and a second bitstream corresponding to the drone;

[0007] Decoding and alignment processes are performed on the first and second bitstreams to obtain a unified frame sequence.

[0008] Based on the unified frame sequence, personnel detection and scene element object generation are performed to obtain personnel trajectory sequence and scene element objects;

[0009] Based on the personnel trajectory sequence and the scene element objects, the dangerous boundary relationship is calculated, and the target relationship sequence is determined; wherein, the target relationship sequence represents the relationship sequence between people and boundaries;

[0010] Based on the personnel trajectory sequence and the target relationship sequence, the two are fused to obtain candidate events;

[0011] Based on the candidate events, an analysis is performed to determine the final identification result; and the final identification result is then visualized.

[0012] In the above scheme, the step of decoding and aligning the first bitstream and the second bitstream to obtain a unified frame sequence includes:

[0013] Decode the first bitstream and the second bitstream, and determine the system time and the display timestamp obtained from decoding each frame of each bitstream in the first bitstream and the second bitstream, respectively.

[0014] The time offset is determined based on the displayed timestamp and the system time;

[0015] Based on the display timestamp and the time offset, a unified timestamp for each frame of each bitstream in the first bitstream and the second bitstream is determined respectively;

[0016] Based on the unified timestamp, the first bitstream and the second bitstream are output at a unified frame rate to determine a unified frame sequence; wherein, the unified frame sequence includes a video source number, a video increment signal, a display timestamp, a unified timestamp, an image matrix, and an additional information dictionary.

[0017] In the above scheme, the step of performing personnel detection and scene element object generation based on the unified frame sequence to obtain personnel trajectory sequences and scene element objects includes:

[0018] For each frame in the unified frame sequence, perform multi-person operation personnel detection to obtain a personnel detection set for each frame; and perform non-maximum suppression on each detection box in the personnel detection set for each frame to obtain the final personnel detection set for each frame.

[0019] Based on the final personnel detection set for each frame, continuous trajectory generation is performed to determine the personnel trajectory sequence; wherein, the personnel trajectory sequence includes personnel landing point pixels;

[0020] High-altitude operation risk clue detection is performed on each frame in the unified frame sequence to obtain the clue detection set for each frame;

[0021] Based on the clue detection set of each frame, scene element objects are generated and the scene element objects are determined.

[0022] In the above scheme, the scene element objects include adjacent polylines and opening polygons;

[0023] The step of calculating hazard boundary relationships and determining target relationship sequences based on the personnel trajectory sequence and the scene element objects includes:

[0024] Determine the transformation matrix from the pixel plane to the world plane for each frame in the unified frame sequence;

[0025] Based on the transformation matrix, the pixel of the landing point of the person in the trajectory sequence is transformed to obtain the final pixel of the landing point of the person in the world coordinate plane.

[0026] Based on the final landing point pixel and the adjacent edge polyline, calculate the first shortest distance value between the person and the adjacent edge; and based on the final landing point pixel and the opening polygon, determine the second distance value between the person and the opening.

[0027] Based on the first shortest distance value and the second distance value, smoothing and trend calculations are performed to determine the first distance trend value and the second distance trend value;

[0028] The target relationship sequence is determined based on the first shortest distance value, the second distance value, the first distance trend value, and the second distance trend value.

[0029] In the above scheme, determining the transformation matrix from the pixel plane to the world plane for each frame in the unified frame sequence includes:

[0030] Obtain the pixel coordinates of any frame in the unified frame sequence, and the world coordinates of a pre-set marker point; then, perform calculations using the pixel coordinates and the world coordinates to determine the transformation matrix; or,

[0031] Obtain the camera intrinsic parameters, as well as the rotation matrix and translation vector in the world coordinate system; and determine the transformation inverse matrix using the camera intrinsic parameters, the rotation matrix, and the translation vector, and perform inverse operation on the transformation inverse matrix to determine the transformation matrix; wherein, the transformation inverse matrix is ​​the matrix that transforms from the world plane to the pixel plane.

[0032] In the above scheme, the step of fusing the personnel trajectory sequence and the target relationship sequence to obtain candidate events includes:

[0033] Based on the personnel trajectory sequence and the target relationship sequence, the event type, event segment index, key frame index, and triggering basis are determined; wherein, the key frame index corresponds to the first frame, the last frame, and the smallest frame in the event segment; the event type includes hole entry type, edge buffer zone continuous type, and edge approach type.

[0034] Based on the event fragment index and the keyframe index, a risk assessment is performed to determine a risk score;

[0035] Candidate events are determined using the event type, the event fragment index, the keyframe index, the triggering criteria, and the risk score.

[0036] In the above scheme, the step of analyzing the candidate events to determine the final identification result includes:

[0037] Based on the candidate events, the first identification result of high-altitude operations is determined through risk scoring and preset risk thresholds;

[0038] Receive the second identification result of high-altitude operations; wherein, the second identification result is the identification result made by the safety officer for the candidate event;

[0039] Based on the first identification result and the second identification result, a comprehensive analysis is performed to determine the final identification result.

[0040] Secondly, embodiments of this application provide a visual recognition system for high-altitude work behavior oriented towards human-machine collaboration. This system includes: an acquisition module, a decoding and alignment module, a detection and generation module, and a determination module.

[0041] The acquisition module is used to acquire multi-source streams from the high-altitude operation construction site; wherein, the multi-source streams include a first stream corresponding to the fixed equipment and a second stream corresponding to the drone;

[0042] The decoding alignment module is used to perform decoding and alignment processing on the first bitstream and the second bitstream to obtain a unified frame sequence;

[0043] The detection and generation module is used to perform personnel detection and scene element object generation based on the unified frame sequence, so as to obtain personnel trajectory sequence and scene element object;

[0044] The determining module is used to calculate the dangerous boundary relationship and determine the target relationship sequence based on the personnel trajectory sequence and the scene element object; wherein the target relationship sequence represents the relationship sequence between people and the boundary; based on the personnel trajectory sequence and the target relationship sequence, perform fusion to obtain candidate events; based on the candidate events, perform analysis to determine the final identification result; and visualize the final identification result.

[0045] Thirdly, embodiments of this application provide a visual recognition device for high-altitude work behavior in a human-machine collaborative manner, comprising: a processor and a memory; wherein,

[0046] The memory is used to store computer programs;

[0047] The processor is configured to call and run the computer program from the memory to perform the method as described in the first aspect.

[0048] Fourthly, embodiments of this application provide a computer-readable storage medium storing executable instructions for causing a processor to perform the method described in the first aspect.

[0049] This application provides a visual recognition method and system for high-altitude operation behavior oriented towards human-machine collaboration. The method includes: acquiring multi-source code streams from a high-altitude operation construction site; wherein the multi-source code streams include a first code stream corresponding to fixed equipment and a second code stream corresponding to a drone; decoding and aligning the first and second code streams to obtain a unified frame sequence; based on the unified frame sequence, performing personnel detection and scene element object generation to obtain a personnel trajectory sequence and scene element objects; calculating dangerous boundary relationships based on the personnel trajectory sequence and the scene element objects to determine a target relationship sequence; wherein the target relationship sequence represents the relationship sequence between people and boundaries; fusing the personnel trajectory sequence and the target relationship sequence to obtain candidate events; performing analysis based on the candidate events to determine the final recognition result; and visually displaying the final recognition result. In the above scheme, acquiring multi-source code streams from a high-altitude operation construction site can avoid blind spots caused by a single device, thereby improving the visual coverage of high-altitude operation risk areas. The first and second bitstreams are decoded and aligned to obtain a unified frame sequence. Alignment reduces the time difference between different bitstreams, thus improving the accuracy of the unified frame sequence. Based on the unified frame sequence, personnel detection and scene element object generation are performed to obtain personnel trajectory sequences and scene element objects. Based on the personnel trajectory sequences and scene element objects, hazard boundary relationships are calculated to determine target relationship sequences. These sequences are then fused to obtain candidate events. This allows for the selection of potentially unsafe candidate events from multiple videos, improving the efficiency of identifying safety hazards in high-altitude operations. By analyzing the candidate events, the final identification result is determined, improving the accuracy of identifying safety hazards in high-altitude operations. The final identification result is visualized, providing on-site verification and evidence entry points to further validate the accuracy of identifying safety hazards in high-altitude operations. Attached Figure Description

[0050] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the specification, serve to explain the technical solutions of this application. Obviously, the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.

[0051] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0052] Figure 1 This is an optional flowchart illustrating a visual recognition method for high-altitude operations in a human-machine collaborative manner, provided in an embodiment of this application.

[0053] Figure 2 A schematic diagram of the structure of a visual recognition system for high-altitude operations oriented towards human-machine collaboration, provided in an embodiment of this application;

[0054] Figure 3 This is a structural schematic diagram of a visual recognition device for high-altitude operations in a human-machine collaborative manner, provided as an embodiment of this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the specific technical solutions of this application will be further described in detail below with reference to the accompanying drawings of the embodiments of this application. The following embodiments are used to illustrate this application, but are not intended to limit the scope of this application.

[0056] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. The terminology used in this application is for the purpose of describing embodiments of this application only and is not intended to be limiting of this application.

[0057] In the following description, references to "some embodiments," "this embodiment," "this application embodiment," and examples, etc., describe a subset of all possible embodiments. However, it is understood that "some embodiments" may be the same subset or different subset of all possible embodiments and may be combined with each other without conflict.

[0058] If the application documents contain similar descriptions such as "first / second", the following explanation shall be added: In the following description, the terms "first / second / third" are used only to distinguish similar objects and do not represent a specific order of objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0059] This application provides a visual recognition method for high-altitude operation behavior in a human-machine collaborative manner. Figure 1This is an optional flowchart illustrating a visual recognition method for high-altitude work behavior in a human-machine collaborative manner, provided as an embodiment of this application. Figure 1 The steps shown are explained.

[0060] S101. Obtain the multi-source stream from the high-altitude operation construction site; wherein, the multi-source stream includes the first stream corresponding to the fixed equipment and the second stream corresponding to the drone.

[0061] In some embodiments of this application, a first stream of data from a high-altitude construction site is acquired using a fixed device, and this first stream contains construction workers. A second stream of data from the high-altitude construction site is acquired using a drone, and this second stream also contains construction workers.

[0062] In some embodiments of this application, the visual recognition method for high-altitude operation behavior oriented towards human-machine collaboration is adapted to building construction scenarios.

[0063] In some embodiments of this application, the visual recognition method for high-altitude operation behavior oriented towards human-machine collaboration is adapted to the visual recognition system for high-altitude operation behavior oriented towards human-machine collaboration.

[0064] S102. Decode and align the first and second bitstreams to obtain a unified frame sequence.

[0065] In some embodiments of this application, the first bitstream and the second bitstream are decoded to determine the system time and the display timestamp obtained from decoding each frame of each bitstream in the first bitstream and the second bitstream, respectively; a time offset is determined based on the display timestamp and the system time; a unified timestamp for each frame of each bitstream in the first bitstream and the second bitstream is determined based on the display timestamp and the time offset, respectively; and a unified frame rate is output for the first bitstream and the second bitstream based on the unified timestamp to determine a unified frame sequence; wherein, the unified frame sequence includes a video source number, a video increment signal, a display timestamp, a unified timestamp, an image matrix, and an additional information dictionary.

[0066] S103. Based on the unified frame sequence, perform personnel detection and scene element object generation to obtain personnel trajectory sequence and scene element object.

[0067] In some embodiments of this application, scene element objects include adjacent polylines and opening polygons.

[0068] In some embodiments of this application, multiple workers are detected in each frame of a unified frame sequence to obtain a set of detected workers for each frame; non-maximum suppression is applied to each detection box in the set of detected workers for each frame to obtain a final set of detected workers for each frame; based on the final set of detected workers for each frame, continuous trajectory generation is performed to determine a sequence of worker trajectories; wherein, the sequence of worker trajectories includes the pixels of worker landing points; high-altitude operation risk clue detection is performed in each frame of the unified frame sequence to obtain a set of clue detection for each frame; based on the set of clue detection for each frame, scene element objects are generated to determine scene element objects.

[0069] S104. Based on the personnel trajectory sequence and scene element objects, calculate the dangerous boundary relationship and determine the target relationship sequence; wherein, the target relationship sequence represents the relationship sequence between people and the boundary.

[0070] In some embodiments of this application, a transformation matrix is ​​determined for each frame in the unified frame sequence to transform from the pixel plane to the world plane; based on the transformation matrix, the landing point pixels of the personnel in the personnel trajectory sequence are transformed to obtain the final landing point pixels of the personnel in the world coordinate plane; based on the final landing point pixels of the personnel and the adjacent edge polyline, a first shortest distance value between the personnel and the adjacent edge is calculated; and based on the final landing point pixels of the personnel and the opening polygon, a second distance value between the personnel and the opening is determined; based on the first shortest distance value and the second distance value, smoothing and trend calculations are performed to determine a first distance trend value and a second distance trend value; based on the first shortest distance value, the second distance value, the first distance trend value, and the second distance trend value, a target relationship sequence is determined.

[0071] S105. Based on the personnel trajectory sequence and the target relationship sequence, perform fusion to obtain candidate events.

[0072] In some embodiments of this application, event types, event fragment indexes, keyframe indexes, and triggering criteria are determined based on personnel trajectory sequences and target relationship sequences. The keyframe index corresponds to the first frame, the last frame, and the smallest frame in the event fragment. Event types include entry into openings, continuous edge buffer zones, and edge approach. Risk assessment is performed based on the event fragment index and keyframe index to determine a risk score. Candidate events are determined using the event type, event fragment index, keyframe index, triggering criteria, and risk score.

[0073] S106. Based on the candidate events, conduct analysis to determine the final identification result; and then visualize the final identification result.

[0074] In some embodiments of this application, a first identification result for high-altitude operations is determined based on candidate events, through risk scoring and preset risk thresholds; a second identification result for high-altitude operations is received; wherein the second identification result is the identification result made by the safety officer for the candidate events; based on the first identification result and the second identification result, a comprehensive analysis is performed to determine the final identification result; and the final identification result is visualized.

[0075] For example, based on the obtained candidate events, the four actions of "display, adjudication, database entry, and update" are completed on a per-candidate-event basis. Each event, when pushed, carries the following information: event_id, event_type, source_id, region_id, track_id, t_start_ms, t_end_ms, risk_score, risk_level, boundary_version, param_version, keyframes (F_start, F_peak, F_end), and evidence_range (start_frame_no, end_frame_no). This allows for review and display without relying on frame-by-frame algorithm output. Specifically, event_id is a unique identifier for the event, used to distinguish different events; event_type is the event type, marking the business type to which the event belongs (e.g., EDGE_APPROACH, HOLE_ENTER, etc. for high-altitude operations); and source_id is a unique identifier for the data source, marking... The event originates from the original data source; `region_id` is the region identifier, marking the geographical or monitoring area where the event occurred; `track_id` is the target tracking identifier, used to associate continuous trajectory records of the same target; `t_start_ms` is the event start time; `t_end_ms` is the event end time; `risk_score` is the event risk score, used to quantify the event's risk level; `risk_level` is the event risk level, mapped from the risk score (e.g., HIGH, MEDIUM, LOW); `boundary_version` is the hazard boundary version identifier, marking the version of the hazard boundary rule used when the event was triggered; `param_version` is the algorithm parameter version identifier, marking the version of the algorithm parameters used when the event was triggered; `keyframes` (F_start, F_peak, F_end) is the set of event keyframes, containing three keyframes: F_start is the frame corresponding to the event start time; F_peak is the frame corresponding to the peak risk of the event or the moment of the key action, F_end is the frame corresponding to the end of the event, and evidence_range (start_frame_no, end_frame_no) is the frame range of the evidence segment, including: start_frame_no is the starting frame number of the evidence segment, and end_frame_no is the ending frame number of the evidence segment.When replaying evidence, extract evidence fragments using `t_clip0=t_start_ms-1500` and `t_clip1=t_end_ms+1500`, and overlay the personnel frame, personnel landing point, danger boundary, buffer zone outline, and trigger summary on the playback screen. The trigger summary directly reads the event field `TriggerSummary` (duration_ms, d_min, k_min, buffer_time_ms, hole_time_ms, helmet_ratio, belt_ratio, rope_ratio, vest_ratio). Here, `duration_ms` is the event duration, `d_min` is the minimum shortest distance from the personnel to the edge within the event time window, and `k_min` is the minimum distance between the personnel and the edge within the event time window. The minimum slope of the distance trend is displayed, with negative values ​​indicating that the person is continuously approaching the edge. `buffer_time_ms` represents the cumulative time the person spends within the edge buffer zone within the event time window. `hole_time_ms` represents the cumulative time the person spends in a stable state inside the hole area within the event time window. `helmet_ratio` represents the proportion of frames in the event time window where the person is wearing a helmet. `belt_ratio` represents the proportion of frames in the event time window where the person is wearing a safety belt. `rope_ratio` represents the proportion of frames in the event time window where the person is using a safety rope. `vest_ratio` represents the proportion of frames in the event time window where the person is wearing a safety vest. Two curves, `d_edge_s(t)` and `k_edge(t)`, are also displayed. The data points for these curves come from the RelItem, a relational sequence record for each frame within the event window.

[0076] Based on candidate events, the first identification result of high-altitude operation is determined by risk score and preset risk threshold. For example, if the preset risk threshold is 85 and the risk score is greater than 85, the first identification result is determined to be that high-altitude operation is unsafe. The safety officer makes only one decision on each candidate event to obtain the second identification result, which is either that high-altitude operation is unsafe or that high-altitude operation is safe.

[0077] The event database is organized into three tables for easy querying and tracing. The first table, CandidateEvent, stores the original structure fields of candidate events (including evidence index, keyframe, trigger summary, risk score, model version, boundary version, and parameter version). The second table, ReviewRecord, stores the assessment conclusions and reasons. The third table, AuditLog, records changes in event status (e.g., from UNREVIEWED to CONFIRMED or REJECTED). UNREVIEWED indicates no evaluation, CONFIRMED indicates confirmation, and REJECTED indicates rejection. Fields include at least event_id, action, operator, time_utc_ms, old_status, new_status, boundary_version, and param_version. Event_id represents the unique identifier of the event; action represents the operation; operator represents the operator; time_utc_ms represents the operation timestamp; old_status represents the status before the change; new_status represents the status after the change; boundary_version represents the boundary version effective at the time of the operation; and param_version represents the parameter version effective at the time of the operation. Parameters and boundaries are stored using version tables: the ParamSet table stores the version parameter as (source_id, region_id, event_type, param_version), and the BoundarySet table stores it as (source_id, region_id, boundary_version). The storage boundary geometry is defined by the following: source_id represents a unique identifier for the data source, region_id represents a unique identifier for the region, event_type represents the event type, param_version represents the parameter version, and boundary_version represents the boundary version. This allows any event to be retrieved via event_id, providing: evidence fragments, trigger sequences, analysis conclusions, and the parameter and boundary versions at the time of the event.

[0078] The backflow adaptation maintains a set of online effective parameters, ParamSet, based on (source_id, region_id, event_type). Here, source_id represents the unique identifier of the data source, region_id represents the unique identifier of the region, and event_type represents the event type. Parameter fields include D0 (base distance), D1 (dynamic distance), k0 (coefficient baseline), N_start (number of starting frames), N_app (number of appearing frames), N_end (number of ending frames), N_gap (frame interval), and L_smooth (smoothing coefficient). The system triggers an update using the "most recent 50 analyzed records" as the update window W. The selection rule for window W is to select the 50 most recent records whose review_result (analysis result) is not empty and matches the same (source_id, region_id, event_type) in reverse order of review_time_utc_ms (analysis timestamp). The statistics are first calculated based on the confirmation rate and rejection rate: p_confirm (confirmation rate) equals the number of CONFIRMED events within the window divided by 50, and p_reject (rejection rate) equals the number of REJECTED events within the window divided by 50. Then, the percentage of rejection reasons is calculated, using the number of rejected events (rejected_count) as the denominator. When rejected_count = 0, each percentage is directly recorded as 0; otherwise, r_boundary (rejection percentage due to boundary deviation) equals the number of BOUNDARY_BIAS events divided by rejected_count, r_visibility (rejection percentage due to visibility issues) equals the sum of OCCLUSION and VIEW_UNCLEAR events divided by rejected_count, and r_track (rejection percentage due to track anomalies) equals the number of TRACK_ID_SWITCH events divided by rejected_count.

[0079] Provide a spatial geometry object, RegionModel, for the region to which the event belongs. The AR device caches and loads RegionModel locally. RegionModel contains at least: region_id (unique identifier for the region), boundary_world (the sequence of points of the dangerous boundary polyline in the region coordinate system R, in meters), buffer_polygon_world (the sequence of polygon points of the buffer zone), hole_polygon_world (the sequence of polygon points of the hole entrance, which can be empty), and event_anchor_R (the 3D coordinates of the event anchor point in the region coordinate system R, in meters). Among them, event_anchor_R is used to stably place the event label in the AR scene, avoiding inconsistencies in anchor points caused by the AR device recalculating the nearest point.

[0080] The AR side first performs spatial alignment to establish the correspondence between the AR world coordinate system and the region coordinate system R. AprilTag is used as the alignment medium: an AprilTag label with a side length of 0.20 m is placed in the field area corresponding to region_id, and the label's pose in the region coordinate system R is saved in the system configuration. (4×4 homogeneous transformation matrix). The AR device detects the AprilTag in real time, obtaining the tag's pose in the camera coordinate system C. Simultaneously, the device's SLAM obtains the camera's pose in the AR world coordinate system A. The homogeneous transformation matrix from the regional coordinate system R to the AR world coordinate system A can be calculated using the above three factors. The calculation formula is as follows:

[0081]

[0082] in, This represents the inverse of the homogeneous transformation matrix. To reduce the impact of single-frame jitter on the alignment results, the values ​​detected in 30 consecutive frames are... Steady-state estimation of candidate values: Take the median for translation components, and calculate and normalize the mean of rotation components using quaternions to obtain the final alignment result. After successful alignment, the AR device saves the data. It then enters display mode; when subsequent labels are temporarily lost, the alignment result remains valid, and users can trigger a re-estimation through the "Realign" interaction.

[0083] After alignment, the AR side maps the region geometry and event anchor points from the region coordinate system R to the AR world coordinate system A and renders them. For any point in the region coordinate system... Its coordinates in the AR world coordinate system Calculate using the following formula:

[0084]

[0085] AR endpoints are configured to process each point in boundary_world (a dangerous boundary polyline in world coordinates) by... The boundary_A polyline is obtained through transformation. Similarly, buffer_polygon_world (the buffer zone polygon in world coordinates) and hole_polygon_world (the hole opening polygon in world coordinates) are transformed to obtain the point set of the corresponding polygon in coordinate system A. During rendering, the dangerous boundary polyline is drawn using a line rendering component, while the buffer zone and hole opening areas are generated into meshes using polygon triangulation and rendered with a semi-transparent material. To avoid performance fluctuations caused by repeated creation, the AR side generates only one region geometry object for each region_id, and controls its display and hiding through a display switch.

[0086] Event tags are displayed using an overlay of "spatial anchor points plus a panel facing the user". The event anchor point is directly provided by the analysis unit as event_anchor_R (event anchor point), and the AR unit uses... Calculate its position pos_event_A (event anchor point position) in the AR world coordinate system. The event label panel always faces the user's line of sight. The panel content displays event_type, time window (converted from t_start_ms (event start timestamp) and t_end_ms (event end timestamp) to a local time string), risk_level, track_id (target unique identifier), and a trigger summary. The summary content is directly generated by trigger_summary, for example, "Minimum distance d_min=0.7 m, approaching trend k_min=-0.32 m / s, duration 3.2 s". The AR side provides filtering capabilities, with filtering conditions including region_id, event_type, and time range; the filtering logic filters the event list by field, and the filtering results simultaneously control the display of labels and list items in the scene. The event list is sorted from high to low by risk_score, so that high-risk events are seen first on site.

[0087] When a user selects an event tag on the AR device, the AR device enters the event details view and provides evidence entry points and trigger basis displays. Evidence entry points include two types of content: keyframe images and evidence fragment videos. Keyframe images are generated and uploaded by the analysis end when the event is confirmed, and must contain at least three frames: start (start frame), peak (peak / trigger frame), and end (end frame). The filename is bound to the event_id, and the AR device retrieves and displays them via an API. Evidence fragment videos are exported from the original video as MP4 files by the edge device based on the evidence_range (evidence fragment time range) and named with the event_id. The AR device plays these videos using its embedded player. The trigger basis display directly reads the trigger_summary field and displays, in order, duration_ms (duration), d_min (minimum distance), k_min (approach rate), buffer_time_ms (time spent within the buffer zone), hole_time_ms (time spent in the hole area), and the visible percentage of PPE (personal protective equipment). To facilitate on-site understanding, the AR terminal also displays explanatory text for the event type, such as "Personnel remain in the edge buffer zone for longer than the trigger condition" or "Personnel continue to approach the edge and reach the trigger condition." The explanatory text is generated by the event_type mapping table and saved together with the version number to ensure consistent interpretation.

[0088] The AR platform supports on-site review marking and writes back to the event database to form a closed-loop record. Users can select the on-site review status (review_tag) in the event details view. Enumerated values ​​include: ON_SITE_CONFIRMED (on-site review confirmed), SCENE_CHANGED (on-site situation has changed), NEED_MORE_VIEW (requires additional viewpoints), and FALSE_ALARM_ON_SITE (on-site review was a false alarm). When a user submits a review mark, the AR platform writes it to the OnsiteReviewRecord (on-site review record table). The fields must include at least event_id, review_tag, operator_id, review_time_utc_ms, and comment_text, along with alignment status information (align_confidence) and alignment matrix summary information (e.g., ...). The translation vector and rotation quaternion); where review_tag is the enumeration value of the on-site review marker, including on-site review confirmed, on-site situation has changed, additional perspective is needed, and on-site review is a false alarm; operator_id is the ID of the person who performed the on-site review operation; review_time_utc_ms is the time of the on-site review operation; comment_text is the comment text filled in during the on-site review. The server associates and saves OnsiteReviewRecord with ReviewRecord, and updates the event field onsite_status to the corresponding enumeration value. If the on-site review is FALSE_ALARM_ON_SITE, the system records the event as an "on-site false alarm sample" and increases the weight of this sample in the next feedback window statistics, so that the parameter update responds to on-site feedback more quickly.

[0089] By employing the aforementioned spatial alignment solution based on AprilTag, coordinate mapping between regional geometry and event anchor points, overlay display of regional highlighting and event tags, on-site visualization entry point for evidence fragments and trigger summaries, and on-site review mark write-back mechanism, confirmed events can be located at the regional level on-site, and evidence and trigger basis can be directly viewed. At the same time, the review results of on-site personnel can be written back to the event database to form a closed loop record, thereby reducing the understanding loss of the judgment results transmitted to the site and improving the review efficiency in high-altitude operation scenarios.

[0090] It is understandable that this application obtains multi-source bitstreams from high-altitude construction sites, which can avoid blind spots caused by single equipment, thereby improving the visual coverage of high-altitude operation risk areas. Decoding and aligning the first and second bitstreams yields a unified frame sequence. Alignment reduces the time difference between different bitstreams, improving the data accuracy of the unified frame sequence. Based on the unified frame sequence, personnel detection and scene element object generation are performed, resulting in personnel trajectory sequences and scene element objects. Based on the personnel trajectory sequences and scene element objects, hazard boundary relationships are calculated to determine target relationship sequences. These sequences are then fused to obtain candidate events. This allows for the selection of potentially unsafe candidate events from multiple videos, improving the efficiency of identifying safety hazards in high-altitude operations. By analyzing the candidate events, the final identification result is determined, improving the accuracy of identifying safety hazards in high-altitude operations. Visualizing the final identification result provides on-site verification and evidence entry points, further validating the accuracy of identifying safety hazards in high-altitude operations.

[0091] In some embodiments of this application, S102 can be implemented by S201-S204, as follows:

[0092] S201. Decode the first bitstream and the second bitstream, and determine the system time and the display timestamp obtained from decoding each frame of each bitstream in the first bitstream and the second bitstream respectively.

[0093] S202. Determine the time offset based on the displayed timestamp and system time.

[0094] S203. Based on the display timestamp and time offset, determine the unified timestamp for each frame of each bitstream in the first bitstream and the second bitstream respectively.

[0095] S204. Based on the unified timestamp, output the first bitstream and the second bitstream at a unified frame rate to determine the unified frame sequence; wherein, the unified frame sequence includes the video source number, video increment signal, display timestamp, unified timestamp, image matrix and additional information dictionary.

[0096] For example, the system uniformly accesses the RTSP stream from a fixed camera (i.e., the first stream) and the RTSP stream from a drone (i.e., the second stream), and completes decoding, unified timeline, unified frame rate, and unified preprocessing at the edge computing end, so that subsequent recognition, event construction, and evidence interception are all based on the same frame sequence standard input. Specifically, each stream is pulled and decoded into 1920×1080 BGR image frames using FFmpeg via TCP. All videos are converted into a unified frame structure Frame{source_id, frame_no, pts_ms, t_utc_ms, image_bgr, meta}, where source_id is the video source number, frame_no is the incrementing frame number of the video, pts_ms is the decoded display timestamp (ms), t_utc_ms is the unified timestamp (ms) under a unified time base, image_bgr is the image matrix, and meta is the additional information dictionary.

[0097] To achieve traceable time alignment across video sources, the system clock is used as a unified time reference, and the system clock is calibrated to UTC via NTP. For each video stream, the system performs an offset estimation for the first 200 frames after startup: recording the system time t_recv_ms at the moment each frame completes decoding, and simultaneously reading the pts_ms of that frame, calculating the time offset Δt_ms = median(t_recv_ms - pts_ms); after estimation, Δt_ms is permanently written to the configuration and remains unchanged throughout the entire runtime of that video stream. Thereafter, the unified timestamp for each frame is strictly calculated according to t_utc_ms = pts_ms + Δt_ms. Through this offset method, frames from different video sources can be mapped to the same UTC timeline, ensuring the time consistency of subsequent event start and end time windows, evidence fragment extraction, and analysis records, and enabling cross-source retrieval and alignment verification via t_utc_ms. After establishing a unified timeline, all videos are output at a uniform frame rate of 10 fps to ensure that all subsequent time thresholds can be directly converted into frame counts.

[0098] It should be noted that the unified frame structure refers to a unified frame sequence.

[0099] In some embodiments of this application, S103 can be implemented by S301-S304, as follows:

[0100] S301. Perform multi-person operation detection on each frame in the unified frame sequence to obtain the personnel detection set for each frame; and perform non-maximum suppression on each detection box in the personnel detection set of each frame to obtain the final personnel detection set for each frame.

[0101] S302. Based on the final personnel detection set of each frame, perform continuous trajectory generation to determine the personnel trajectory sequence; wherein, the personnel trajectory sequence includes the personnel landing point pixels.

[0102] S303. Perform high-altitude operation risk clue detection on each frame in the unified frame sequence to obtain the clue detection set for each frame.

[0103] S304. Based on the clue detection set of each frame, generate scene element objects and determine scene element objects.

[0104] For example, the system performs multi-person worker detection on each frame in a unified frame sequence. The detection model is YOLOv11-person (detection category is person, output is a multi-target set), and the input size is 640×640. The model inference outputs a person detection set DetSet={det_i}, where each det_i is det{bbox_xyxy, score}, and satisfies the confidence threshold score≥0.35. Subsequently, non-maximum suppression (NMS) is performed on DetSet, with an IoU threshold of 0.60, to eliminate duplicate detection boxes for the same person, resulting in the final person detection set for each frame. To ensure the uniqueness of the location definition and the consistency of subsequent calculations, for each detection box bbox_xyxy=( , , , Define the pixel position of the person (foot_px) = ( , ), and write foot_px and t_utc_ms as the standard position observation of the person in this frame into the subsequent trajectory sequence.

[0105] The system employs ByteTrack (byte trajectory tracking algorithm) to organize the final set of detected persons frame by frame into continuous trajectories and generate stable person IDs. For each source_id, the system independently maintains a tracker state set TrackSet={trk_j}, where the trk field for each trajectory is trk{track_id (track number), bbox_xyxy (rectangular detection box), score (confidence score), age (track survival frames), lost_count (loss count)}, and uses a Kalman filter to perform motion prediction on bbox_xyxy to obtain bbox_pred (predicted detection box). Data association employs a two-round matching process with the following thresholds: The first round uses only high-confidence detection (score ≥ 0.50) to match existing trajectories, with a matching cost of 1 - IoU(bbox_pred, bbox_det), and only accepts matches with IoU ≥ 0.30; the second round uses low-confidence detection (0.35 ≤ score < 0.50) to supplement the matching of unmatched trajectories from the first round, with the cost remaining consistent with the threshold. Unmatched trajectories generate new trajectories and are assigned new track_ids; unmatched trajectories increment lost_count by 1, and when lost_count = 15, the trajectory terminates and no longer participates in matching.

[0106] To suppress ID breaks caused by short-term occlusion or rapid turning from the perspective of UAV maneuvering, a short-term reconnection mechanism is added to ByteTrack. Specifically, when a trajectory is terminated due to lost_count=15, the system caches the bbox_xyxy and foot_px (person landing pixel) of the last 10 frames before termination in the DormantPool (dormant trajectory pool) and retains them for 20 frames. If a new trajectory new_trk appears during the retention period, the system calculates the pixel distance between the landing point of its first frame and the landing point of the old trajectory in the last frame, as well as the area ratio of the detection box. When the pixel distance is less than 60 pixels and the area ratio is in the range of [0.5, 2.0], new_trk.track_id is reconnected to the track_id of the old trajectory, and the two trajectories are concatenated into the same person trajectory in the time series. Through this reconnection rule, short track breaks caused by short-term occlusion or sudden changes in perspective will not generate new person IDs, thus ensuring the continuous and stable identification of persons within the event time window.

[0107] During the trajectory output stage, to simultaneously satisfy both "computational stability" and "evidence verifiability," two sets of trajectory information are output: the original landing point and the smoothed landing point. The system performs two-dimensional Kalman smoothing on the foot_px (person landing pixel) sequence of each trajectory to obtain foot_px_s, and specifies that foot_px_s (smoothed person landing pixel) is used for the overlay display of evidence keyframes. The spatial location used for risk distance calculation is mapped from pixels to world coordinates in the third part, followed by world coordinate smoothing. This maintains the stability of distance and trend calculations under scale changes and recognition jitter, while preserving a complete traceability link from the original detection to the smoothed result. Simultaneously with the generation of personnel trajectories, "visible clues related to high-altitude operation risks" are extracted and written into personnel trajectories or scene elements according to defined rules to improve event interpretability and judgment efficiency. The system runs the object cue detection model YOLOv8-risk in parallel for each frame. The model category set is: helmet, safety_belt, safety_rope, reflective_vest, guardrail, safety_net, hole, and hole_cover. Among them, helmet is a safety helmet, safety_belt is a safety belt, safety_rope is a safety rope, reflective_vest is a reflective vest, guardrail is a guardrail, safety_net is a safety net, hole is a hole, and hole_cover is a hole cover. The model input size is 640×640 (letterbox), and the output cue detection set RiskSet={r_k}, where each r_k field is r{bbox_xyxy,cls, score}, the threshold is score≥0.30, and NMS is performed once. The IoU threshold of NMS is 0.50. For wearable cues (helmet, safety_belt, safety_rope, reflective_vest), the system uses an overlap rule to assign them to personnel: for each person's trajectory bounding box bbox_person, if there is a wearable cue bounding box that satisfies IoU(bbox_risk, bbox_person)≥0.10, then the binary sequence flag_cls(t) of that person's trajectory is written as 1, otherwise it is 0; and time window smoothing is performed on flag_cls(t) to suppress flicker: if flag_cls=1 in any of the last 10 frames, then visible_cls=1 is output, otherwise it is 0. visible_cls is used as an event interpretation field and as input for subsequent risk scoring.For scene facility-related clues (guardrail, safety_net, hole, hole_cover), the system does not bind them to specific personnel, but writes them into the scene clue set of that frame, and then detects the scene element objects by checking the scene clue set.

[0108] In some embodiments of this application, S104 can be implemented by S401-S405, as follows:

[0109] S401. Determine the transformation matrix for each frame in the unified frame sequence from the pixel plane to the world plane.

[0110] Obtain the pixel coordinates of any frame in the unified frame sequence, as well as the world coordinates of pre-set marker points; then perform calculations using the pixel coordinates and world coordinates to determine the transformation matrix; or,

[0111] Obtain the camera intrinsic parameters, as well as the rotation matrix and translation vector in the world coordinate system; and determine the transformation inverse matrix using the camera intrinsic parameters, rotation matrix, and translation vector, and perform inverse operation on the transformation inverse matrix to determine the transformation matrix; wherein, the transformation inverse matrix is ​​the matrix that transforms from the world plane to the pixel plane.

[0112] For example, the "spatial relationship between hazardous boundaries and personnel trajectories" is used as the core constraint for event triggering, outputting the distance, trend, and status values ​​of each worker relative to the adjacent edge and opening at each moment. The system uniformly adopts the world plane coordinate system W as the calculation coordinate system: the working floor plane is the Z=0 plane, and X and Y are metric coordinates (unit: meters) within this plane. Boundary models, personnel positions, distances, and trends are all calculated in W. To map the personnel pixel landing points to the world plane, a unique pixel-to-world plane homography matrix is ​​provided for each frame. The coordinates are calibrated and fixed at one time during deployment: Four marker points with known world coordinates are set up on the working plane (taking the center point of AprilTag as an example), and the pixel coordinates are obtained by acquiring a frame of image. , ) and corresponding world coordinates ( , (unit: m), solved using DLT to satisfy The identity matrix is ​​generated and written to the configuration in the form of H_version; during runtime, each frame of this video stream reuses the same identity matrix. For the drone's maneuvering perspective, calculations are performed frame by frame. Known camera intrinsic parameters The UAV's RTK position is transformed into the local ENU coordinate system, and the camera's rotation matrix in the world coordinate system is obtained by combining the UAV's attitude and the gimbal's attitude. With translation vector (Unit: m). For the work plane Construct the homography matrix from the world plane to the pixel plane;

[0113]

[0114] in, for The first two columns, then take:

[0115]

[0116] Before entering this calculation, the image has already undergone distortion correction as described in the previous section, therefore the same Consistent results can be obtained from the same attitude input on different devices. The system saves either H_frame (for drones) or H_version (for fixed cameras) for each frame.

[0117] S402. Based on the transformation matrix, the pixel values ​​of the landing points of the personnel in the personnel trajectory sequence are transformed to obtain the final pixel values ​​of the landing points of the personnel in the world coordinate plane.

[0118] S403. Based on the final landing point pixel and the adjacent edge polyline, calculate the first shortest distance value between the person and the adjacent edge; and based on the final landing point pixel and the opening polygon, determine the second distance value between the person and the opening.

[0119] S404. Based on the first shortest distance value and the second distance value, perform smoothing and trend calculations to determine the first distance trend value and the second distance trend value.

[0120] S405. Based on the first shortest distance value, the second distance value, the first distance trend value, and the second distance trend value, determine the target relationship sequence.

[0121] For example, labeling the vertex sequence of adjacent or platform boundary polylines from a typical perspective. With the polygon vertex sequence of the opening area Pass each of the above vertices through the currently active [process / mechanism]. Convert to a world coordinate vertex sequence and save as adjacent polylines L_world=[ , ,..., (Unit: m) and the polygon of the opening O_world = [ , ,..., (unit: m), and generate boundary_version.

[0122] The location of the person is represented by the foot_px output in the previous section. ), and use the current frame Convert it to world location points For the adjacent polyline L_world, the first shortest distance value from the person to the adjacent edge. Calculate based on the minimum distance from the point to each segment of the broken line: for each segment (in ), calculate the projection coefficient:

[0123]

[0124] Nearest point The distance of the line segment is ,final Take the minimum distance of all line segments (in meters). For the opening polygon O_world, the opening state hole_state(t) is determined using the ray casting method, where a point is located within the polygon: if... Within O_world, it is 1; otherwise, it is 0. Both the distance and the hole's state are calculated in the world coordinate system to avoid threshold drift caused by pixel scale changes. 1 or 0 represents the second distance value.

[0125] To suppress single-frame detection jitter and mapping noise, smoothing and trend calculations are performed on the distance and state sequences. Distance smoothing employs a two-stage process: first, it processes the first shortest distance value... Perform median filtering on 5 frames to obtain Then perform an exponential moving average:

[0126]

[0127] and with As initialization, the distance trend value is calculated within a 2-second window: the nearest value is taken. Frames The slope is obtained by least-squares linear fitting:

[0128]

[0129] in, In seconds, The unit is m / s, serving as the sole quantitative indicator of "distance approach trend." The hole opening state smoothing employs a hysteresis rule: when hole_state(t) is 1 for 5 consecutive frames, output hole_state_s(t)=1; when it is 0 for 5 consecutive frames, output hole_state_s(t)=0; otherwise, maintain the previous state to avoid frequent flipping caused by jitter near the hole opening edge. The danger buffer zone threshold is... And define the buffer band state in_buffer(t) = 1 if and only if Otherwise, the value is 0. The final output is the target relation sequence.

[0130] In some embodiments of this application, S105 can be implemented by S501-S503, as follows:

[0131] S501. Based on the personnel trajectory sequence and target relationship sequence, determine the event type, event segment index, key frame index and triggering basis; among which, the key frames corresponding to the key frame index include the first frame, the last frame, and the smallest frame in the event segment; the event types include the hole entry type, the edge buffer zone continuous type, and the edge approach type.

[0132] S502. Based on the event fragment index and key frame index, conduct a risk assessment and determine the risk score.

[0133] S503. Candidate events are determined by event type, event fragment index, keyframe index, triggering basis, and risk score.

[0134] For example, the core of a candidate event is the "time window," meaning the event must have clearly defined start and end times t_start and t_end, and simultaneously provide evidence fragment indexes, keyframe indexes, and a summary of triggering criteria to facilitate verification by the analysis team. A uniform frame sequence of 10fps is used, and all duration thresholds are represented by frame numbers. Event triggering and termination only use the world coordinate distance and trend value output in the third part, without using pixel distance thresholds.

[0135] The system input consists of two types of frame-by-frame records, aligned by time. The first type is the trajectory record TrajItem, which includes at least t_utc_ms, source_id, region_id, track_id, and the visible clue fields visible_helmet, visible_belt, visible_rope, and visible_vest. Among them, visible_helmet indicates whether a person can be observed wearing a helmet in the frame, visible_belt indicates whether a person can be observed wearing a seat belt in the frame, visible_rope indicates whether a person can be observed using a safety rope in the frame, and visible_vest indicates whether a person can be observed wearing a safety vest in the frame. The second category is the relation record RelItem, which contains at least t_utc_ms, source_id, region_id, track_id, d_edge_s, k_edge, in_buffer, hole_state_s, and boundary_version. Here, d_edge_s is the smoothed shortest distance from the person to the edge (in meters), k_edge is the distance trend slope (in meters per second; negative values ​​indicate continuous approach), in_buffer indicates whether the person has entered the danger buffer zone, and hole_state_s indicates whether the person is in a stable state inside the hole area. The fusion method is an inner join: using (source_id, region_id, track_id, t_utc_ms) as the key, TrajItem and RelItem are merged into a frame-by-frame observation Obs(t).

[0136] There are three event types, prioritized to ensure that only one primary event type occurs at any given time for the same person. The event types, ranked from highest to lowest severity, are: HOLE_ENTER (entry through a hole), EDGE_BUFFER_STAY (persistent edge buffer zone), and EDGE_APPROACH (edge ​​approach). The thresholds are as follows: buffer zone distance threshold D0 = 1.5 m; approach judgment distance threshold D1 = 2.5 m; approach trend threshold k0 = 0.25 m / s; Event start duration N_start=20 (corresponding to 2.0 s); Approach start duration N_app=15 (corresponding to 1.5 s); Event end duration N_end=15 (corresponding to 1.5 s); Adjacent event merging interval threshold N_gap=10 (corresponding to 1.0 s).

[0137] The system independently maintains a two-state state machine, SAFE (safe state) and ACTIVE (active event state), for each (source_id, region_id, track_id). For each frame, three triggering conditions are calculated, expressed as strict equivalence relations:

[0138] C1(t) = 1 if and only if hole_state_s(t) = 1; C2(t) = 1 if and only if d_edge_s(t) ≤ D0; C3(t) = 1 if and only if d_edge_s(t) ≤ D1 and k_edge(t) ≤ k0;

[0139] The event begins using a "continuous fulfillment of a certain number of frames" rule, and is judged sequentially according to priority: If C1(t) is 1 for a continuous period, reaching frame N_start, then t_start is defined at the start frame of that continuous segment, HOLE_ENTER is generated, and the event enters ACTIVE; if C1 is not fulfilled, C2 is checked. If C2(t) is 1 for a continuous period, reaching frame N_start, then EDGE_BUFFER_STAY is generated with the start frame of that continuous segment as t_start, and the event enters ACTIVE; if this is still not met, C3 is checked. If C3(t) is 1 for a continuous period, reaching frame N_app, then EDGE_APPROACH is generated with the start frame of that continuous segment as t_start, and the event enters ACTIVE. Since only one ACTIVE event is allowed to exist at a time, the same person will not push multiple events repeatedly within the same time window.

[0140] The event ends based on the criterion of "risk decay and maintaining stability for N_end frames". For HOLE_ENTER, it ends when hole_state_s(t) is continuously 0 for N_end frames; for EDGE_BUFFER_STAY and EDGE_APPROACH, it ends when d_edge_s(t) > D1 and k_edge(t) > -0.05 m / s for N_end frames. The event end frame t_end is taken as the frame preceding the start frame of the continuous segment that meets the end condition, to ensure that the end boundary is consistent under different environments. The event escalation rule is as follows: if the current ACTIVE event is EDGE_APPROACH, and C2(t) is continuously 1 for N_start during the event duration, the event type is escalated to EDGE_BUFFER_STAY; if C1(t) is continuously 1 for N_start, it is escalated to HOLE_ENTER; after the escalation, the original t_start is retained, and only the event type and trigger basis summary are updated. The event merging and deduplication rule is as follows: For the same event type (source_id, region_id, track_id), if the interval between the t_start of the later event and the t_end of the earlier event does not exceed N_gap frames, they are merged into a single event. The t_start of the later event is taken, and the t_end of the earlier event is taken. After merging, the keyframes and trigger summaries are recalculated to avoid duplicate pushes caused by brief occlusion or brief exit from the buffer.

[0141] To ensure rapid verification of the analysis, keyframes and evidence fragment indexes are generated for each candidate event. Keyframes consist of three frames: F_start (the frame corresponding to t_start), F_end (the frame corresponding to t_end), and F_peak (the frame where d_edge_s reaches its minimum value within the event window). If the event type is HOLE_ENTER, F_peak is the frame where hole_state_s first equals 1, visually representing the crucial moment of entering the hole. Evidence fragment time windows are created by adding buffers on both sides of the event window: t_clip0 = t_start - 1500ms, t_clip1 = t_end + 1500ms. The system uses a unified frame index file to search the frame number range [t_clip0, t_clip1] on the t_utc_ms timeline, generating evidence_range (evidence fragment frame number range) {start_frame_no (starting frame number), end_frame_no (ending frame number)}, and saves the source_id and this time window for playback.

[0142] Each candidate event outputs a TriggerSummary, which explains why the event was triggered. The summary field must include at least: duration_ms = t_end - t_start; d_min is the minimum value of d_edge_s within the event window; k_min is the minimum value of k_edge within the event window; buffer_time_ms is the cumulative duration of in_buffer=1; hole_time_ms is the cumulative duration of hole_state_s=1; and belt_ratio, helmet_ratio, rope_ratio, and vest_ratio are the percentage of frames with the corresponding visible cue set to 1 within the event window. The RiskScore is only used for ranking and is not a final conclusion. The output is an integer score from 0 to 100, calculated according to the following rules: s_dist = min(1, max(0, When k_min When s_trend is 0, s_trend = 0; otherwise, s_trend = min(1, max(0, )), s_time = min(1,max(0, When event_type is HOLE_ENTER, s_hole = 1; otherwise, s_hole = min(1, max(0, )), s_ppe = 1 belt_ratio;

[0143] RiskScore = round (100 (0.35s_dist + 0.25s_trend + 0.20s_time +0.15s_hole + 0.05 s_ppe));

[0144] RiskLevel is mapped from RiskScore: RiskScore 80 is HIGH, 50 RiskScore < 80 is MEDIUM, the rest are LOW.

[0145] Based on the human-machine collaborative high-altitude operation behavior visual recognition method of the above embodiments, this application also provides a human-machine collaborative high-altitude operation behavior visual recognition system, such as... Figure 2 As shown, Figure 2This is a schematic diagram of the structure of a human-machine collaborative high-altitude operation behavior visual recognition system provided in an embodiment of this application. The human-machine collaborative high-altitude operation behavior visual recognition system 2 includes: an acquisition module 201, a decoding and alignment module 202, a detection and generation module 203, and a determination module 204, wherein...

[0146] The acquisition module 201 is used to acquire multi-source streams from the high-altitude operation construction site; wherein, the multi-source streams include a first stream corresponding to the fixed equipment and a second stream corresponding to the drone;

[0147] The decoding alignment module 202 is used to perform decoding and alignment processing on the first bitstream and the second bitstream to obtain a unified frame sequence.

[0148] The detection and generation module 203 is used to perform personnel detection and scene element object generation based on the unified frame sequence to obtain personnel trajectory sequence and scene element object;

[0149] The determining module 204 is used to calculate the dangerous boundary relationship and determine the target relationship sequence based on the personnel trajectory sequence and the scene element object; wherein the target relationship sequence represents the relationship sequence between people and the boundary; based on the personnel trajectory sequence and the target relationship sequence, perform fusion to obtain candidate events; based on the candidate events, perform analysis to determine the final identification result; and visualize the final identification result.

[0150] Based on the human-machine collaborative high-altitude operation behavior visual recognition method of the above embodiments, this application also provides a human-machine collaborative high-altitude operation behavior visual recognition device, such as... Figure 3 As shown, Figure 3 This is a schematic diagram of the structure of a human-machine collaborative high-altitude operation behavior visual recognition device provided in an embodiment of this application. The human-machine collaborative high-altitude operation behavior visual recognition device 3 includes a processor 301 and a memory 302. The memory 302 is used to store computer programs; the processor 301 is used to call and run the computer programs from the memory to execute the human-machine collaborative high-altitude operation behavior visual recognition method as described in the above embodiment.

[0151] In the embodiments of this application, the processor 301 described above can be at least one of the following: Application-Specific Integrated Circuit (ASIC), Digital Signal Processor (DSP), Digital Signal Processing Device (DSPD), Programmable Logic Device (PLD), Field Programmable Gate Array (FPGA), Central Processing Unit (CPU), Controller, Microcontroller, and Microprocessor. It is understood that for different devices, the electronic device used to implement the above processor function can also be other types, and the embodiments of this application do not specifically limit it.

[0152] This application provides a computer-readable storage medium storing a computer program for implementing, when executed by a processor, a visual recognition method for high-altitude operations oriented towards human-machine collaboration as described in any of the above embodiments.

[0153] For example, the program instructions corresponding to the human-machine collaborative high-altitude operation behavior visual recognition method in this embodiment can be stored on storage media such as optical discs, hard disks, and USB flash drives. When the program instructions corresponding to the human-machine collaborative high-altitude operation behavior visual recognition method in the storage media are read or executed by an electronic device, the human-machine collaborative high-altitude operation behavior visual recognition method as described in any of the above embodiments can be realized.

[0154] Furthermore, the functional modules in the embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional module.

[0155] If the integrated unit is implemented as a software functional module and is not sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this embodiment, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the method of this embodiment. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0156] It should be understood that the phrases "one embodiment," "an embodiment," or "some embodiments" mentioned throughout the specification mean that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "in one embodiment," "in one embodiment," or "in some embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. It should be understood that in the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the embodiments in this application are merely for descriptive purposes and do not represent the superiority or inferiority of the embodiments. The descriptions of the various embodiments above tend to emphasize the differences between the various embodiments; their similarities or commonalities can be referred to mutually, and for the sake of brevity, these will not be repeated here.

[0157] The modules described above as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules. They may be located in one place or distributed across multiple network units. Some or all of the modules may be selected to achieve the purpose of this embodiment according to actual needs.

[0158] In addition, each functional module in the various embodiments of this application can be integrated into one processing unit, or each module can be a separate unit, or two or more modules can be integrated into one unit; the integrated modules can be implemented in hardware or in the form of hardware plus software functional units.

[0159] Those skilled in the art will understand that all or part of the steps of the above method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, it performs the steps of the above method embodiments. The aforementioned storage medium includes various media that can store program code, such as mobile storage devices, read-only memory (ROM), magnetic disks, or optical disks.

[0160] The methods disclosed in the several method embodiments provided in this application can be arbitrarily combined without conflict to obtain new method embodiments.

[0161] The features disclosed in the several product embodiments provided in this application can be arbitrarily combined without conflict to obtain new product embodiments.

[0162] The features disclosed in the several method or device embodiments provided in this application can be arbitrarily combined without conflict to obtain new method or device embodiments.

[0163] The above description is merely an embodiment of this application, but the protection scope of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. Therefore, the protection scope of this application should be determined by the protection scope of the claims.

Claims

1. A method for visual recognition of aerial work behavior oriented to human-computer collaboration, characterized in that, The method includes: Acquire multi-source bitstreams from the high-altitude construction site; wherein, the multi-source bitstreams include a first bitstream corresponding to the fixed equipment and a second bitstream corresponding to the drone; Decoding and alignment processes are performed on the first and second bitstreams to obtain a unified frame sequence. Based on the unified frame sequence, personnel detection and scene element object generation are performed to obtain personnel trajectory sequences and scene element objects; wherein, the scene element objects include edge polylines and opening polygons; Based on the personnel trajectory sequence and the scene element objects, the dangerous boundary relationship is calculated, and the target relationship sequence is determined; wherein, the target relationship sequence represents the relationship sequence between people and boundaries; Based on the personnel trajectory sequence and the target relationship sequence, the two are fused to obtain candidate events; Based on the candidate events, an analysis is performed to determine the final identification result; and the final identification result is then visualized. The step of performing personnel detection and scene element object generation based on the unified frame sequence to obtain personnel trajectory sequences and scene element objects includes: For each frame in the unified frame sequence, perform multi-person operation personnel detection to obtain a personnel detection set for each frame; and perform non-maximum suppression on each detection box in the personnel detection set for each frame to obtain the final personnel detection set for each frame. Based on the final personnel detection set for each frame, continuous trajectory generation is performed to determine the personnel trajectory sequence; wherein, the personnel trajectory sequence includes personnel landing point pixels; High-altitude operation risk clue detection is performed on each frame in the unified frame sequence to obtain the clue detection set for each frame; Based on the clue detection set of each frame, scene element objects are generated and the scene element objects are determined. The step of calculating the danger boundary relationship and determining the target relationship sequence based on the personnel trajectory sequence and the scene element objects includes: Determine the transformation matrix from the pixel plane to the world plane for each frame in the unified frame sequence; Based on the transformation matrix, the pixel of the landing point of the person in the trajectory sequence is transformed to obtain the final pixel of the landing point of the person in the world coordinate plane. Based on the final landing point pixel and the adjacent edge polyline, calculate the first shortest distance value between the person and the adjacent edge; and based on the final landing point pixel and the opening polygon, determine the second distance value between the person and the opening. Based on the first shortest distance value and the second distance value, smoothing and trend calculations are performed to determine the first distance trend value and the second distance trend value; The target relationship sequence is determined based on the first shortest distance value, the second distance value, the first distance trend value, and the second distance trend value.

2. The method of claim 1, wherein, The decoding and alignment of the first and second bitstreams to obtain a unified frame sequence includes: Decode the first bitstream and the second bitstream, and determine the system time and the display timestamp obtained from decoding each frame of each bitstream in the first bitstream and the second bitstream, respectively. The time offset is determined based on the displayed timestamp and the system time; Based on the display timestamp and the time offset, a unified timestamp for each frame of each bitstream in the first bitstream and the second bitstream is determined respectively; Based on the unified timestamp, the first bitstream and the second bitstream are output at a unified frame rate to determine a unified frame sequence; wherein, the unified frame sequence includes a video source number, a video increment signal, a display timestamp, a unified timestamp, an image matrix, and an additional information dictionary.

3. The method of claim 1, wherein, The determination of the transformation matrix from the pixel plane to the world plane for each frame in the unified frame sequence includes: Obtain the pixel coordinates of any frame in the unified frame sequence, and the world coordinates of a pre-set marker point; then, perform calculations using the pixel coordinates and the world coordinates to determine the transformation matrix; or, Obtain the camera intrinsic parameters, as well as the rotation matrix and translation vector in the world coordinate system; and determine the transformation inverse matrix using the camera intrinsic parameters, the rotation matrix, and the translation vector, and perform inverse operation on the transformation inverse matrix to determine the transformation matrix; wherein, the transformation inverse matrix is ​​the matrix that transforms from the world plane to the pixel plane.

4. The method of claim 1, wherein, The process of fusing the personnel trajectory sequence and the target relationship sequence to obtain candidate events includes: Based on the personnel trajectory sequence and the target relationship sequence, the event type, event segment index, key frame index, and triggering basis are determined; wherein, the key frame index corresponds to the first frame, the last frame, and the smallest frame in the event segment; the event type includes hole entry type, edge buffer zone continuous type, and edge approach type. Based on the event fragment index and the keyframe index, a risk assessment is performed to determine a risk score; Candidate events are determined using the event type, the event fragment index, the keyframe index, the triggering criteria, and the risk score.

5. The method of claim 1, wherein, The step of analyzing the candidate events to determine the final identification result includes: Based on the candidate events, the first identification result of high-altitude operations is determined through risk scoring and preset risk thresholds; Receive the second identification result of high-altitude operations; wherein, the second identification result is the identification result made by the safety officer for the candidate event; Based on the first identification result and the second identification result, a comprehensive analysis is performed to determine the final identification result.

6. A human-machine collaborative high-altitude operation behavior visual recognition system, characterized in that, The high-altitude operation behavior visual recognition system for human-machine collaboration includes: an acquisition module, a decoding and alignment module, a detection and generation module, and a determination module, wherein... The acquisition module is used to acquire multi-source streams from the high-altitude operation construction site; wherein, the multi-source streams include a first stream corresponding to the fixed equipment and a second stream corresponding to the drone; The decoding alignment module is used to perform decoding and alignment processing on the first bitstream and the second bitstream to obtain a unified frame sequence; The detection and generation module is used to perform personnel detection and scene element object generation based on the unified frame sequence to obtain personnel trajectory sequence and scene element objects; wherein, the scene element objects include edge polylines and opening polygons; The determining module is used to calculate the dangerous boundary relationship and determine the target relationship sequence based on the personnel trajectory sequence and the scene element objects; wherein, the target relationship sequence represents the relationship sequence between people and boundaries; based on the personnel trajectory sequence and the target relationship sequence, perform fusion to obtain candidate events; based on the candidate events, perform analysis to determine the final identification result; and visualize the final identification result. The detection generation module is further configured to perform multi-person operation detection on each frame of the unified frame sequence to obtain a personnel detection set for each frame; and perform non-maximum suppression on each detection box in the personnel detection set of each frame to obtain a final personnel detection set for each frame; based on the final personnel detection set of each frame, perform continuous trajectory generation to determine the personnel trajectory sequence; wherein, the personnel trajectory sequence includes personnel landing point pixels; perform high-altitude operation risk clue detection on each frame of the unified frame sequence to obtain a clue detection set for each frame; and based on the clue detection set of each frame, perform scene element object generation to determine the scene element object; The determining module is also used to determine the transformation matrix of each frame in the unified frame sequence from the pixel plane to the world plane; Based on the transformation matrix, the personnel landing point pixels in the personnel trajectory sequence are transformed to obtain the final personnel landing point pixels in the world coordinate plane; based on the final personnel landing point pixels and the adjacent edge polyline, the first shortest distance value between the personnel and the adjacent edge is calculated; and based on the final personnel landing point pixels and the opening polygon, the second distance value between the personnel and the opening is determined; based on the first shortest distance value and the second distance value, smoothing and trend calculations are performed to determine the first distance trend value and the second distance trend value; based on the first shortest distance value, the second distance value, the first distance trend value, and the second distance trend value, the target relationship sequence is determined.

7. A human-machine collaborative aerial work behavior visual recognition device, characterized in that, include: Processor and memory, of which, The memory is used to store computer programs; The processor is configured to call and run the computer program from the memory to perform the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, It stores executable instructions for causing a processor to execute, thereby implementing the method of any one of claims 1 to 5.

Images