High-altitude operation behavior recognition method based on multi-modal perception and related products thereof

Abstract

This application relates to the field of computer vision technology, and provides a method for recognizing high-altitude work behavior based on multimodal perception and related products. It obtains a set of perceptual data frames by performing time synchronization and spatial pose calibration on multi-view video streams. Based on the perceptual data frames, it performs work scene matching and multi-view 3D localization to obtain a set of limb joint coordinates and a standard work template. Based on the limb joint coordinate set and the standard work template, it performs compliance verification of preparatory actions and identifies violations during the work process to obtain compliance data for preparatory actions and violation risk indicators. Based on the compliance data for preparatory actions and violation risk indicators, it performs a high-altitude work compliance assessment to generate a high-altitude work training score report. This application, by completing multi-view alignment and then using 3D joint positions to perform correlation analysis between personnel actions and equipment status, provides traceable and comparable recognition results for high-altitude work training examinations.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method for recognizing high-altitude work behavior based on multimodal perception and related products. Background Technology

[0002] Working at heights is widely used in construction, power maintenance, communications maintenance, bridge maintenance, and industrial equipment installation. Due to the complex working environment, limited working space, and high risk of falls, the requirements for workers' standardized and safe operation are very high. During training and on-site supervision of working at heights, it is necessary not only to determine whether workers have completed the prescribed preparatory actions and standardized operations, but also to continuously monitor dangerous behaviors during the operation and combine the results of the entire operation to form an objective and fair evaluation, thereby providing a reliable basis for personnel training assessment, safety management, and risk warning.

[0003] Existing methods for identifying behaviors during high-altitude operations typically employ monocular cameras or fixed-view video to perform target detection, posture recognition, and behavior analysis on workers. Based on the identification results, they determine whether workers are violating regulations. Some methods combine the detection of protective equipment such as safety helmets and safety belts to achieve work safety supervision, while others introduce multiple cameras for multi-view monitoring to improve target recognition capabilities in complex scenarios. Based on the detection results, they generate work records or safety alarms to achieve automated monitoring and management of the high-altitude operation process.

[0004] Existing methods for identifying behaviors in high-altitude operations mostly focus on detecting single actions or localized violations. They lack the unified integration of information from multiple perspectives and the continuous correlation analysis between the preparation stage and the entire operation process. They are easily affected by factors such as viewpoint obstruction, differences in camera position, and insufficient continuity of actions, leading to misjudgments or omissions in the behavior recognition results. They are difficult to comprehensively and accurately reflect the overall level of compliance with work procedures and to form a comprehensive evaluation result that takes into account preparation actions, the operation process, and violations. This affects the reliability of training and assessment for high-altitude operations and on-site safety supervision. Summary of the Invention

[0005] This application provides a method for recognizing high-altitude work behavior based on multimodal perception and related products to solve the problem of unreliable high-altitude work behavior recognition leading to inaccurate training and assessment.

[0006] In a first aspect, embodiments of this application provide a method for recognizing high-altitude work behavior based on multimodal perception, including: The acquired multi-view video streams are processed for time synchronization and spatial pose calibration to obtain a group of perceptual data frames with spatiotemporal alignment. Based on the perception data frame group, the operation scene is matched and multi-view three-dimensional positioning is processed to obtain the coordinate set of the limb joint points of the operator and the corresponding operation standard template. Based on the set of limb joint coordinates and the standard work template, the compliance of preparatory actions is verified and violations in the work process are identified to obtain compliance data of preparatory actions and violation risk indicators of the work process. Based on the compliance data of the preparation actions and the violation risk indicators, a compliance assessment of high-altitude operations is conducted, and a high-altitude operations training score report is generated.

[0007] In an optional implementation, the step of performing time synchronization and spatial pose calibration processing on the acquired multi-view video streams to obtain a spatiotemporally aligned group of sensory data frames includes: Based on the timestamp, the acquired multi-view video streams are time-synchronized and aligned to obtain the synchronized view image set corresponding to each acquisition moment. The synchronous viewpoint image set is calibrated by shooting pose according to preset spatial calibration points to obtain the projection parameters corresponding to each viewpoint. Based on the projection parameters, the spatial coordinates of each image in the synchronized viewpoint image set are unified to obtain a corrected standard visual image. The standard visual images are associated and encapsulated according to time sequence to obtain a spatiotemporally aligned perceptual data frame group.

[0008] In an optional implementation, the step of performing work scene matching and multi-view 3D positioning processing based on the perceived data frame group to obtain the limb joint coordinate set of the worker and the corresponding work standard template includes: Local scene features are extracted from each image in the perception data frame group, and the local scene features are correlated and fused to obtain global scene features; The current job scenario type is identified based on the global scenario features, and the corresponding job standard template is matched from the preset standard template library based on the job scenario type. Target detection of workers and multi-view association matching are performed on each image of the perception data frame group to obtain the detection association data of the same worker in each view, the detection dataset of each worker, and the image region data corresponding to each worker in each view. Based on the detection association data, extract the limb joint image coordinates of each worker from the detection dataset from each viewpoint. Based on the projection parameters, the coordinates of the limb joint points in the image are processed using multi-view three-dimensional positioning to obtain the coordinate set of the limb joint points of each worker.

[0009] In an optional implementation, the step of verifying the compliance of preparatory actions and identifying violations in the work process based on the limb joint coordinate set and the work standard template, to obtain compliance data for preparatory actions and violation risk indicators for the work process, includes: By performing temporal correlation motion analysis on the set of limb joint coordinates through a preset continuous time window, the limb movement characteristics of each worker can be obtained. The compliance of the preparatory movements of the limb movement features is verified according to the aforementioned work standard template to obtain compliance data of the preparatory movements; Based on the aforementioned operational standard template, the image area data is used to detect the wearing of security equipment, thereby obtaining the compliance identification of each operator's wearing of the equipment. Based on the aforementioned work standard template, the limb movement characteristics are used to identify violations during the work process, thereby obtaining a violation risk indicator for the work process.

[0010] In an optional implementation, the step of conducting a compliance assessment of high-altitude operations based on the compliance data of the preparation actions and the violation risk indicators, and generating a high-altitude operations training score report, includes: The completion rate of the preparation actions is obtained by statistically analyzing the compliance data of the preparation actions and the compliance identification of the wearer. Based on the violation risk identifier, time-series cumulative statistics are performed to obtain the violation frequency and the number of continuous violation events for various types of violations. Based on the frame-level violation frequency and the number of continuous violation events, the deduction value for various types of violations is calculated to obtain the total violation monitoring deduction. The score is calculated and weighted summed based on the completion rate of the preparatory actions and the total number of points deducted for violations, to obtain the comprehensive score for high-altitude operations. According to the preset report format, the compliance data of the preparation actions, the violation risk identification, and the comprehensive score of the high-altitude operation are combined into a high-altitude operation training score report.

[0011] In an optional implementation, the method further includes: The movement pattern features corresponding to high-risk actions are extracted from the limb movement features, and the movement pattern features are matched with a preset high-risk feature set to generate a risk signal. When the risk signal is a preset high-risk signal, an alarm is triggered according to the preset alarm method.

[0012] Secondly, embodiments of this application provide a high-altitude work behavior recognition device based on multimodal perception, comprising: The spatiotemporal synchronization module is used to perform time synchronization and spatial pose calibration processing on the acquired multi-view video streams to obtain spatiotemporally aligned perception data frame groups. The job recognition module is used to perform job scene matching and multi-view three-dimensional positioning processing based on the perception data frame group to obtain the coordinate set of the limb joint points of the worker and the corresponding job standard template. The compliance verification module is used to verify the compliance of preparatory actions and identify violations in the operation process based on the limb joint point coordinate set and the operation standard template, and to obtain compliance data of preparatory actions and violation risk indicators of the operation process. The work assessment module is used to conduct a compliance assessment of high-altitude operations based on the compliance data of the preparation actions and the violation risk indicators, and generate a high-altitude operation training score report.

[0013] Thirdly, embodiments of this application provide a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the above-described method for recognizing high-altitude work behavior based on multimodal perception.

[0014] Fourthly, embodiments of this application provide a readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method for recognizing high-altitude work behavior based on multimodal perception.

[0015] Fifthly, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the above-described method for recognizing high-altitude work behavior based on multimodal perception.

[0016] The solution provided by the above-mentioned high-altitude operation behavior recognition method and related products based on multimodal perception first uniformly calibrates the data collected from different perspectives, and then combines it with the standard operating procedures corresponding to the high-altitude operation scenario to continuously analyze the spatial movement trajectory and operation behavior of the workers. It also comprehensively evaluates the pre-operation preparation state and violations during the operation process, rather than judging based solely on a single frame image or a single behavior. This effectively reduces misjudgments and omissions caused by perspective obstruction, camera position differences, and insufficient continuity of actions, and improves the accuracy, completeness, and stability of the high-altitude operation behavior recognition results. At the same time, it makes the generated training score results more objective and consistent, providing reliable data support for high-altitude operation training assessment and on-site safety supervision. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of the operating environment of the high-altitude operation behavior recognition method based on multimodal perception in an embodiment of the present invention; Figure 2 This is a flowchart illustrating the high-altitude operation behavior recognition method based on multimodal perception in an embodiment of the present invention. Figure 3 yes Figure 1 Functional module diagram of a high-altitude operation behavior recognition device based on multimodal perception integrated in a server; Figure 4 This is a schematic diagram of the structure of the computer device in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0020] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof. It should also be understood that, as used in this specification and the appended claims, the term "and / or" refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0021] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0024] To illustrate the technical solution of the present invention, specific embodiments are described below.

[0025] Working at heights is widely used in construction, power maintenance, communications maintenance, bridge maintenance, and industrial equipment installation. Due to the complex working environment, limited working space, and high risk of falls, the requirements for workers' standardized and safe operation are very high. During training and on-site supervision of working at heights, it is necessary not only to determine whether workers have completed the prescribed preparatory actions and standardized operations, but also to continuously monitor dangerous behaviors during the operation and combine the results of the entire operation to form an objective and fair evaluation, thereby providing a reliable basis for personnel training assessment, safety management, and risk warning.

[0026] Existing methods for identifying behaviors during high-altitude operations typically employ monocular cameras or fixed-view video to perform target detection, posture recognition, and behavior analysis on workers. Based on the identification results, they determine whether workers are violating regulations. Some methods combine the detection of protective equipment such as safety helmets and safety belts to achieve work safety supervision, while others introduce multiple cameras for multi-view monitoring to improve target recognition capabilities in complex scenarios. Based on the detection results, they generate work records or safety alarms to achieve automated monitoring and management of the high-altitude operation process.

[0027] Existing methods for identifying behaviors in high-altitude operations mostly focus on detecting single actions or localized violations. They lack the unified integration of information from multiple perspectives and the continuous correlation analysis between the preparation stage and the entire operation process. They are easily affected by factors such as viewpoint obstruction, differences in camera position, and insufficient continuity of actions, leading to misjudgments or omissions in the behavior recognition results. They are difficult to comprehensively and accurately reflect the overall level of compliance with work procedures and to form a comprehensive evaluation result that takes into account preparation actions, the operation process, and violations. This affects the reliability of training and assessment for high-altitude operations and on-site safety supervision.

[0028] To address the aforementioned issues, this application proposes a method for recognizing high-altitude work behavior based on multimodal perception and related products. By first uniformly calibrating data collected from different perspectives, and then combining this with standard operating procedures corresponding to the high-altitude work scenario, the spatial movement trajectory and work behavior of the workers are continuously analyzed. Furthermore, the pre-work preparation state and violations during the work process are comprehensively evaluated, rather than relying solely on a single frame image or a single behavior. This effectively reduces misjudgments and omissions caused by perspective obstruction, camera position differences, and insufficient continuity of actions, improving the accuracy, completeness, and stability of the high-altitude work behavior recognition results. Simultaneously, it makes the generated training score results more objective and consistent, providing reliable data support for high-altitude work training assessments and on-site safety supervision.

[0029] The high-altitude operation behavior recognition method based on multimodal perception provided in this invention can be applied to, for example... Figure 1 The operating environment shown includes: multiple cameras deployed at the high-altitude operation training site, a server integrating all the processing logic of the recognition method, and a data transmission network connecting each camera and the server.

[0030] Specifically, multiple cameras are fixedly installed at different locations around the high-altitude operation training area. The image acquisition field of each camera covers the range of the workers' activities, allowing for real-time capture of video images of the high-altitude operation area from different perspectives. The acquired multi-view video streams are then transmitted to the server in real-time via the data transmission network. The server is communicatively connected to each camera, receiving the multi-view video streams transmitted from each camera and executing the high-altitude operation behavior recognition method based on multimodal perception described in this application based on the acquired multi-view video streams. The data transmission network establishes data connections between each camera and the server, transmitting the video stream data acquired by each camera to the server to ensure that the server can acquire multi-view video stream data for the recognition method in real time. The server is also connected to a display device, which displays the high-altitude operation training scoring report generated by the server and the intermediate recognition results of each processing stage in real time.

[0031] In one embodiment, such as Figure 2 As shown, a method for recognizing high-altitude work behavior based on multimodal perception is provided, which is then applied to... Figure 1 Taking the runtime environment of [the system] as an example, the following steps are included: Step S1: Perform time synchronization and spatial pose calibration processing on the acquired multi-view video streams to obtain a spatiotemporally aligned group of perception data frames.

[0032] It should be understood that multi-view video streams refer to the raw video stream data collected by multiple cameras deployed at different locations around a high-altitude work training site. For example, in a high-altitude work training site located on a power pole climbing training tower or a building high-altitude simulation work area, multiple cameras are installed at ground-level elevation positions, side positions in the middle of the tower, and top-level overhead positions. Each camera continuously captures RGB visible light video images of the work area at a fixed sampling frequency. Due to individual differences in the clock crystals built into each camera, their startup times have millisecond-level deviations, and each maintains an independent time counter. Therefore, the video streams do not have natural time alignment characteristics. At the same time, the installation height, elevation angle, yaw angle, and lens distortion parameters of each camera are different, resulting in a lack of a unified mapping benchmark for the image content captured by each perspective at the same moment in the spatial coordinate system.

[0033] To this end, time synchronization alignment processing is performed on the acquired multi-view video streams based on timestamps. Specifically, when each camera acquires a video stream, a hardware synchronization signal generator outputs a synchronization timing signal based on the GPS second pulse to each camera encoder, so that each frame of image is assigned a timestamp tag in microseconds at the end of exposure. Using the clock axis of one of the main cameras as the time reference axis, for each frame of image from the other subordinate cameras, the timestamp it carries is matched with the timestamp sequence of the reference axis, and image frames with an absolute time difference less than a preset threshold (e.g., 40ms) are searched as candidate frames for the same acquisition time. For the inconsistency in motion blur caused by the difference in exposure time of each camera in the candidate frame set, a linear interpolation compensation method is used to normalize and correct the brightness value of each pixel according to the exposure time weight. After the above time synchronization alignment processing, the images of each camera at the same acquisition time are associated as a synchronized view image set. This image set eliminates the time offset between the video streams, ensuring that the view images used for subsequent spatial pose calibration are precisely corresponding in time, and providing a unified time reference for multi-view joint processing.

[0034] Then, the shooting pose of the synchronized viewpoint image set is calibrated according to the preset spatial calibration points. At least six spatial calibration points with known three-dimensional coordinates in the world coordinate system are pre-arranged in the training area. These calibration points are spherical targets made of high infrared reflectivity material, and their center coordinates are obtained by total station measurement with millimeter-level accuracy. Sub-pixel-level corner coordinates are extracted from the calibration point images captured by each camera to obtain the two-dimensional pixel coordinates of each calibration point in the image plane of each camera. Based on the pinhole imaging geometric model, a projection equation is established to map three-dimensional points in the world coordinate system to two-dimensional pixels in the image plane. For each camera, its internal parameter matrix and external parameter matrix are obtained by solving this projection equation. The internal parameter matrix includes the normalized focal length, principal pixel coordinates, radial distortion coefficient, and tangential distortion coefficient in the x-axis and y-axis directions of the image; the external parameter matrix includes the rotation matrix and translation vector relative to the world coordinate system, where the rotation matrix is ​​transformed into a three-dimensional rotation vector using Rodrigues transformation to reduce the dimensionality of the parameters to be solved. After independent calibration of each camera, one camera is arbitrarily selected as the spatial reference camera. Its rotation matrix is ​​set to the identity matrix, and its translation vector is set to zero. The relative rotation matrices and relative translation vectors of the other cameras are solved relative to this reference camera. A joint nonlinear optimization of the internal and external parameters of each camera is then performed by constructing a reprojection error objective function to minimize the sum of squared reprojection errors of all calibration points from each camera's viewpoint. After the above shooting pose calibration process, complete projection parameters of each camera in the world coordinate system are obtained. These projection parameters include the internal parameter matrix, distortion coefficients, and rotation and translation vectors relative to the world coordinate system for each camera, providing a mathematical mapping reference for the subsequent conversion of two-dimensional image coordinates to three-dimensional spatial coordinates.

[0035] Next, spatial coordinate unification processing is performed on each image in the synchronized viewpoint image set according to the projection parameters. Specifically, firstly, based on the radial and tangential distortion coefficients of each camera, distortion correction is performed on all pixels of each frame in the synchronized viewpoint image set, restoring pixels whose positions have shifted due to lens manufacturing deviations in actual imaging to their theoretical positions under the ideal pinhole imaging model. Specifically, for any pixel in the image, the radial and tangential distortion displacements are calculated based on the radial distance of the pixel from the principal point of the image. The distortion displacements are then subtracted from the actual observed coordinates of the pixel to obtain the ideal pixel coordinates after distortion correction. Then, based on the relative rotation matrix and relative translation vector between each camera, epipolar correction processing is performed on each viewpoint-corrected image. By constructing the correction transformation matrix for each camera, the pixel planes of each viewpoint image are reprojected onto the same virtual plane, so that the same spatial point has the same vertical coordinate value in each viewpoint-corrected image. After the above spatial coordinate unification processing, a corrected standard visual image is obtained. This standard visual image eliminates the pixel position deviation caused by lens distortion and the image plane tilt inconsistency caused by the difference in camera orientation between different viewpoints, so that the projection of the same spatial target in the images of each viewpoint satisfies the standardized epipolar geometric constraints.

[0036] Finally, the standard visual images are associated and encapsulated according to time sequence to obtain spatiotemporally aligned perceptual data frame groups. Specifically, the standard visual images of each viewpoint, after time synchronization alignment, distortion correction, and epipolar correction, are grouped and encapsulated according to the timestamp order of their acquisition time. All viewpoint images at each time point constitute a data frame group, and all data frame groups are arranged chronologically to form a standard visual image sequence. Each frame group also encapsulates the projection parameters of each camera at the corresponding time point, so that subsequent steps can directly obtain the projection matrix of the corresponding viewpoint for 3D coordinate calculation at any time point. After the above association and encapsulation processing, the obtained spatiotemporally aligned perceptual data frame group is a standardized multi-view image data set with consistent spatiotemporal reference. Each frame group contains corrected images of multiple viewpoints at the same time point and the corresponding camera projection parameters, providing data input for subsequent scene matching and multi-view 3D positioning processing.

[0037] Step S2: Perform work scene matching and multi-view 3D positioning processing based on the perception data frame group to obtain the coordinate set of the limb joint points of the worker and the corresponding work standard template.

[0038] First, local scene features are extracted from each image in the perception data frame group, and these features are then correlated and fused. Specifically, for each viewpoint standard visual image in the perception data frame group, the image is divided into uniformly distributed local image blocks using a sliding window of fixed size. Statistical calculations are performed on the pixel values ​​within each image block, including the mean pixel value of each color channel, the standard deviation of the pixel value of each color channel, and the gradient direction histogram. The gradient direction histogram is constructed as follows: for each pixel within the image block, the horizontal and vertical gradients are calculated based on the gray-level difference between adjacent pixels. Then, the gradient direction angle at that point is determined based on the ratio of the horizontal and vertical gradients. Simultaneously, the gradient magnitude at that point is calculated based on the square root of the sum of the squares of the horizontal and vertical gradients. The gradient direction angles of all pixels are uniformly divided into nine intervals from 0 degrees to 180 degrees. The cumulative value of the gradient magnitude falling into each interval is statistically analyzed to form a nine-dimensional gradient direction feature vector. The statistics of all color channels and histogram vectors of gradient directions for each image patch are concatenated in spatial order to form a vector describing the local scene features of the image from that viewpoint (hereinafter collectively referred to as the local scene feature description vector). For the local scene feature description vectors of multiple viewpoints in the same frame group, since the shooting orientation of each viewpoint is different, the scale ratio and image coordinate position of the same scene area in the images of each viewpoint are different. A weighted average fusion strategy based on feature vector normalization is adopted for multi-view correlation fusion. First, the mean of the dimension of each dimension in the local scene feature description vector of each viewpoint is subtracted from the mean of the dimension across all viewpoints and divided by the standard deviation of the dimension across all viewpoints to unify the feature dimensions of each viewpoint to the same scale. Then, the fusion weight of each viewpoint feature vector after normalization is determined according to the spatial distance of each camera from the center point of the working area. The closer the camera is installed to the center point of the working area, the clearer the scene details in its image, and the higher the fusion weight is assigned. The normalized feature vectors of all viewpoints are weighted and summed according to their respective weights to obtain the global scene feature vector of the current frame group. Since a single-view image cannot fully reflect the entire work scene due to limited field of view and occlusion issues, multi-view feature fusion can integrate scene information from all directions to form a complete feature description of the environmental state of the entire work area.

[0039] Then, the current work scenario type is identified based on the global scene features, and the corresponding work standard template is matched from the preset standard template library according to the work scenario type. Specifically, the fused global scene feature vector is input into a pre-built scene classification discriminator, which is built based on support vector machine and stores the judgment boundary parameters of various work scenario categories in a high-dimensional feature space. The discriminator calculates the distance between the input global scene feature vector and each category in the feature space, outputs the probability confidence value of the frame group belonging to each preset scene category, and selects the category with the highest probability confidence value that exceeds the preset threshold as the work scenario type label of the current frame group. The high-altitude operation training covers different work types such as climbing power poles, climbing power towers, high-altitude hoisting equipment operations, and high-altitude suspended platform operations. The safety preparation action sequences and operation procedures for each type are significantly different. For example, climbing power poles requires a special inspection of the tower guy wires and pole base before climbing, while climbing power towers requires confirmation of the tower materials and connecting hardware one by one. The preparation actions and judgment criteria for the two types of operations are completely different. Therefore, based on the identified job scenario type label, a job standard template that uniquely corresponds to the job scenario type is extracted from the preset standard template library. This template contains a list of all types of security equipment required for the job scenario type and a description of the correct wearing position, a prescribed sequence of pre-job preparation actions and a description of the key postures of each action, and the types of violations prohibited during the job and their kinematic judgment parameters. This template serves as the reference benchmark for all subsequent compliance judgments.

[0040] Next, target detection of workers and multi-view association matching are performed on each image of the perception data frame group to obtain the detection association data of the same worker in each view, the detection dataset of each worker, and the image region data corresponding to each worker in each view. Specifically, for the standard visual image of each viewpoint in the perception data frame group, a detection network based on a multi-scale feature pyramid structure is used to independently perform worker target detection. This detection network performs multi-layer convolutional downsampling on the input image to extract feature maps at different semantic levels. Then, the high-level semantic features are passed to the low-level feature maps via lateral connections to generate a feature pyramid that integrates multi-scale information. At each level of the feature pyramid, anchor boxes with sizes and aspect ratios matching the typical scale of human targets in high-altitude work scenarios are generated in a dense sliding window manner. Each anchor box is used to calculate the probability confidence that the output box contains a human target through the classification branch, and the center coordinate offset and width and height scaling of the output box relative to the actual human bounding box are calculated through the regression branch. All anchor boxes are sorted from high to low probability confidence and redundant detection boxes with overlap exceeding a preset threshold are removed using a non-maximum suppression method. The detection box with the highest probability confidence is retained as the human detection result under that viewpoint. In this way, the image region data corresponding to each worker in each viewpoint is obtained. This image region data is specifically represented as the two-dimensional bounding box coordinates of each human target in the image of each viewpoint. Since the detection results of the same worker in different viewpoints have not yet established a correspondence after independent detection from each viewpoint, multi-view association matching needs to be further performed. For a bounding box of a worker detected in an image from a certain viewpoint, the pixel coordinates of the bottom center point of the bounding box are mapped to a 3D ray in the world coordinate system through the projection parameters of that viewpoint. The epipolar projection of this ray on the image plane of other viewpoints is calculated, and it is checked whether the bottom center points of the bounding boxes of each worker detected in other viewpoints fall within a preset pixel distance threshold near the epipolar line. If they meet the requirement, it is determined that it is the detection result of the same worker target in different viewpoints. After performing the above cross-validation on all detection results in all viewpoints, a cross-view correspondence is established between the detected workers targets in each viewpoint, forming the detection association data of the same worker in each viewpoint. At the same time, the bounding box coordinates of each worker in each viewpoint and the corresponding worker target confidence are summarized into the detection dataset of each worker. The purpose of the above operations is to determine the spatial range of the target worker in each viewpoint image for subsequent limb joint extraction and 3D localization, and to establish a cross-view correspondence to ensure that the 2D observation points of each viewpoint belong to the same physical worker when calculating the subsequent 3D coordinates.

[0041] Then, based on the detection association data, the limb joint image coordinates of each worker from each viewpoint are extracted from the detection dataset. Specifically, for each worker, the corresponding image region in each viewpoint image is determined based on the detection association data. For the region inside the two-dimensional bounding box defined by the image region data corresponding to the worker in each viewpoint, a pose estimation algorithm based on part fields is used to extract the two-dimensional coordinates of human joints. The pose estimation algorithm first calculates the response maps of each human body part within the bounding box region of the image. The human body parts include the head, torso, left upper arm, right upper arm, left forearm, right forearm, left thigh, right thigh, left calf, and right calf. The response map of each part represents the probability distribution of each pixel location in the image belonging to that part. Then, it calculates the association vector field between adjacent parts, which encodes the direction vector and strength information of the connection between parts. Combining the response maps of all parts and the association vector field between all parts, the algorithm connects the parts into a complete human skeleton by solving a global optimal matching problem. Finally, it outputs the two-dimensional image coordinates of fifteen key joints: top of head, neck, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, hip center, left hip, right hip, left knee, right knee, left ankle, and right ankle. For the same worker, the above pose estimation is performed independently at each viewpoint to obtain the image coordinates of the limb joints of that worker at all viewpoints.

[0042] Finally, based on the projection parameters, multi-view 3D localization processing is performed on the limb joint image coordinates to obtain the limb joint coordinate set for each worker. Specifically, for a specific limb joint of a worker, the 2D image coordinates of the joint at each viewpoint are obtained from the detection association data, and the camera projection parameters corresponding to each viewpoint are read from the perception data frame group. Since the 2D observation point coordinates of each viewpoint are back-projected into 3D space according to the calibration-obtained projection parameters, the projection rays of each viewpoint usually do not intersect precisely at the same spatial point due to the existence of calibration residuals and 2D coordinate extraction errors, a multi-view triangulation method is used to solve for the 3D spatial point coordinates that minimize the overall reprojection error of each viewpoint. Specifically, for all viewpoint 2D homogeneous coordinate observations of the joint point and the projection matrices of each viewpoint, the projection equations of each viewpoint are stacked to eliminate the scale factor and form a homogeneous linear equation system. Singular value decomposition is performed on the coefficient matrix of this equation system, the right singular vector corresponding to the smallest singular value is taken, and the first three components of the vector are divided by the fourth component to obtain the 3D coordinate value of the joint point in the world coordinate system. If a joint is observed in more than three viewpoints, a weighting coefficient is introduced in the singular value decomposition process described above. This weighting coefficient is set based on the detection confidence value of the joint in each viewpoint; the higher the detection confidence, the greater the weight contribution of the observation value from that viewpoint in the 3D coordinate calculation. The above multi-view 3D positioning process is performed on all fifteen joints of each worker to obtain a complete 3D limb skeleton coordinate set for each worker. This coordinate set contains the 3D coordinate values ​​of each joint in the world coordinate system, providing precise spatial location input for subsequent compliance verification of preparatory actions and identification of violations during the operation.

[0043] Step S3: Based on the set of limb joint coordinates and the standard work template, perform compliance verification of preparatory actions and identification of violations in the work process to obtain compliance data of preparatory actions and violation risk indicators of the work process.

[0044] After obtaining the set of limb joint coordinates of the operator and the corresponding standard template for the operation, the compliance of the preparation action and the identification of violations in the operation process are further verified based on the set of limb joint coordinates and the standard template for the operation, so as to obtain compliance data of the preparation action and the violation risk identification of the operation process. The specific processing is performed as follows.

[0045] First, a temporal correlation motion analysis is performed on the set of limb joint coordinates using a preset continuous time window to obtain the limb movement characteristics of each worker. Specifically, from the output set of limb joint coordinates for each frame, the three-dimensional coordinate sequence of joints for the same worker at multiple consecutive frame groups is extracted, and the skeleton association of the same worker between adjacent frames is established. For two adjacent frame groups, the three-dimensional coordinate sets of all fifteen joints of the worker in the previous frame and the three-dimensional coordinate sets of the corresponding joints in the current frame are obtained respectively. The displacement vector of the same joint in three-dimensional space between adjacent frames is calculated, and the displacement vector is divided by the time interval between adjacent frames to obtain the instantaneous velocity vector of the joint at the current frame. Then, the difference between the instantaneous velocity vector at the current frame and the instantaneous velocity vector at the previous frame is divided by the time interval to obtain the instantaneous acceleration vector of the joint at the current frame. For each frame within the temporal window, the sum of the magnitudes of the instantaneous velocity vectors of all joints at the same moment is used to obtain the motion energy value of that frame. The change sequence of motion energy values ​​within the preset continuous time window is used as the motion characteristic reflecting the overall activity intensity of the worker. Furthermore, for the left and right wrist joints, their relative displacement vectors relative to the hip center joint are extracted. The trajectory sequence formed by these relative displacement vectors within the time window reflects the movement pattern of the arm relative to the torso. For the left and right ankle joints, the sequence of changes in their vertical coordinate values ​​in the world coordinate system over time is extracted to characterize the vertical movement pattern of a person's climbing or standing posture. The above-mentioned temporally correlated motion analysis transforms discrete single-frame spatial location information into temporally continuous limb motion features, providing multi-dimensional kinematic parameter inputs, including velocity, acceleration, and relative displacement, for subsequent temporal matching of preparatory actions and dynamic identification of violations.

[0046] Then, the compliance of the preparatory actions for the limb movement features is verified according to the operation standard template to obtain compliance data for the preparatory actions. Specifically, the preparatory action sequence corresponding to the current scenario type is read from the operation standard template. Taking the pole climbing operation scenario as an example, its standard preparatory action sequence is as follows in chronological order: environmental monitoring action, placement of work fence and warning signs, and safety rope attachment action. Each preparatory action is described in the template as a combination of keyframe posture skeleton sequence and motion feature threshold. The continuous limb movement feature sequence obtained from the temporal correlation motion analysis is segmented into temporal action segments using a sliding window-based motion energy threshold segmentation method. Time periods in which the motion energy value continuously exceeds a preset resting threshold are marked as candidate action segments, and the start frame time and end frame time of each candidate action segment are extracted. For each segmented candidate action fragment, its frame-by-frame pose skeleton sequence is dynamically time-warped and matched with the keyframe pose sequences of each standard preparation action in the standard template. The dynamic time-warping method constructs an Euclidean distance cost matrix between the pose vectors of each frame of the candidate fragment and the pose vectors of each frame of the standard template. It then searches the matrix for the path with the minimum cumulative cost from the starting point to the ending point, and normalizes the cumulative distance values ​​at each point along the path to the sequence length as the similarity score between the candidate fragment and the standard action. The standard preparation action with the highest similarity score exceeding a preset matching threshold is selected as the matching result for the candidate fragment. Based on the matching order of each standard preparation action on the timeline, it is determined whether the operator executed each preparation action in the sequence specified in the standard template. For each successfully matched preparatory action, a fine-grained quality check is performed based on the motion characteristic parameters extracted from the corresponding action segment in the temporal correlation motion analysis. For example, for environmental monitoring actions, the cumulative value of the angle change of the neck joint points around the vertical axis of the world coordinate system within the action segment is calculated to determine whether the cumulative value of the angle change reaches the threshold of not less than 180 degrees of horizontal scanning range specified in the standard template. For safety rope attaching actions, the relative distance change pattern between the left and right wrist joint points within the action segment is calculated to determine whether the hands have completed the rope buckle insertion action within the specified space in front of the body. The matching results and quality check results of each preparatory action are compiled into a binary identifier sequence according to the action order specified in the standard template. Each dimension in the sequence corresponds to one preparatory action. A complete and standardized execution of the action is marked as compliant, while an action that is not executed, has an incorrect temporal order, or is executed improperly is marked as non-compliant. This binary identifier sequence is the compliant data for preparatory actions.

[0047] Next, the image area data is subjected to security equipment wearing detection according to the aforementioned work standard template to obtain the compliance identification of each worker's wearing. Specifically, starting from the image area data of each worker from each viewpoint obtained in step S2, dedicated target detection of four types of security equipment—safety helmet, work clothes, gloves, and safety belt—is performed on the area inside the bounding box of each worker in each viewpoint image. Safety helmet detection uses the head area of ​​the bounding box of the worker as the detection range, and determines the presence of a safety helmet by analyzing the curvature and color distribution characteristics of the semi-ellipsoidal shape of the edge curve in this area; work clothes detection uses the torso area of ​​the bounding box of the worker as the detection range, and determines whether work clothes are worn by identifying the color characteristics and reflective strip texture characteristics in this area; glove detection uses the wrist end area of ​​the bounding box of the worker as the detection range, and determines whether gloves are worn by identifying the presence of fabric texture characteristics covering the hands in this area; safety belt detection uses the torso area of ​​the bounding box of the worker as the detection range, and determines whether the safety belt is fastened by identifying the presence of edge lines with intersecting strip structures in this area. For a security device detected in a certain viewpoint, the center pixel coordinates of its detection bounding box are mapped to a three-dimensional ray in the world coordinate system according to the projection parameters obtained in step S1. The epipolar line of the ray in other viewpoint images is calculated, and it is checked whether the detection results of the same device in other viewpoints fall within the preset pixel distance threshold range near the epipolar line. If the same device is detected by at least two viewpoints at the same time, the existence of the device is confirmed. The detection results of each viewpoint are cross-validated and fused to obtain the three-dimensional position coordinates of each security device in the world coordinate system. The spatial positional relationship between the detected 3D positions of each security device and the corresponding 3D coordinates of body joints in the limb joint point coordinate set is compared. A helmet is considered compliant if the 3D Euclidean distance between its position and the head joint point is less than a preset position matching threshold and it is located above the head joint point. Similarly, a work uniform is considered compliant if the overlap volume ratio between the 3D point set covered by the work uniform and the torso surface area enclosed by the central joint points of the neck, left shoulder, right shoulder, and hip exceeds a preset coverage threshold. Gloves are considered compliant if the 3D Euclidean distance between the left glove and the left wrist joint point coordinates, and the 3D Euclidean distance between the right glove and the right wrist joint point coordinates are both less than a preset matching threshold. A seatbelt is considered compliant if the normal distance sequence between the 3D sampling points of the seatbelt strap path and the torso surface is within a preset range and passes through the attachment point area specified in the standard template. The compliance determination results for these four types of security devices are combined into a four-dimensional binary identifier vector, which serves as the compliance identifier.

[0048] Finally, based on the work standard template, the limb movement features are used to identify violations during the work process to obtain a violation risk identifier. Specifically, the violation judgment rule set corresponding to the current scenario type is read from the work standard template. This rule set quantifies various violations in the form of spatiotemporal motion patterns. The judgment rule for non-standard safety rope position adjustment is: the horizontal translation distance of the key point of the safety belt relative to the initial attachment position exceeds a preset position deviation threshold and the duration exceeds a preset duration threshold; the judgment rule for throwing objects from a height is: the absolute value of the downward component of the instantaneous velocity of the wrist joint in the vertical direction exceeds a preset velocity upper limit threshold and is accompanied by a sharp change in the velocity modulus of the wrist joint. In the continuous limb movement feature sequence obtained by temporal correlation motion analysis, violation pattern matching calculation is performed on each frame using a sliding window method. For identifying violations related to improper safety rope position adjustment, the magnitude of the spatial displacement vector of each sampling point of the safety belt relative to the initial attachment moment is extracted from the three-dimensional sampling point coordinate sequence of the safety belt strap obtained from the compliance inspection. The system detects periods where the magnitude value consistently exceeds a preset position deviation threshold. Simultaneously, the strap position change sequence is linked to the coordinates of joint points on the torso surface to determine whether the offset is a linked displacement within the normal range of body movement or an abnormal sliding displacement independent of torso movement. If it is determined to be an abnormal sliding displacement, the frame is marked as a suspected violation of improper safety rope position adjustment. For identifying violations related to objects thrown from heights, the component of the instantaneous velocity vector of the wrist joint in the vertical direction of the world coordinate system is extracted from limb movement features. The system detects whether this downward vertical component exceeds a preset velocity upper limit threshold. Simultaneously, the rate of change of the magnitude of the instantaneous velocity vector of the wrist joint at that moment within each preceding and following frame window is calculated to detect any abrupt stops or rebounds in hand movement. If the downward vertical velocity component exceeds the threshold and the rate of change of the velocity magnitude exceeds a preset rate of change threshold, the frame is marked as a suspected violation of objects thrown from heights. The aforementioned suspected violation moments are confirmed and judged within a continuous time window. If more than a preset proportion of frames within the window are marked as suspected violation moments of the same type of behavior, then the violation is finally confirmed, and the start frame, end frame, and duration of the violation are recorded. The real-time judgment results of various violations are combined into a multi-dimensional binary identifier vector according to the violation type. This vector is the violation risk identifier of the operation process. Each dimension in the vector corresponds to a predefined type of violation. If the violation is confirmed to have occurred in the current frame group, it is marked as a violation; otherwise, it is marked as no violation.

[0049] Step S4: Conduct a compliance assessment of high-altitude operations based on the compliance data of the preparation actions and the violation risk indicators, and generate a high-altitude operations training score report.

[0050] First, the completion rate of preparation actions is calculated based on the compliance data and the compliance identification of the protective equipment worn during the work preparation phase. Specifically, the compliance identification of the protective equipment worn during the work preparation phase is considered as the compliance judgment result of the protective equipment worn during the work preparation phase, and is combined with the compliance identification of each work preparation action already included in the compliance data of preparation actions. Specifically, the total number of all mandatory preparation action items specified for the current scenario type is read from the work standard template. This total number includes protective equipment worn preparation items and other work preparation action items. Taking the scenario of climbing utility poles as an example, the mandatory preparation action items include wearing a safety helmet, wearing work clothes, wearing gloves, wearing a safety belt, environmental monitoring actions, placing work fences and warning signs, and attaching safety ropes, totaling seven items. The four dimensions in the compliance identification of the protective equipment worn during the work preparation phase are combined and arranged with the various dimensions in the compliance data of preparation actions to form a complete set of compliance identification of preparation actions. The dimensions marked as compliant in this complete set are counted to obtain the number of compliant action items. The number of compliant action items is divided by the total number of mandatory preparation action items to obtain the completion rate of preparation actions. The results of security equipment wear detection and the timing matching results of preparation actions obtained from other sources are summarized in a unified manner under the same scoring dimension, so as to provide a quantitative indicator that comprehensively reflects the completeness of the operator's pre-operation preparation for subsequent score calculation.

[0051] Then, based on the violation risk identifiers, time-series cumulative statistics are performed to obtain the violation frequency and the number of continuous violation events for each type of violation. The deduction value for each type of violation is then calculated based on the frame-level violation frequency and the number of continuous violation events to obtain the total violation monitoring deduction. Specifically, the output violation risk identifier is a binary identifier sequence marked frame by frame. For each type of violation, the violation frequency is calculated as follows: the number of frames marked as violations for that type of violation during the entire operation period is counted, and the counted number of violation frames is divided by the total number of valid frames during the operation period to obtain the frame-level violation frequency for that type of violation. For each type of violation, the specific calculation method for the number of continuous violation events in time sequence is as follows: scan the mark value of the violation in the violation risk identification sequence, define the frame segment that is continuously marked as a violation in time sequence as a continuous violation event, count the length of the continuous frame of the continuous violation event, multiply the length of the continuous frame by the time interval between adjacent frames to obtain the duration of the continuous violation event, if the duration is longer than the preset violation event confirmation duration threshold, then the continuous violation event is included in the number of continuous violation events of the type of violation. After scanning the entire operation period, the statistical value of the number of continuous violation events of each type of violation is obtained.

[0052] After obtaining the frame-level violation frequency and the number of persistent violation events for each type of violation, the specific calculation method for the individual deduction value of the j-th type of violation is as follows: The frame-level violation frequency of this type of violation... Multiply by the preset single-frame violation penalty base In addition, the number of repeated violations of this type of behavior Multiply by the preset single continuous event deduction base The single deduction value for the j-th type of violation is obtained. The above calculation relationship can be expressed as: in, This represents the individual point deduction for the j-th type of violation, in points. The frame-level violation frequency of the j-th type of violation is represented by a dimensionless real number that is greater than or equal to 0 and less than or equal to 1. This indicates the preset penalty score for violations in a single frame, in points. The number of consecutive violation events for the j-th type of violation is a non-negative integer. This represents the preset base score for a single continuous event, expressed in points. After calculating the above deduction value for each predefined type of violation, the individual deduction values ​​for all types of violations are summed to obtain the total violation monitoring deduction. The purpose of this operation is to cumulatively quantify the frame-by-frame violation risk indicators in a temporal dimension, transforming discrete frame-level violation markers into continuous deduction indicators that reflect the severity of violations, thus providing a basis for the quantitative assessment of operational safety.

[0053] Next, a score is calculated and weighted summed based on the completion rate of the preparatory actions and the total deductions for violations, to obtain the comprehensive score for high-altitude operations. Specifically, first, the completion rate of the preparatory actions is multiplied by a preset total score for the preparatory actions section to obtain the preparatory actions score; then, the total deductions for violations are deducted from the preset total score for the violation monitoring section. If the result after deduction is less than zero, the violation monitoring score is set to zero; otherwise, the deduction result is retained as the violation monitoring score. The preparatory actions score and the violation monitoring score are added together to obtain the comprehensive score for high-altitude operations. The above calculation relationship can be expressed as: in, This indicates the overall score for high-altitude operations, expressed in points. This represents the completion rate of the preparatory action, and is a dimensionless real number greater than or equal to 0 and less than or equal to 1. This indicates the total score for the preparatory actions section, in points. This represents the total score for the pre-set violation monitoring section, in points. This indicates the total number of points deducted for violations of monitoring regulations, expressed in points. Indicates taking 0 and The larger of the two values ​​is taken as the actual score for the violation monitoring section. Taking the scenario of climbing utility poles as an example, if the preset total score for the preparatory actions is 40 points and the total score for the violation monitoring section is 60 points, and a worker's preparatory action completion rate is 7 / 7 (1.0), then their preparatory action score is 40 points. If their total violation monitoring deduction is 15 points, then their violation monitoring score is 60 minus 15 equals 45 points. The overall score for working at height is 40 plus 45 equals 85 points. By weighting and summing these two different evaluation indicators—preparatory action completion rate and total violation monitoring deduction—and mapping them to a percentage-based scoring system, a quantitative score that can intuitively reflect the overall performance of the practical training exam is formed.

[0054] Finally, the compliance data of the preparation actions, the violation risk indicators, and the comprehensive score of the high-altitude operation are combined into a high-altitude operation training score report according to the preset report format. Specifically, a preset training score report format template is read from the system, which specifies the structural framework of the report and the location for filling in various data. The specific names of the preparation actions and their compliance indicators corresponding to each dimension in the compliance data of the preparation actions obtained in step S3 are filled into the preparation action details table of the report. The names of the security equipment and their compliance indicators corresponding to each dimension in the compliance indicators are filled into the security equipment wearing details table of the report. The judgment results of various violations in the violation risk indicators obtained in step S3, as well as the frame-level violation frequency, number of continuous violation events, and single deduction value of various violations obtained in step S4, are filled into the violation behavior details table of the report. The calculated comprehensive score of the high-altitude operation is filled into the total score column of the report. The above detailed tables and total score columns are arranged and combined according to the preset report format template to form a complete structured high-altitude operation training scoring report. The report covers all the evaluation details of the three dimensions of safety equipment wearing, preparatory action execution and operation process safety, as well as the final quantitative scoring results.

[0055] In an optional implementation, the method further includes: After step S3 verifies the compliance of preparatory actions and identifies violations in the operation process based on the limb joint coordinate set and operation standard template to obtain compliance data for preparatory actions and violation risk indicators for the operation process, an optional real-time alarm processing flow is further included.

[0056] First, motion pattern features corresponding to high-risk actions are extracted from the limb movement features, and these motion pattern features are matched with a preset high-risk feature set to generate a risk signal. Specifically, for the limb movement features at each frame, based on the key kinematic parameter types specified for various high-risk actions in the high-risk feature set, the corresponding parameter values ​​are extracted from the limb movement features to form the motion pattern feature vector for that frame. Taking the identification of high-risk actions such as throwing objects from a height as an example, the key kinematic parameter types corresponding to this type of action are read from the high-risk feature set as the component value of the instantaneous velocity vector of the right wrist joint in the vertical direction of the world coordinate system and the rate of change value of the magnitude of the instantaneous velocity vector of the right wrist joint in several frame windows before and after. These two parameter values ​​are extracted from the limb movement features at the current frame to form the motion pattern feature vector used for high-risk object throwing at that frame. Taking the identification of high-risk actions such as abnormal slippage of the safety rope attachment point as an example, the key kinematic parameters corresponding to this type of action are read from the high-risk feature set. These parameters are the modulus sequence of the spatial displacement vector of each three-dimensional sampling point on the safety belt strap path relative to the initial attachment moment, and the difference sequence of the modulus sequence relative to the normal displacement of the torso surface. The above parameter values ​​are extracted from the limb motion features and safety belt strap position monitoring data at the current frame moment to form the motion pattern feature vector for judging abnormal slippage of the safety rope attachment point at that frame moment.

[0057] After obtaining the motion pattern feature vector of the current frame, it is matched with the corresponding high-risk action judgment vector in the high-risk feature set. For the parameter value of each dimension in the motion pattern feature vector, the difference between the parameter value and the judgment threshold of the corresponding dimension in the high-risk feature set is calculated, and the absolute value and direction of the difference are used to determine whether the dimension exceeds the preset alarm trigger threshold. When the number of dimensions in the motion pattern feature vector that exceed the preset alarm trigger threshold reaches the number of alarm trigger dimensions specified in the high-risk feature set, the frame is marked as a high-risk frame. The proportion of frames marked as high-risk frames within a continuous time window is statistically analyzed. If this proportion exceeds the preset high-risk confirmation proportion threshold, a high-risk action type label corresponding to the frame is generated as a risk signal. This risk signal contains two fields: a high-risk action type code and a risk level value. Taking the scenario of climbing utility poles as an example, if the vertical downward velocity component of the right wrist joint exceeds the preset parabolic velocity alarm threshold in the high-risk feature set in the motion pattern feature vector extracted at a certain frame, and the velocity modulus change rate exceeds the preset emergency stop change rate alarm threshold, and more than 60% of the frames in the continuous time window meet the above conditions, then a risk signal with a type code corresponding to high-altitude parabolic throwing and a risk level value of the highest level is generated.

[0058] Finally, when the risk signal is a preset high-risk signal, an alarm is triggered according to the preset alarm method. Specifically, the risk level value in the generated risk signal is compared with the system's preset high-risk level threshold. If the risk level value is greater than or equal to the high-risk level threshold, the risk signal is determined to be a high-risk signal, triggering the alarm execution process. The preset alarm methods include two parallel methods: on-site audible and visual alarm and remote monitoring terminal alarm. The specific execution process of the on-site audible and visual alarm method is as follows: the server sends an alarm drive command to the audible and visual alarms installed around the high-altitude operation training site through the data transmission network. The alarm drive command contains pre-coded alarm light flashing frequency parameters and alarm tone frequency parameters. After receiving the alarm drive command, the audible and visual alarm lights up a red warning light according to the specified flashing frequency and emits an intermittent buzzing sound according to the specified tone frequency until the server confirms that the high-risk action corresponding to the frame time will no longer be marked as a high-risk frame in subsequent continuous time windows, and then sends an alarm cancellation command. The specific execution process of the remote monitoring terminal alarm method is as follows: Based on the high-risk action type code and risk level value contained in the generated risk signal, the server generates an alarm message containing a timestamp, worker identifier, high-risk action type, risk level value, and on-site image index according to a preset alarm information format template. This alarm message is then pushed to the remote monitoring terminal device via the data transmission network. Upon receiving the alarm message, the remote monitoring terminal device displays the alarm content in a pop-up window and highlights the text in red on the monitoring interface. Simultaneously, it triggers vibration and an alarm sound to attract the attention of monitoring personnel. The on-site audible and visual alarm and the remote monitoring terminal alarm are executed independently. An alarm triggered by either method is considered a successful alarm. The simultaneous execution of both methods provides redundancy, ensuring that alarm information is promptly received by both on-site workers and remote monitoring personnel. This real-time alarm processing flow runs independently of the training score report generation flow in step S4. Its purpose is to provide real-time response to high-risk actions with immediate personal injury risks or major safety hazards. During high-altitude operation training, it alerts on-site workers and remote monitoring personnel to take emergency avoidance measures through audible and visual alarms and remote notifications before a safety accident occurs, thereby preventing sudden safety accidents during the training process.

[0059] In one embodiment, a high-altitude operation behavior recognition device based on multimodal perception is provided. This high-altitude operation behavior recognition device 2 corresponds one-to-one with the high-altitude operation behavior recognition method based on multimodal perception in the above embodiments. For example... Figure 3 As shown, the high-altitude operation behavior recognition device 2 based on multimodal perception may include: a spatiotemporal synchronization module 21, an operation recognition module 22, a compliance verification module 23, and an operation evaluation module 24. Detailed descriptions of each functional module are as follows: The spatiotemporal synchronization module 21 is used to perform time synchronization and spatial pose calibration processing on the acquired multi-view video streams to obtain spatiotemporally aligned perception data frame groups. The job recognition module 22 is used to perform job scene matching and multi-view three-dimensional positioning processing based on the perception data frame group to obtain the coordinate set of the limb joint points of the worker and the corresponding job standard template. The compliance inspection module 23 is used to perform compliance verification of preparatory actions and identification of violations in the operation process based on the limb joint point coordinate set and the operation standard template, and to obtain compliance data of preparatory actions and violation risk identification of the operation process. The work assessment module 24 is used to conduct a compliance assessment of high-altitude operations based on the compliance data of the preparation actions and the violation risk indicators, and generate a high-altitude operation training score report.

[0060] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0061] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0062] This application also provides a computer device, such as... Figure 4 As shown, the computer device includes: at least one processor 32, a memory 31, and a computer program stored in the memory 31 and executable on the at least one processor 32. When the processor 32 executes the computer program, it implements the steps in any of the above method embodiments, or when the processor 32 executes the computer program, it implements the functions of each module / unit in the above device embodiments.

[0063] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 31 and executed by the processor 32 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the computer device.

[0064] Those skilled in the art will understand that Figure 4 The computer device described is merely an example and does not constitute a limitation on the computer device. It may include more or fewer components than shown, or combine certain components, or different components. For example, the computer device may also include input / output devices, network access devices, buses, etc.

[0065] The processor 32 mentioned above can be a Central Processing Unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs). Programmable Gate Array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor 32 can be a microprocessor or any conventional processor, etc.

[0066] The memory 31 can be an internal storage unit of the computer device, such as a hard drive or memory. The memory 31 can also be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 31 can include both internal and external storage units of the computer device.

[0067] This application also provides a readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0068] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0069] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0070] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0071] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0072] In the embodiments provided in this application, it should be understood that the disclosed apparatus / devices and methods can be implemented in other ways. For example, the apparatus / device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0073] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0074] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for recognizing high-altitude work behavior based on multimodal perception, characterized in that, The method includes: The acquired multi-view video streams are processed for time synchronization and spatial pose calibration to obtain a group of perceptual data frames with spatiotemporal alignment. Based on the perception data frame group, the operation scene is matched and multi-view three-dimensional positioning is processed to obtain the coordinate set of the limb joint points of the operator and the corresponding operation standard template. Based on the set of limb joint coordinates and the standard work template, the compliance of preparatory actions is verified and violations in the work process are identified to obtain compliance data of preparatory actions and violation risk indicators of the work process. Based on the compliance data of the preparation actions and the violation risk indicators, a compliance assessment of high-altitude operations is conducted, and a high-altitude operations training score report is generated.

2. The method for recognizing high-altitude work behavior based on multimodal perception as described in claim 1, characterized in that, The process of performing time synchronization and spatial pose calibration on the acquired multi-view video streams to obtain spatiotemporally aligned perceptual data frame groups includes: Based on the timestamp, the acquired multi-view video streams are time-synchronized and aligned to obtain the synchronized view image set corresponding to each acquisition moment. The synchronous viewpoint image set is calibrated by shooting pose according to preset spatial calibration points to obtain the projection parameters corresponding to each viewpoint. Based on the projection parameters, the spatial coordinates of each image in the synchronized viewpoint image set are unified to obtain a corrected standard visual image. The standard visual images are associated and encapsulated according to time sequence to obtain a spatiotemporally aligned perceptual data frame group.

3. The method for recognizing high-altitude work behavior based on multimodal perception as described in claim 2, characterized in that, The step of performing work scene matching and multi-view 3D positioning processing based on the perceived data frame group to obtain the coordinate set of the limb joint points of the worker and the corresponding work standard template includes: Local scene features are extracted from each image in the perception data frame group, and the local scene features are correlated and fused to obtain global scene features; The current job scenario type is identified based on the global scenario features, and the corresponding job standard template is matched from the preset standard template library based on the job scenario type. Target detection of workers and multi-view association matching are performed on each image of the perception data frame group to obtain the detection association data of the same worker in each view, the detection dataset of each worker, and the image region data corresponding to each worker in each view. Based on the detection association data, extract the limb joint image coordinates of each worker from the detection dataset from each viewpoint. Based on the projection parameters, the coordinates of the limb joint points in the image are processed using multi-view three-dimensional positioning to obtain the coordinate set of the limb joint points of each worker.

4. The high-altitude operation behavior recognition method based on multimodal perception as described in claim 3, wherein the preparation action compliance data includes a wearing compliance identifier, characterized in that, The step of verifying the compliance of preparatory actions and identifying violations during the operation process based on the limb joint coordinate set and the operation standard template, and obtaining compliance data for preparatory actions and violation risk indicators for the operation process, includes: By performing temporal correlation motion analysis on the set of limb joint coordinates through a preset continuous time window, the limb movement characteristics of each worker can be obtained. The preparatory action compliance is verified based on the aforementioned work standard template to obtain preparatory action compliance data. Based on the aforementioned operational standard template, the image area data is used to detect the wearing of security equipment, thereby obtaining the compliance identification of each operator's wearing of the equipment. Based on the aforementioned work standard template, the limb movement characteristics are used to identify violations during the work process, thereby obtaining a violation risk indicator for the work process.

5. The method for recognizing high-altitude work behavior based on multimodal perception as described in claim 4, characterized in that, The step of conducting a compliance assessment of high-altitude operations based on the compliance data of the preparatory actions and the violation risk indicators, and generating a high-altitude operations training score report, includes: The completion rate of the preparation actions is obtained by statistically analyzing the compliance data of the preparation actions and the compliance identification of the wearer. Based on the violation risk identifier, time-series cumulative statistics are performed to obtain the violation frequency and the number of continuous violation events for various types of violations. Based on the frame-level violation frequency and the number of continuous violation events, the deduction value for various types of violations is calculated to obtain the total violation monitoring deduction. The score is calculated and weighted summed based on the completion rate of the preparatory actions and the total number of points deducted for violations, to obtain the comprehensive score for high-altitude operations. According to the preset report format, the compliance data of the preparation actions, the violation risk identification, and the comprehensive score of the high-altitude operation are combined into a high-altitude operation training score report.

6. The method for recognizing high-altitude work behavior based on multimodal perception as described in claim 4, characterized in that, The method further includes: The movement pattern features corresponding to high-risk actions are extracted from the limb movement features, and the movement pattern features are matched with a preset high-risk feature set to generate a risk signal. When the risk signal is a preset high-risk signal, an alarm is triggered according to the preset alarm method.

7. A high-altitude operation behavior recognition device based on multimodal perception, applied to the high-altitude operation behavior recognition method based on multimodal perception as described in claim 1, characterized in that, The device includes: The spatiotemporal synchronization module is used to perform time synchronization and spatial pose calibration processing on the acquired multi-view video streams to obtain spatiotemporally aligned perception data frame groups. The job recognition module is used to perform job scene matching and multi-view three-dimensional positioning processing based on the perception data frame group to obtain the coordinate set of the limb joint points of the worker and the corresponding job standard template. The compliance verification module is used to verify the compliance of preparatory actions and identify violations in the operation process based on the limb joint point coordinate set and the operation standard template, and to obtain compliance data of preparatory actions and violation risk indicators of the operation process. The work assessment module is used to conduct a compliance assessment of high-altitude operations based on the compliance data of the preparation actions and the violation risk indicators, and generate a high-altitude operation training score report.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the high-altitude operation behavior recognition method based on multimodal perception as described in any one of claims 1 to 6.

9. A readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the high-altitude operation behavior recognition method based on multimodal perception as described in any one of claims 1 to 6.

10. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the high-altitude operation behavior recognition method based on multimodal perception as described in any one of claims 1 to 6.

Images