An image recognition-based hoisting operation dynamic risk grading and early warning system

Abstract

The application discloses a kind of based on image recognition's hoisting operation dynamic risk grading early warning system, it is related to operation safety early warning technical field, including: acquisition operation area continuous image, and establish image to ground coordinate mapping;To reference image carries out target detection, contour segmentation and cross-frame association, identifies hook, hoist load, personnel and fixed obstacle boundary, obtains corresponding trajectory, determines hoisting monitoring state based on hook and hoist load position corresponding relationship, continuous frame stability and hoist load displacement state;In hoisting state, according to hoist load trajectory, size and swing information generate dynamic danger zone that changes with motion direction, and revise in combination with fixed obstacle boundary;Then according to personnel trajectory, spatial distance, approach relationship and danger zone residence length of time calculates personnel risk value.The application solves static area and motion envelope mismatch by dynamic danger zone, enhances collision risk perception by boundary revision, realizes continuous evaluation by multidimensional spatiotemporal risk quantification.

Description

Technical Field

[0001] This invention relates to the field of operational safety early warning technology, and in particular to a dynamic risk classification and early warning system for hoisting operations based on image recognition. Background Technology

[0002] In the field of hoisting operation safety monitoring, traditional protection methods mainly rely on physical isolation, manual observation, and area limiting technologies based on sensors such as lasers and radar. In recent years, with the rapid development of computer vision and image recognition technologies, image processing-based target detection and tracking methods have been gradually applied to hoisting operation scenarios. Conventional solutions typically use monocular or multi-view cameras to acquire images of the work area, identify the positions of the hook, load, and surrounding personnel through target detection algorithms, and trigger audible and visual alarms based on preset fixed safety distance thresholds or static area boundaries. These methods provide a fundamental technical path for improving the automation and safety level of hoisting operations, and to some extent reduce the risk of human oversight.

[0003] However, conventional image monitoring methods typically employ a single, static hazard zone setting strategy in practical engineering applications, which does not change with the operational status. On the one hand, fixed safety zone boundaries are difficult to adapt to the dynamic motion envelope generated by the load's size, speed, and spatial sway during hoisting, resulting in a significant mismatch between the hazard coverage and the actual risk level. On the other hand, existing methods lack effective temporal correlation analysis of the spatial approach behavior between personnel and moving loads, as well as the duration of personnel's continuous stay within the hazard zone, leading to insufficient continuity and accuracy in risk assessment results. Summary of the Invention

[0004] In view of the aforementioned existing problems, the present invention is proposed.

[0005] Therefore, this invention provides an image recognition-based dynamic risk classification and early warning system for hoisting operations to address the problems in existing technologies, such as the difficulty in dynamically adjusting the safety monitoring area based on the real-time movement status of the hoist and surrounding environmental obstacles, and the lack of continuous and quantitative assessment methods for the risks of complex spatiotemporal interactions between personnel and hoist.

[0006] To solve the above-mentioned technical problems, the present invention provides the following technical solution: This invention provides a dynamic risk classification and early warning system for hoisting operations based on image recognition. The system includes: acquiring continuous images of the hoisting operation area; performing distortion correction, perspective correction, and ground benchmark calibration on the continuous images to obtain a benchmark monitoring image; and establishing a mapping relationship between image coordinates and ground coordinates; performing target detection, contour segmentation, and cross-frame association on the benchmark monitoring image to identify the boundaries of the hook, load, personnel, and fixed obstacles, obtaining the hook trajectory, load trajectory, and personnel trajectory; and determining the hoisting monitoring state based on the positional correspondence between the hook center and the upper edge of the load, continuous frame stability, and load displacement state; generating a dynamic danger zone that changes with the direction of load movement based on the load trajectory, load size, and swing information, and correcting the dynamic danger zone in conjunction with the fixed obstacle boundaries; calculating the personnel risk value based on the personnel trajectory, the spatial distance and proximity relationship between the personnel and the load, and the duration of residence in the danger zone; continuously determining the personnel risk value in a time sequence; and outputting graded early warning information or intervention control information.

[0007] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the step of performing distortion correction, perspective correction, and ground reference calibration on continuous images includes: acquiring continuous images of the hoisting operation area; performing radial distortion correction and tangential distortion correction on the continuous images based on the intrinsic parameters and distortion parameters of the image acquisition device to obtain a distortion-corrected image; setting ground reference points on the ground of the hoisting operation area, acquiring the image coordinates of each ground reference point in the distortion-corrected image and its actual coordinates in the ground coordinate system; calculating the perspective transformation matrix based on the image coordinates and actual coordinates of each ground reference point, and using the perspective transformation matrix to perform perspective correction on the distortion-corrected image to obtain a perspective-corrected image; using a fixed reference point on the ground of the hoisting operation area as the origin of the ground coordinate system, and using the main running direction of the hoisting load as the reference direction of the ground coordinate system, performing ground reference calibration on the perspective-corrected image to obtain a reference monitoring image.

[0008] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the establishment of the mapping relationship between image coordinates and ground coordinates includes: extracting the midpoint of the bottom edge of the circumscribed rectangle of the personnel target as the personnel ground projection reference point, and extracting the midpoint of the bottom edge of the outline of the hoisted target as the hoisted ground projection reference point; obtaining the image coordinates of the personnel ground projection reference point and the hoisted ground projection reference point in the reference monitoring image; and performing coordinate transformation processing on the image coordinates based on the perspective transformation matrix to make the image coordinates correspond to the ground coordinates in a unified ground coordinate system; the perspective transformation matrix is ​​obtained by corresponding the image coordinates of the ground reference point in the reference monitoring image and the actual coordinates of the ground reference point in the ground coordinate system.

[0009] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the target detection, contour segmentation, and cross-frame association of the reference monitoring image include: performing target detection on the reference monitoring image to extract hook targets, hoisting targets, personnel targets, and fixed obstacle boundary targets; performing contour segmentation on the hook targets, hoisting targets, and personnel targets respectively to obtain hook contours, hoisting contours, and personnel contours, and extracting boundary contour point sets for fixed obstacle boundary targets; establishing candidate matching relationships between targets in the current frame and targets in the previous frame based on the overlap of the bounding rectangles of similar targets between adjacent frames; filtering candidate matching relationships based on the displacement of the center points of similar targets between adjacent frames, and eliminating target matching pairs that do not meet the displacement constraint conditions; performing cross-frame association on target matching pairs that meet the overlap and displacement constraint conditions to generate hook trajectories, hoisting trajectories, and personnel trajectories respectively.

[0010] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the specific steps for identifying the boundaries of the hook, load, personnel, and fixed obstacles to obtain the hook trajectory, load trajectory, and personnel trajectory are as follows: Extracting the center of the hook contour and obtaining the image coordinates of the hook center; extracting the center of the upper edge and the midpoint of the bottom edge of the load contour and obtaining the image coordinates of the center of the upper edge and the midpoint of the bottom edge of the load, respectively; extracting the midpoint of the bottom edge of the bounding rectangle of the personnel contour and obtaining the image coordinates of the midpoint of the bottom edge of the personnel; extracting the boundary contour point set of the fixed obstacle boundary target to obtain the fixed obstacle boundary contour data; performing cross-frame association of the hook center image coordinates in consecutive frames to obtain the hook trajectory; performing cross-frame association of the load bottom edge midpoint image coordinates in consecutive frames to obtain the load trajectory; and performing cross-frame association of the personnel bottom edge midpoint image coordinates in consecutive frames to obtain the personnel trajectory.

[0011] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the specific steps for determining the hoisting monitoring status based on the positional correspondence between the hook center and the upper edge of the load, continuous frame stability, and load displacement state are as follows: Calculate the positional correspondence distance between the image coordinates of the hook center and the image coordinates of the center of the upper edge of the load; when the hook center is located above the center of the upper edge of the load and the positional correspondence distance is less than a distance threshold, determine that the hook and load satisfy the positional correspondence relationship; and calculate the continuous frame stability coefficient of the hook target based on the successful association of the hook trajectory within continuous analysis frames. When the stability coefficient of consecutive frames is greater than the stability threshold, the hook trajectory is determined to meet the consecutive frame stability. Based on the successful association of the hoist trajectory within consecutive analysis frames, the consecutive frame stability coefficient of the hoist target is calculated, and when the consecutive frame stability coefficient is greater than the stability threshold, the hoist trajectory is determined to meet the consecutive frame stability. The displacement of the hoist between adjacent frames is calculated, and if the displacement is greater than the displacement threshold, the hoist is determined to be in an effective displacement state. When the hook and hoist satisfy the position correspondence, the hook trajectory meets the consecutive frame stability, the hoist trajectory meets the consecutive frame stability, and the hoist is in an effective displacement state, the hoisting device is determined to be in the monitoring state.

[0012] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the generation of a dynamic hazard zone that changes with the direction of hoisting movement includes: mapping the hoisting ground projection reference point in the hoisting trajectory to a ground coordinate system to obtain the hoisting ground projection trajectory; calculating the hoisting movement direction and hoisting projection speed based on the hoisting ground projection trajectory; extracting hoisting size information based on the hoisting contour to obtain the equivalent size of the hoisting; calculating hoisting swing information based on the hoisting ground projection trajectory to obtain the hoisting swing offset; calculating the hazard zone expansion coefficient based on the equivalent size of the hoisting, the hoisting projection speed, and the hoisting swing offset; and generating a dynamic hazard zone with the hoisting ground projection center as the center and the hoisting movement direction as the major axis direction, based on the hazard zone expansion coefficient.

[0013] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the step of correcting the dynamic hazard zone by combining fixed obstacle boundaries includes: extracting the boundary contour point set of the fixed obstacle and the boundary point set of the dynamic hazard zone; calculating the minimum distance between the boundary point set of the dynamic hazard zone and the boundary contour point set of the fixed obstacle based on the boundary contour point set of the fixed obstacle and the boundary point set of the dynamic hazard zone, and obtaining the minimum boundary distance; calculating the boundary approximation factor based on the minimum boundary distance and the boundary reference distance threshold; correcting the major axis radius of the dynamic hazard zone based on the boundary approximation factor, and obtaining the corrected major axis radius; keeping the center of the dynamic hazard zone and the hoisting movement direction unchanged, and obtaining the corrected dynamic hazard zone based on the corrected major axis radius.

[0014] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the calculation of personnel risk values ​​based on personnel trajectory, spatial distance between personnel and hoisting load, proximity relationship, and dwell time in the danger zone includes: mapping the personnel ground projection reference point in the personnel trajectory to the ground coordinate system to obtain the personnel ground projection coordinates; calculating the spatial distance between personnel and hoisting load based on the personnel ground projection coordinates and the hoisting load ground projection center coordinates; extracting the personnel movement direction vector from the personnel trajectory, extracting the hoisting load movement direction vector from the hoisting load trajectory, and calculating the personnel proximity factor based on the angle between the personnel movement direction vector and the hoisting load movement direction vector; determining whether personnel are in a danger zone dwelling state based on the positional relationship between the personnel ground projection coordinates and the dynamic danger zone, and calculating the continuous dwell time of personnel in the dynamic danger zone or within the boundary zone of the dynamic danger zone to obtain a dwell time normalization factor; and calculating the personnel risk value based on the spatial distance, proximity factor, and dwell time normalization factor.

[0015] As a preferred embodiment of the image recognition-based dynamic risk classification and early warning system for hoisting operations described in this invention, the specific steps for continuously determining personnel risk values ​​and outputting graded early warning information or intervention control information are as follows: Time-smoothing processing is performed on personnel risk values ​​at consecutive time intervals to obtain smoothed risk values; the smoothed risk values ​​are compared with attention-level risk thresholds, alert-level risk thresholds, and intervention-level risk thresholds to obtain the over-limit status of each corresponding risk threshold; the over-limit count values ​​corresponding to each risk threshold within a time window are statistically analyzed, and the risk level corresponding to the personnel is determined based on the comparison result between the over-limit count value and the minimum effective over-limit frame number of the corresponding level; when the personnel reach the attention-level risk level, attention-level early warning information is output; when the personnel reach the alert-level risk level, alert-level early warning information is output; when the personnel reach the intervention-level risk level, intervention control information is output.

[0016] The beneficial effects of this invention are as follows: by generating a dynamic danger zone that changes with the direction of the load's movement, the problem of mismatch between the static area and the actual motion envelope is solved; by combining the dynamic danger zone with the boundary of fixed obstacles, the coverage of the danger zone is automatically enhanced when the load approaches structures such as fences and columns; by calculating the personnel risk value, a continuous quantitative assessment of the multi-dimensional spatiotemporal risk between personnel and moving loads is achieved; and by performing temporal smoothing and graded over-limit counting judgment on the risk value, instantaneous false detection fluctuations are suppressed. Attached Figure Description

[0017] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is a schematic diagram of a dynamic risk classification and early warning system for hoisting operations based on image recognition.

[0019] Figure 2 A flowchart for determining the status of hoisting monitoring.

[0020] Figure 3 A flowchart for performing cross-frame association.

[0021] Figure 4 A flowchart for outputting tiered early warning information or intervention and control information.

[0022] Figure 5 This is a comparison chart showing the adaptive coverage effect of the dynamic hazard zone on the motion envelope of the hoisting load.

[0023] Figure 6 A comparison chart showing the effects of continuous quantification and graded early warning of spatiotemporal risks to personnel. Detailed Implementation

[0024] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0025] Many specific details are set forth in the following description in order to provide a full understanding of the invention. However, the invention may also be practiced in other ways different from those described herein, and those skilled in the art can make similar extensions without departing from the spirit of the invention. Therefore, the invention is not limited to the specific embodiments disclosed below.

[0026] Secondly, the term "one embodiment" or "embodiment" as used herein refers to a specific feature, structure, or characteristic that may be included in at least one implementation of the present invention. The phrase "in one embodiment" appearing in different places in this specification does not necessarily refer to the same embodiment, nor is it a single or selective embodiment that is mutually exclusive with other embodiments.

[0027] Reference Figures 1-6 This is one embodiment of the present invention, which provides a dynamic risk classification and early warning system for hoisting operations based on image recognition, including the following steps: The image correction module acquires continuous images of the hoisting operation area, performs distortion correction, perspective correction, and ground benchmark calibration on the continuous images to obtain benchmark monitoring images, and establishes a mapping relationship between image coordinates and ground coordinates.

[0028] Specifically, by using image acquisition devices installed around the hoisting operation area, continuous imaging is performed on the hoisting operation area, the area directly below the hook, and the area where personnel are active, thereby obtaining continuous images of the hoisting operation area.

[0029] Furthermore, at least two fixed image acquisition devices are installed around the hoisting operation area. One image acquisition device faces the main running path of the hoist, and the other image acquisition device faces the personnel activity area and the hoisting intersection area. The installation height, pitch angle and horizontal orientation of each image acquisition device are kept fixed after installation to avoid the subsequent drift between the correspondence between image coordinates and ground coordinates.

[0030] Furthermore, the images output by each image acquisition device are continuously acquired to obtain the original image sequence.

[0031] Among them, the An image acquisition device at time The original image acquired is denoted as .

[0032] in, Indicates the image acquisition device number. Indicates the time of data collection. Indicates the first An image acquisition device at time The raw image data collected.

[0033] Furthermore, the original image sequences output by each image acquisition device are time-synchronized, so that different image acquisition devices output synchronized images corresponding to the same operation time at the same time, obtaining a synchronized continuous image sequence, denoted as . .

[0034] Distortion correction is performed on continuous images to obtain a distortion-corrected image sequence.

[0035] Furthermore, due to radial and tangential distortion in the lens of the image acquisition device, straight boundaries in the working area are curved in the image. Therefore, the synchronized continuous image sequence must first be processed. Perform distortion correction.

[0036] Furthermore, let the ideal normalized coordinates be... The distorted normalized coordinates are The image coordinate distortion relationship is expressed as: ; ; in, .

[0037] in, and These represent the first radial distortion coefficient and the second radial distortion coefficient, respectively. and These represent the first tangential distortion coefficient and the second tangential distortion coefficient, respectively.

[0038] It should be noted that the radial distortion coefficient and The calibration corner coordinates are obtained by pre-capturing images of the calibration board and extracting them. A camera calibration algorithm is then used to solve for the radial deviation between the ideal and actual imaging coordinates using least-squares fitting. The tangential distortion coefficient is also obtained. and The calibration board image was captured in advance, the coordinates of the calibration corner points were extracted, and the tangential offset caused by lens mounting eccentricity was obtained by least squares fitting using a camera calibration algorithm.

[0039] Furthermore, based on the pre-acquired lens intrinsic parameters and distortion parameters, the pixel positions in the original image are reverse-mapped for correction to obtain a distortion-corrected image.

[0040] Among them, the An image acquisition device at time The distortion-corrected image is denoted as .

[0041] Perform perspective correction on the distortion-corrected image to obtain a perspective-corrected image sequence.

[0042] Furthermore, due to the overhead angle between the image acquisition device and the ground, the working area appears as a perspective distortion (objects appear larger when closer and smaller when farther away) in the image. Therefore, the distortion-corrected image... Perform perspective correction.

[0043] Furthermore, multiple ground reference points are pre-set on the ground in the hoisting operation area, and the actual coordinates of each ground reference point in the ground coordinate system are measured. Simultaneously, the image coordinates of each ground reference point in the distortion-corrected image are extracted. .

[0044] Furthermore, based on the image coordinates and actual ground coordinates of the ground reference point, a perspective transformation relationship is established from the image plane to the ground reference plane, expressed as: ; in, Indicates the first The perspective transformation matrix corresponding to each image acquisition device Indicates the scaling factor. Represents image coordinates, Represents the ground reference coordinates.

[0045] It should be noted that the scaling factor By substituting the actual ground coordinates and corresponding image coordinates of the ground reference point into the perspective transformation equation, the perspective transformation matrix is ​​solved. The process involves synchronous normalization, and the reciprocal of the third component of the transformation result is used for calculation during coordinate mapping. The value range is usually greater than 0.

[0046] Furthermore, using the perspective transformation matrix Distortion-corrected images Resampling is performed to obtain a perspective-corrected image.

[0047] Among them, the An image acquisition device at time The perspective-corrected image is denoted as .

[0048] Ground benchmark calibration is performed on the perspective-corrected images to obtain benchmark monitoring images.

[0049] Furthermore, a fixed reference point on the ground in the hoisting operation area is used as the origin of the ground coordinate system, and the main direction of hoisting travel is used as the origin of the ground coordinate system. axial direction, in relation to The direction perpendicular to the ground is taken as the axis direction. Establish a unified ground coordinate system along the axis.

[0050] Furthermore, the perspective transformation matrices corresponding to each image acquisition device are uniformly associated with the ground coordinate system, so that images acquired by different image acquisition devices can be mapped to the same ground coordinate reference after perspective correction.

[0051] Furthermore, the image after perspective correction and ground benchmark calibration will be used as the benchmark monitoring image.

[0052] Among them, the An image acquisition device at time The baseline monitoring image is denoted as .

[0053] Furthermore, to reduce the impact of changes in ambient brightness and static background on subsequent target recognition, a background reference image is established for the perspective-corrected image, represented as: ; in, Indicates the first An image acquisition device at time Background reference image, This represents the background reference image from the previous time step. Indicates the background update coefficient, and .

[0054] It should be noted that the background update coefficient Background modeling was performed by setting multiple candidate values ​​on pre-acquired continuous images of the hoisting operation area, and the selection was based on the evaluation criteria of minimizing the residual in the static background area and maintaining the integrity of the moving target contour.

[0055] Establish a mapping relationship between image coordinates and ground coordinates.

[0056] Furthermore, in order to ensure that the identified locations of cranes and personnel can be uniformly converted to the ground coordinate system, ground projection reference point extraction rules for different targets are pre-set.

[0057] Among them, the midpoint of the bottom edge of the bounding rectangle of the personnel target is used as the reference point for the personnel ground projection, the midpoint of the bottom edge of the hoisting outline of the hoisting target is used as the reference point for the hoisting ground projection, and the center point of the hook outline of the hook target is used as the reference point for the hook image.

[0058] Furthermore, let the first... The coordinates of the reference point of the target class in the image are The corresponding ground coordinates are The mapping relationship from image coordinates to ground coordinates is expressed as: ; in, Indicates the first Image reference point coordinates of the target class Indicates the first Ground coordinates corresponding to the target reference point.

[0059] in, Used to distinguish between personnel targets, hoisted targets, or hook targets.

[0060] The further mapping from image coordinates to ground coordinates does not involve directly converting arbitrary pixels of the target to ground coordinates. Instead, it first extracts reference points that match the actual spatial location of the target, and then uses a perspective transformation matrix. The reference points are converted to a unified ground coordinate system to ensure that subsequent calculations of spatial distance between personnel and the hoisting load, determination of hazardous area coverage, and statistics of dwell time are all carried out under a unified spatial reference.

[0061] The target tracking module performs target detection, contour segmentation, and cross-frame correlation on the baseline monitoring image to identify the boundaries of the hook, load, personnel, and fixed obstacles, obtain the hook trajectory, load trajectory, and personnel trajectory, and determine the hoisting monitoring status based on the positional correspondence between the hook center and the upper edge of the load, the stability of continuous frames, and the load displacement status.

[0062] Furthermore, target detection is performed on the baseline monitoring images to identify the boundaries of hooks, loads, personnel, and fixed obstacles.

[0063] Furthermore, the acquired baseline monitoring images Target detection processing is performed to extract hook targets, hoisted targets, personnel targets, and fixed obstacle boundary targets in the work area.

[0064] Fixed obstacle boundaries include ground fence boundaries, post boundaries, wall boundaries, equipment boundaries, and other stationary structural boundaries within the work area.

[0065] Specifically, when processing the acquired benchmark monitoring image for target detection, the benchmark monitoring image is first subjected to grayscale normalization and noise suppression to reduce the impact of ambient light fluctuations and random noise on the recognition results. Then, the preprocessed benchmark monitoring image is input into the pre-trained target detection model to locate and detect hook targets, load targets, and personnel targets in the image, and output the bounding rectangle, category label, and confidence score of each target. At the same time, for fixed obstacle boundaries, since fixed obstacles usually appear as continuously distributed static structural edges, edge enhancement and region segmentation processing are performed on the benchmark monitoring image to extract the boundary regions corresponding to the outer edges of fences, columns, walls, or equipment, and output the boundary contour point set. Afterwards, based on the category label and confidence threshold output by the target detection model, hook targets, load targets, and personnel targets that meet the preset confidence conditions are retained, and the static structural boundaries obtained from region segmentation are determined as fixed obstacle boundary targets, thereby completing the recognition of hooks, loads, personnel, and fixed obstacle boundaries in the benchmark monitoring image.

[0066] The target detection model includes a feature extraction network, a feature fusion network, and a detection output network. The feature extraction network performs multi-layer convolutional feature extraction on the input benchmark monitoring image to obtain multi-scale feature maps representing edge texture information, local shape information, and high-level semantic information, respectively. The feature fusion network performs upsampling, downsampling, and lateral connection fusion on the multi-scale feature maps at different levels to enhance the joint representation ability of small-sized hook targets, slender load contours, and personnel targets of different scales. The detection output network outputs target position parameters, target category parameters, and target confidence parameters based on the fused feature maps. The target position parameter represents the bounding rectangle position of the hook target, load target, and personnel target in the benchmark monitoring image; the target category parameter distinguishes between hook targets, load targets, and personnel targets; and the target confidence parameter represents the credibility of the corresponding detection result. Preferably, the target detection model also includes an input preprocessing layer and a non-maximum suppression layer. The input preprocessing layer performs size unification and numerical normalization on the benchmark monitoring image, and the non-maximum suppression layer filters out overlapping candidate boxes to retain the final target detection result.

[0067] The training process for the target detection model specifically involves collecting hoisting operation image samples containing hook targets, hoisting targets, and personnel targets. Hook targets, hoisting targets, and personnel targets in each image sample are manually labeled to obtain corresponding target category labels and bounding rectangles, thus constructing a target detection training sample set. Subsequently, the target detection training sample set undergoes data augmentation processing, including image scaling, random cropping, brightness perturbation, contrast perturbation, rotation transformation, and occlusion perturbation, to improve the target detection model's adaptability to different lighting conditions, different shooting scales, and partially occluded scenes. Afterwards, the image samples are processed... The training images after data augmentation are input into the target detection model. A feature extraction network extracts multi-scale features, a feature fusion network fuses these features, and the detection output network outputs the target location parameters, target category parameters, and target confidence parameters of the candidate targets. Then, the target location parameters are compared with the manually labeled bounding rectangles to calculate the location regression loss, the target category parameters are compared with the target category labels to calculate the classification loss, and the target confidence parameters are compared with the actual existence status of the candidate targets to calculate the confidence loss. The location regression loss, classification loss, and confidence loss are then weighted and summed to construct the total loss function, expressed as: ,in, Represents the total loss function. This represents the position regression loss. Represents classification loss. Indicates confidence loss. , and These represent the weight coefficients of the corresponding loss terms. Finally, based on the total loss function, the backpropagation method is used to iteratively update the network parameters of the target detection model until the total loss function converges or the detection accuracy of the model on the validation sample set reaches the preset requirements, thus obtaining the trained target detection model.

[0068] It should be noted that the weight coefficients of the corresponding loss terms are determined by training multiple sets of candidate parameters on pre-labeled hoisting device scene training samples, and by iteratively optimizing the comprehensive optimality of target detection accuracy, false negative rate and positioning error on the validation set as the evaluation criterion. The weight coefficients of each loss term are usually in the range of 0 to 1, and the sum of each weight coefficient satisfies 1.

[0069] Furthermore, to ensure consistency in subsequent trajectory extraction, a unified detection output format is adopted for various targets in the benchmark monitoring image.

[0070] The detection results for hook targets, load targets, and personnel targets include at least the target's bounding rectangle, target confidence level, and target category identifier; the detection results for fixed obstacle boundaries include at least the boundary region mask and boundary contour point set.

[0071] Furthermore, foreground constraint processing is performed on the benchmark monitoring image; specifically, the acquired background reference image is... The image is differentially analyzed with the current baseline monitoring image to suppress false response areas caused by non-operational targets in the static background, retaining only the foreground response areas related to the hook, load, and personnel, thereby improving the accuracy of target detection results.

[0072] The target detection results are segmented to obtain the hook contour, load contour, personnel contour, and fixed obstacle boundary contour.

[0073] Furthermore, contour segmentation is performed on the detected hook target, load target, and personnel target to obtain the fine contour of each target.

[0074] Among them, the hook contour is used to extract the center of the hook, the load contour is used to extract the upper edge of the load, the lower edge of the load, and the outer area of ​​the load contour, and the personnel contour is used to extract the midpoint of the bottom edge of the personnel outer frame and the personnel contour area.

[0075] Furthermore, boundary contour extraction is performed on the fixed obstacle boundary target to obtain the fixed obstacle boundary contour point set.

[0076] Among them, the fixed obstacle boundary contour point set is used for subsequent danger zone boundary correction and boundary approximation analysis.

[0077] Furthermore, the center point of the hook profile is extracted and denoted as... .

[0078] in, Indicates time Image coordinates of the hook center in the reference monitoring image.

[0079] Furthermore, the center point of the upper edge of the hoisting profile is extracted and denoted as... And extract the midpoint of the bottom edge of the hoist, and record it as... .

[0080] The center point of the upper edge is used to determine the positional correspondence between the hook and the load, while the midpoint of the bottom edge of the load is used to calculate the ground projection coordinates of the load.

[0081] Furthermore, the midpoint of the base edge of the bounding rectangle of the person is extracted from the person's profile and denoted as... .

[0082] in, This indicates the image coordinates of the personnel ground projection reference point in the benchmark monitoring image.

[0083] Cross-frame association is performed on the hook, load, and personnel to obtain the hook trajectory, load trajectory, and personnel trajectory, respectively.

[0084] Furthermore, since the fixed obstacle boundary remains stationary in continuous images, cross-frame association processing is only performed on the hook target, the load target, and the personnel target to construct the hook trajectory, the load trajectory, and the personnel trajectory, respectively.

[0085] Furthermore, the specific steps for constructing the trajectory are as follows: Steps A1-A4: A1. Regarding time and time Calculate the overlap of the circumscribed rectangle and the displacement of the center point among similar targets, and determine the targets that meet the overlap condition and displacement constraint condition as the same continuous target.

[0086] The overlap of the target's circumscribed rectangle is characterized by the intersection-union ratio, and the displacement of the target's center point is used to constrain the cross-frame matching distance.

[0087] A2. Set the time. The The bounding rectangle of the target is ,time The The bounding rectangle of the target is The degree of overlap between the two , represented as: ; in, Indicates time The One goal and one moment The The overlap of the bounding rectangles between targets.

[0088] A3. Set the time. The The coordinates of the center point of each target are Set time The The coordinates of the center point of each target are Then the displacement of the center point between the two , represented as: ; in, Indicates time The One goal and one moment The The displacement of the center point between the targets.

[0089] A4. When the overlap of the target's bounding rectangle The overlap threshold is greater than the center point displacement. When the displacement is less than the center displacement threshold, the time will be... The Each target is associated with a specific time. The In the continuous trajectory corresponding to the target; otherwise, create a new trajectory for the current target or determine that the original trajectory has terminated.

[0090] It should be noted that the overlap threshold is obtained by statistically analyzing the distribution of the overlap of the circumscribed rectangles of the same target across frames in continuous image samples of the pre-labeled hoisting device scene, and selecting the highest trajectory association accuracy as the evaluation criterion. The value range is usually 0~1. The center displacement threshold is obtained by statistically analyzing the distribution of the center point displacement of the same target between adjacent frames in continuous image samples of the pre-labeled hoisting device scene, and combining the image sampling frame rate and the maximum motion speed of the target to ensure that the real same target is not mistakenly split as the evaluation criterion. The value range is usually greater than 0.

[0091] Furthermore, following steps A1-A4 of the trajectory construction process, cross-frame association is performed on the hook target, the load target, and the personnel target to obtain the hook trajectory, the load trajectory, and the personnel trajectory.

[0092] The hook trajectory is denoted as The hoisting trajectory is recorded as Personnel trajectory recorded as .

[0093] Furthermore, the hook trajectory Composed of the center points of the hook at consecutive moments, it is represented as: ; Furthermore, the hoisting trajectory Composed of the midpoints of the bottom edge of the load at consecutive moments, it is represented as: ; Furthermore, the movement of people It is composed of the midpoints of the base sides of the circumscribed rectangle of the personnel at consecutive time points, and is represented as: ; Based on the positional correspondence between the center of the hook and the upper edge of the load, the association state between the hook and the load is determined.

[0094] Furthermore, during normal lifting operations, the hook is positioned above the load, and the center of the hook and the center of the upper edge of the load maintain a relatively stable positional correspondence in the image.

[0095] Therefore, extracting the positional relationship between the center of the hook and the center of the upper edge of the load is used to determine whether the hook and the load are in the same hoisting link.

[0096] Furthermore, let's set the time... The coordinates of the center of the hook image are The image coordinates of the center of the upper edge of the hoist are The corresponding distance between the two positions , represented as: ; in, Indicates time The corresponding distance between the center of the hook and the center of the upper edge of the load.

[0097] Furthermore, when the center of the hook is located above the center of the upper edge of the load, and the position corresponds to a distance... If the distance is less than the threshold, it is determined that the hook and the load satisfy the positional correspondence at the current moment; otherwise, it is determined that the hook and the load do not satisfy the positional correspondence at the current moment.

[0098] It should be noted that the distance threshold is obtained by collecting no less than a fixed number of normal hoisting sample distances between the center of the hook and the center of the upper edge of the load at the actual operation site of the hoisting device, and taking the average value after statistical analysis of the sample distances. The value range is usually 1.05 times to 1.30 times the average distance of the normal hoisting sample distances.

[0099] Based on continuous frame stability, it is determined whether the association state between the hook and the load remains valid.

[0100] Furthermore, to avoid instability in single-frame determination due to short-term occlusion, reflection interference, or local false detection, continuous frame stability determination is introduced for hook targets and load targets.

[0101] Furthermore, let the number of consecutive analysis frames be... For example, 10 frames, then the object At any moment Continuous frame stability coefficient Represented as: ; in, Representation Object Stability coefficients within consecutive analysis frames Representation Object In the The value is 1 if the frame is successfully associated with the current trajectory, and 0 otherwise. Indicates an indicator function.

[0102] It should be noted that the number of consecutive analysis frames is The number of stable and continuous frames of the target under normal tracking in continuous images of hoisting operations is statistically analyzed, and the number of frames that can simultaneously cover the short-term occlusion recovery process and maintain trajectory continuity is determined by combining the image acquisition frame rate.

[0103] Furthermore, when the stability coefficients of both the hook target and the load target are greater than the stability threshold in consecutive analysis frames, it is determined that the hook trajectory and the load trajectory remain stable in consecutive frames; otherwise, it is determined that the consecutive frames are unstable.

[0104] It should be noted that the stable threshold is obtained by statistically analyzing the target association success rate in continuous image samples during normal hoisting of the hoisting device, and taking the low quantile or mean of the target association success rate distribution. The value range is usually 0.70~0.95.

[0105] Based on the load displacement status, determine whether the current load is in an effective motion state.

[0106] Furthermore, to avoid erroneously triggering the hoisting monitoring state when the hook and load are only in a static suspended state, the continuous displacement state of the load trajectory is determined.

[0107] Furthermore, let's set the time... and time The midpoints of the bottom edge of the hoisting are respectively and The displacement of the load between two adjacent frames. , represented as: ; in, This indicates the displacement of the load between two adjacent frames.

[0108] Furthermore, when the displacement of the load is carried over several consecutive frames... If all values ​​are greater than the load displacement threshold, the load is determined to be in an effective displacement state; otherwise, the load is determined not to have entered an effective displacement state.

[0109] It should be noted that the load displacement threshold is obtained by statistically analyzing the load inter-frame displacement samples of the hoisting device in normal static suspension state and normal hoisting state, and taking the boundary value between the two types of displacement distributions or the upper limit of the displacement in static suspension state. The value range is usually 2 to 15 pixels.

[0110] The hoisting monitoring status is determined based on the position correspondence, continuous frame stability, and hoisting displacement status.

[0111] Furthermore, when the center of the hook and the upper edge of the load satisfy the positional correspondence, the hook trajectory and the load trajectory meet the continuous frame stability requirements, and the load meets the effective displacement state, the hoisting device is determined to be in monitoring mode, and the hoisting monitoring status is recorded as follows: .

[0112] Furthermore, when the positional correspondence between the hook center and the upper edge of the load is not satisfied, or the hook trajectory and the load trajectory are unstable within consecutive frames, or the load does not meet the effective displacement state, it is determined that the current lifting device is not in the lifting monitoring state, and the lifting monitoring state flag is recorded as follows. .

[0113] Further hoisting monitoring status indicators , represented as: ; in, This indicates the distance threshold corresponding to the location. Indicates the target of the hook at time The continuous frame stability coefficient, Indicates the time when the hoisted target is... The continuous frame stability coefficient, Indicates the stability threshold. This indicates the load displacement threshold.

[0114] The dynamic correction module generates a dynamic danger zone that changes with the direction of the load movement based on the load trajectory, load size, and swing information during the hoisting monitoring state, and corrects the dynamic danger zone by combining the boundaries of fixed obstacles.

[0115] When the hoisting monitoring status indicator is valid, the hoisting trajectory, hoisting outline, and fixed obstacle boundary are extracted to generate a dynamic hazard zone.

[0116] Furthermore, when the acquired hoisting monitoring status flags At that time, extract the hoisting trajectory at the current moment. The lifting profile and the set of fixed obstacle boundary profile points are used as input data for generating dynamic hazard zones.

[0117] Among them, the hoisting trajectory is used to characterize the continuous motion state of the hoisting, the hoisting profile is used to characterize the spatial size range of the hoisting, and the fixed obstacle boundary profile point set is used for subsequent boundary correction of the dynamic hazard area.

[0118] Furthermore, when hoisting the monitoring status indicator At this time, no dynamic danger zone is generated or the dynamic danger zone result of the previous moment is not updated, thereby avoiding accidental triggering of danger zone expansion in non-hoisting state.

[0119] Map the reference point of the hoisting ground projection in the hoisting trajectory to the ground coordinate system to obtain the hoisting ground projection trajectory.

[0120] Furthermore, regarding the hoisting trajectory Midpoint of the bottom edge of the hoisting at various times Perform ground coordinate mapping to obtain the coordinates of the hoist ground projection center at each time point. .

[0121] Furthermore, based on the established mapping relationship between image coordinates and ground coordinates, the midpoint of the bottom edge of the hoist is mapped to the ground coordinate system, as follows: ; in, Indicates time Coordinates of the center of the suspended ground projection.

[0122] Furthermore, by arranging the coordinates of the hoisted ground projection center at consecutive moments in chronological order, the hoisted ground projection trajectory is obtained, represented as: ; in, This represents the projected trajectory of the suspended load in the ground coordinate system.

[0123] The direction and speed of the hoisting motion are calculated based on the ground projection trajectory of the hoisting equipment.

[0124] Furthermore, based on the changes in the coordinates of the hoist's ground projection center at adjacent time points, the motion direction vector of the hoist at the current time is obtained.

[0125] Among them, time The direction vector of the hoisting motion , represented as: ; in, Indicates time The direction vector of motion of the suspended load in the ground coordinate system.

[0126] Furthermore, the projected velocity of the load at the current moment is calculated based on the load's motion direction vector, and is expressed as: ; in, Indicates time The projected velocity of the load in the ground coordinate system. This indicates the time interval between two adjacent frames.

[0127] Furthermore, when the load movement direction vector When the magnitude of the load movement direction vector is greater than the direction threshold, the direction corresponding to the load movement direction vector is taken as the major axis direction of the dynamic danger zone at the current moment; when the magnitude of the load movement direction vector is less than or equal to the direction threshold, the major axis direction of the previous moment remains unchanged.

[0128] It should be noted that the orientation threshold is determined by statistically analyzing the inter-frame displacement modulus of the projected trajectory of the hoisted device on the ground during normal hoisting. The smallest displacement modulus that can distinguish between the effective motion direction and image jitter disturbance is taken as the orientation threshold, and the value range is usually from 1 pixel per frame to 12 pixels per frame.

[0129] Load dimensions are extracted based on the load profile.

[0130] Furthermore, the minimum bounding rectangle is fitted to the hoisting profile at the current moment to obtain the length and width parameters of the hoisting profile.

[0131] Furthermore, let the length of the longer side of the minimum circumscribed rectangle of the hoisting load be... The length of the shorter side is Then the equivalent size of the hoisting , represented as: ; in, Indicates time Equivalent dimensions of the load, This represents the length of the longer side of the minimum circumscribed rectangle of the hoisting load. This represents the length of the shorter side of the minimum circumscribed rectangle of the hoisting load.

[0132] Furthermore, the equivalent dimensions of the hoisting load Used to characterize the overall occupancy of the load at the current moment, serving as a component of the subsequent dynamic hazard zone foundation radius.

[0133] The hoisting swing information is calculated based on the hoisting ground projection trajectory.

[0134] Furthermore, the load will swing left and right or back and forth relative to the main direction of travel during the actual hoisting process.

[0135] Therefore, the degree of offset of the load relative to the average position within a continuous time window is extracted as the load swing information.

[0136] Furthermore, let the length be... The mean x-coordinate of the center of the suspended ground projection within the sliding time window is The mean of the ordinate is Then the hoisting is at the time Swing offset , represented as: ; in, Indicates time The swing offset of the load, This represents the average x-coordinate of the center of the suspended ground projection within the sliding time window. This represents the mean ordinate of the center of the suspended ground projection within the sliding time window. This indicates the length of the time window for the oscillation analysis.

[0137] Furthermore, the load swing offset The larger the value, the greater the deviation of the load from the short-term average running path of the load swing, and the higher the corresponding degree of danger zone expansion.

[0138] The dynamic hazard zone expansion coefficient is calculated based on the load dimensions, load projection speed, and load swing offset.

[0139] Furthermore, to ensure that the dynamic hazard zone can simultaneously reflect the load size, load movement speed, and load sway, a hazard zone expansion coefficient is constructed. , represented as: ; in, Indicates time The danger zone expansion coefficient, Indicates the dimensional reference quantity. Indicates the speed reference quantity. Indicates the reference quantity of the swing. , , Let these represent the weighting coefficients corresponding to the hoisting dimensions, hoisting projected speed, and hoisting swing offset, respectively, and satisfy the following conditions: .

[0140] It should be noted that the weighting coefficients corresponding to the hoisting size, hoisting projection speed, and hoisting swing offset are obtained by normalizing the correlation between each factor in the historical hoisting sample and the actual dangerous events or manually marked hazard levels, and by weighting the coefficients in combination with risk sensitivity requirements. The values ​​are usually in the range of 0.10 to 0.70.

[0141] Furthermore, when the load size increases, the load projection speed increases, or the load swing offset increases, the danger zone expansion coefficient... The corresponding increase allows the dynamic danger zone to adaptively expand as the lifting operation status changes.

[0142] A dynamic hazard zone is generated based on the hazard zone expansion coefficient, which varies with the direction of the hoisting movement.

[0143] Furthermore, using the suspended ground projection center As the center of the dynamic hazard zone, an elliptical dynamic hazard zone is constructed with the current direction of the load's movement as the major axis of the dynamic hazard zone.

[0144] Furthermore, let the major axis radius of the dynamic hazard zone be... The minor axis radius is Then they are represented as: ; ; in, Indicates time The major axis radius of the dynamic hazard zone Indicates time The minor axis radius of the dynamic hazard zone Indicates the basic safety radius. This represents the minor axis scaling factor, and .

[0145] It should be noted that the short axis proportionality coefficient was obtained by statistically analyzing the proportional relationship between the lateral swing range of the load and the dangerous coverage range along the direction of movement in historical hoisting samples, and calibrating it in conjunction with on-site safety margin requirements.

[0146] Furthermore, using the suspended ground projection center Centered on the ellipse, with the direction angle corresponding to the current direction of motion of the load. Dynamic hazard zones are generated along the major axis of the ellipse. .

[0147] Among them, the direction angle , represented as: ; in, Indicates time The direction of the hoisting motion relative to the ground coordinate system The included angle of the axis.

[0148] Furthermore, dynamic hazard zones In the ground coordinate system, the boundary is determined by the center point, the major axis radius, the minor axis radius, and the orientation angle, so that the dynamic danger zone rotates and expands synchronously with the change of the direction of the hoisting movement.

[0149] The dynamic hazard zone is modified by combining the boundaries of fixed obstacles.

[0150] Furthermore, the hoist may approach fences, posts, walls, or equipment boundaries during actual operation.

[0151] Therefore, the dynamic danger zone generated solely based on the motion state of the hoist itself cannot fully reflect the safety risks under boundary compression scenarios, and it is necessary to modify the dynamic danger zone by combining it with the boundary of fixed obstacles.

[0152] Furthermore, the minimum distance between the dynamic hazard zone boundary point set and the fixed obstacle boundary contour point set is calculated to obtain the minimum boundary distance. , represented as: ; in, Indicates time Minimum distance between the boundary of a dynamic hazard zone and the boundary of a fixed obstacle. Represents the set of boundary points of a dynamic danger zone. Represents the set of points representing the boundary contour of a fixed obstacle. This represents the Euclidean distance between boundary points.

[0153] Furthermore, based on the minimum distance at the boundary Calculate the boundary approximation factor , represented as: ; in, Indicates time Boundary approximation factor, This represents the boundary reference distance threshold.

[0154] Furthermore, when the boundary minimum distance Less than the boundary reference distance threshold At that time, the boundary approximation factor A value greater than 0 indicates that the current dynamic hazard zone is approaching the boundary of a fixed obstacle; when the minimum distance to the boundary is reached... Greater than or equal to the boundary reference distance threshold At that time, the boundary approximation factor This indicates that the current dynamic danger zone is not significantly affected by the fixed obstacle boundary.

[0155] It should be noted that the boundary reference distance threshold is obtained by statistical analysis of the minimum safe distance when the hoisting or danger zone boundary approaches the fixed obstacle boundary in historical hoisting samples, combined with on-site collision avoidance safety margin calibration, and the value range is usually 0.10~1.5 meters.

[0156] Furthermore, using the boundary approximation factor The major axis radius of the dynamic hazard zone is corrected to obtain the corrected major axis radius. , represented as: ; in, Indicates time The corrected major axis radius of the dynamic danger zone This represents the boundary correction coefficient.

[0157] It should be noted that the boundary correction coefficient was obtained by fitting and analyzing the relationship between the risk increase when the danger zone is close to the boundary of a fixed obstacle in historical hoisting samples and the boundary approach factor, and combined with the on-site collision avoidance safety margin calibration.

[0158] Furthermore, using the corrected major axis radius Replace the original major axis radius And keep the center of the danger zone and the direction of the long axis unchanged to obtain the corrected dynamic danger zone.

[0159] Among them, the revised dynamic danger zone has a higher spatial coverage on the side closer to the fixed obstacle boundary, so as to enhance the risk characterization capability of boundary compression scenarios.

[0160] The risk warning module calculates personnel risk values ​​based on personnel trajectory, spatial distance and proximity between personnel and load, and duration of stay in dangerous areas. It then performs continuous time-series judgment on personnel risk values ​​and outputs graded warning information or intervention and control information.

[0161] After the corrected dynamic hazard zone is generated, the personnel trajectory and the ground projection trajectory of the hoist are extracted, the spatial distance between the personnel and the hoist, the personnel approach relationship, and the duration of stay in the hazard zone are calculated to obtain the personnel risk value.

[0162] Furthermore, regarding the obtained personnel trajectories and the acquired ground projection trajectory of the hoisting A joint analysis is conducted to extract the ground projection position of each person at the current moment, the projection center position of the current hoist in the ground coordinate system, and the corrected dynamic danger zone range, which are used as input data for calculating the personnel risk value.

[0163] Furthermore, the midpoint of the base edge of the bounding rectangle of the personnel trajectory is... Perform ground coordinate mapping to obtain the ground projection coordinates of personnel. .

[0164] Furthermore, based on the established mapping relationship between image coordinates and ground coordinates, the midpoint of the base of the bounding rectangle of the person is mapped to the ground coordinate system, as shown below: ; in, Indicates time Image coordinates of the midpoint of the base side of the personnel's circumscribed rectangle in the reference monitoring image. Indicates time The projected coordinates of the personnel in the ground coordinate system.

[0165] Calculate the spatial distance between the personnel and the hoisting load.

[0166] Furthermore, using personnel ground projection coordinates Coordinates of the center of the suspended ground projection The Euclidean distance between them represents the spatial distance between personnel and the load.

[0167] Furthermore, let's set the time... The spatial distance between personnel and the hoisted load is Then it is represented as: ; in, Indicates time The spatial distance between personnel and the hoisted load. Indicates time Personnel ground projection coordinates; Indicates time Coordinates of the center of the suspended ground projection.

[0168] Furthermore, when the spatial distance between personnel and the hoisting load... A decrease indicates that personnel are closer to the hoisting area, corresponding to a higher level of risk.

[0169] The calculation personnel are close to each other.

[0170] Furthermore, relying solely on the spatial distance between personnel and the hoisting load cannot fully reflect whether personnel are approaching a danger zone.

[0171] Therefore, the approach relationship of personnel is further calculated based on the angle between the direction of personnel trajectory and the direction of hoisting movement.

[0172] Furthermore, let's set the time... The personnel movement direction vector is , represented as: ; in, Indicates time The direction vector of movement of personnel in the ground coordinate system.

[0173] Furthermore, let's set the time... The direction vector of the hoisting motion is Then the direction vector of personnel movement Vector of the direction of motion of the hoisting load The angle between , represented as: ; in, Indicates time The angle between the direction of personnel movement and the direction of load movement.

[0174] Furthermore, based on the included angle Structural approach factor , represented as: ; in, Indicates time The personnel proximity factor has a range of values ​​of 100. .

[0175] Furthermore, when the direction of personnel movement is more aligned with the direction of hoisting movement, or when personnel move closer to the danger zone along the direction of hoisting movement, the personnel approach factor increases. Larger; when the direction of personnel movement is significantly different from the direction of hoisting movement, the personnel approach factor is larger. Smaller.

[0176] Calculate the duration of stay in the danger zone.

[0177] Furthermore, to reflect the risk level of continuous exposure of personnel in or near the dynamic hazard zone, the duration of continuous stay of personnel in the modified dynamic hazard zone or the boundary zone of the dynamic hazard zone is further statistically analyzed.

[0178] Furthermore, a buffer distance of 10 meters will be extended outward from the modified dynamic hazard zone boundary to form a boundary zone area; when the ground projection coordinates of personnel are located inside the modified dynamic hazard zone or inside the boundary zone area, it is determined that the personnel are in a hazard zone stay state.

[0179] Furthermore, personnel are required to be at all times Danger zone presence status sign is , represented as: ; in, Indicates time Are personnel currently residing in a danger zone? Indicates time Revised dynamic danger zone Indicates time The revised dynamic hazard zone boundary area.

[0180] Furthermore, let's set the time... The continuous stay of personnel is Then it is represented as: ; in, Indicates time The duration of continuous stay of personnel, This indicates the duration of continuous stay of personnel at the previous moment. This indicates the time interval between two adjacent frames.

[0181] Furthermore, a residence duration normalization factor is constructed based on the continuous residence duration. , represented as: ; in, Indicates time The dwell time normalization factor, This indicates the baseline threshold for dwell time.

[0182] Furthermore, when personnel remain continuously within the modified dynamic hazard zone or its boundary zone, the dwell time normalization factor... Gradually increase; after personnel leave the danger zone, the continuous stay duration Once the time is reset to zero, the corresponding residence time normalization factor will start accumulating again.

[0183] The personnel risk value is calculated based on the spatial distance between personnel and the hoisted load, the proximity of personnel, and the duration of stay in the danger zone.

[0184] Furthermore, to comprehensively characterize the risk level of personnel relative to the hoisting operation status and the exposure status in the hazardous area, spatial distance, proximity relationship, dwell time, and acquired boundary approximation factors are combined to construct a personnel risk value. .

[0185] Furthermore, personnel risk value , represented as: ; in, Indicates time Personnel risk value, Indicates time The corrected major axis radius of the dynamic danger zone Indicates time The spatial distance between personnel and the hoisted load. Indicates time Personnel proximity factor Indicates time The dwell time normalization factor, Indicates time Boundary approximation factor, , , , Let each represent the weight coefficient corresponding to each item, and satisfy the following condition: .

[0186] It should be noted that the weight coefficients corresponding to each item in the personnel risk value are obtained by normalizing the correlation between the spatial distance, proximity relationship, duration of stay in the danger zone and boundary approach of personnel and load in historical hoisting samples and actual risk events or manually marked risk levels, and by weighting calibration in combination with the sensitivity requirements of on-site safety control. The value range of each weight coefficient is usually 0.10 to 0.60.

[0187] Continuous time-series determination of personnel risk values.

[0188] Furthermore, since a single frame image may contain occlusions, reflections, partial loss of the target, or short-term false detections, the risk value for personnel is also considered. Continuous time-series smoothing is performed to reduce the impact of instantaneous fluctuations on the early warning results.

[0189] Furthermore, let's set the time... The smoothing risk value is Then it is represented as: ; in, Indicates time The smoothed risk value, This represents the smoothed risk value from the previous time step. Represents the smoothing coefficient, and .

[0190] It should be noted that the smoothing coefficient was obtained by fitting and analyzing the time-series response relationship between the instantaneous risk value sequence and the actual risk status change in the historical hoisting sample, and calibrated with the criteria of taking into account both the ability to suppress risk fluctuations and the timeliness of early warning response.

[0191] Furthermore, set tiered risk thresholds. , and And satisfy These correspond to the attention-level risk threshold, the alert-level risk threshold, and the intervention-level risk threshold, respectively.

[0192] It should be noted that the risk thresholds for attention level, alert level, and intervention level are obtained by performing segmented statistical analysis on the correspondence between the smoothed risk value distribution in historical hoisting samples and the actual risk events or manually marked risk levels, and are then calibrated level by level in conjunction with the on-site safety control level requirements.

[0193] Furthermore, to determine whether an individual's risk value consistently exceeds the limit over a continuous period, a continuous exceedance count is calculated for each risk threshold level, expressed as: ; in, Indicates personnel at the The number of out-of-limit counts within a continuous statistical time window below the risk level threshold. Indicates the number of consecutive frames counted. Indicates the first Level risk threshold.

[0194] Furthermore, when the count value exceeds the limit Greater than or equal to the first level minimum effective number of frames When the current personnel are deemed to have reached the attention level of risk, the count value exceeds the limit. Greater than or equal to the second-level minimum effective number of frames When the current personnel are deemed to have reached a warning level of risk, the count value exceeds the limit. Greater than or equal to the minimum effective number of frames exceeding the third level At that time, it is determined that the current personnel have reached the intervention-level risk status.

[0195] It should be noted that the minimum effective number of over-limit frames is determined by statistically analyzing the number of consecutive over-limit frames of risk events marked in the historical image samples of hoisting operations, and selecting a threshold of consecutive frames that can distinguish between instantaneous false alarms and real risk events.

[0196] Output tiered early warning information or intervention and control information.

[0197] Furthermore, when the current personnel reach the attention level risk status, attention level early warning information is output.

[0198] The alert level warning information includes at least personnel identification, personnel location, and the currently revised dynamic danger zone range, which is used to remind on-site personnel of the risks of approaching the hoisting operation area.

[0199] Furthermore, when the current personnel reach the alert level risk status, an alert level warning message will be output.

[0200] Among them, the alert-level warning information further triggers on-site audio and visual prompts on the basis of the attention-level warning information, in order to enhance the effect of on-site risk reminders.

[0201] Furthermore, when the current personnel reach the intervention-level risk state, intervention and control information is output.

[0202] Intervention and control information is sent to the on-site monitoring terminal to trigger speed limits, suspension of hoisting, or manual intervention prompts.

[0203] Furthermore, the personnel risk level indicator is set as follows: Then it is represented as: ; in, Indicates time Personnel risk level indicators This indicates that the warning conditions have not been met. This indicates that the risk level has reached the warning level. This indicates that the risk level has reached the alert level. This indicates that the risk level has reached the intervention level.

[0204] Furthermore, when Output attention-level warning information in a timely manner; when Output alert level warning information in a timely manner; when It outputs intervention and control information in real time.

[0205] In this embodiment, Figure 5 This demonstrates the effect of different hazardous zone generation methods on the coverage of the actual motion envelope of the load during industrial hoisting.

[0206] As shown in the figure, the traditional static safety radius remains fixed when the load accelerates, swings more, and changes in direction of travel. It is difficult to adjust synchronously with the movement of the load. In some periods, it is lower than the motion envelope requirement obtained by manual verification, resulting in insufficient hazard coverage.

[0207] This scheme generates a dynamic danger zone that changes with the direction of motion based on the hoisting trajectory, projected speed, hoisting size, and swing offset. Furthermore, when the hoisting approaches a fixed obstacle, the major axis radius is corrected by combining the boundary approach situation, so that the danger zone can expand with the motion envelope of the hoisting.

[0208] The results show that the proposed solution can effectively reduce the mismatch between the static area and the actual risk range, and improve the risk coverage capability and on-site safety monitoring reliability under boundary squeezing scenarios.

[0209] In this embodiment, Figure 6 The comparison between continuous risk assessment of personnel in industrial settings and the traditional static distance threshold method is shown.

[0210] As shown in the figure, the traditional method mainly relies on the instantaneous distance between personnel and the load to make risk assessment. The risk curve fluctuates significantly and is easily affected by short-term obstruction, reflection, or instantaneous movement of personnel, resulting in false alarms or missed alarms.

[0211] When calculating personnel risk values, this scheme integrates factors such as the spatial distance between personnel and the hoisting load, the proximity relationship between personnel, the duration of stay in the danger zone, and the approach of the obstacle boundary. It also performs time-series smoothing and graded threshold determination on the risk values. As shown in the figure, the time-series smoothed risk value changes of this scheme are more continuous, which can suppress instantaneous fluctuations in short-term proximity events and stably reflect the risk increase trend in continuous stay and obstacle approach events.

[0212] The results show that the proposed scheme can achieve continuous quantitative assessment of multidimensional spatiotemporal risks between personnel and moving loads, thereby improving the stability and accuracy of early warning judgments.

[0213] In summary, this invention solves the problem of mismatch between static areas and the actual motion envelope by generating a dynamic danger zone that changes with the direction of the load's movement; by combining the dynamic danger zone with the boundary of fixed obstacles to correct the dynamic danger zone, it automatically enhances the danger zone coverage when the load approaches structures such as fences and columns; by calculating personnel risk values, it achieves continuous quantitative assessment of multi-dimensional spatiotemporal risks between personnel and moving loads; and by performing temporal smoothing and graded over-limit counting judgment on risk values, it suppresses instantaneous false detection fluctuations.

[0214] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A dynamic risk classification and early warning system for hoisting operations based on image recognition, characterized in that, include: The image correction module acquires continuous images of the hoisting operation area, performs distortion correction, perspective correction, and ground benchmark calibration on the continuous images to obtain benchmark monitoring images, and establishes a mapping relationship between image coordinates and ground coordinates; The target tracking module performs target detection, contour segmentation, and cross-frame correlation on the benchmark monitoring image, identifies the boundaries of the hook, load, personnel, and fixed obstacles, obtains the hook trajectory, load trajectory, and personnel trajectory, and determines the hoisting monitoring status based on the positional correspondence between the hook center and the upper edge of the load, continuous frame stability, and load displacement status. The dynamic correction module generates a dynamic danger zone that changes with the direction of the load movement based on the load trajectory, load size and swing information during the hoisting monitoring state, and corrects the dynamic danger zone in combination with the boundary of fixed obstacles. The risk warning module calculates personnel risk values ​​based on personnel trajectory, spatial distance and proximity relationship between personnel and load, and duration of stay in dangerous areas. It then performs continuous time-series judgment on personnel risk values ​​and outputs graded warning information or intervention control information. The generation of dynamic danger zones that change with the direction of hoisting movement includes mapping the hoisting ground projection reference point in the hoisting trajectory to the ground coordinate system to obtain the hoisting ground projection trajectory; Calculate the direction of motion and the projected speed of the hoisted load based on its ground projection trajectory. Based on the lifting profile, the lifting size information is extracted to obtain the equivalent size of the lifting load; Calculate the hoisting swing information based on the hoisting ground projection trajectory and obtain the hoisting swing offset; Calculate the hazardous zone expansion factor based on the equivalent dimensions of the hoisting load, the projected speed of the hoisting load, and the swing offset of the hoisting load. With the center of the hoisted ground projection as the center and the direction of hoisting movement as the major axis, a dynamic hazard zone is generated based on the hazard zone expansion coefficient. The method of combining fixed obstacle boundaries to correct dynamic hazard zones includes extracting a set of fixed obstacle boundary contour points and a set of dynamic hazard zone boundary points; Based on the fixed obstacle boundary contour point set and the dynamic hazard zone boundary point set, calculate the minimum distance between the dynamic hazard zone boundary point set and the fixed obstacle boundary contour point set to obtain the minimum boundary distance; Calculate the boundary approximation factor based on the minimum boundary distance and the boundary reference distance threshold; The major axis radius of the dynamic hazard zone is corrected based on the boundary approximation factor to obtain the corrected major axis radius. Keeping the center of the dynamic hazard zone and the direction of load movement unchanged, the modified dynamic hazard zone is obtained based on the modified major axis radius.

2. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 1, characterized in that, The distortion correction, perspective correction, and ground reference calibration of the continuous images include: Acquire continuous images of the hoisting operation area; Based on the intrinsic parameters and distortion parameters of the image acquisition device, radial distortion correction and tangential distortion correction are performed on continuous images to obtain distortion-corrected images; Ground reference points are set up in the hoisting operation area to obtain the image coordinates of each ground reference point in the distortion-corrected image and the actual coordinates in the ground coordinate system; The perspective transformation matrix is ​​obtained by calculating the image coordinates and actual coordinates of each ground reference point, and then the perspective transformation matrix is ​​used to perform perspective correction on the distortion correction image to obtain the perspective correction image. Using a fixed reference point on the ground in the hoisting operation area as the origin of the ground coordinate system and the main direction of the hoisting as the reference direction of the ground coordinate system, the perspective correction image is calibrated to obtain the reference monitoring image.

3. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 1, characterized in that, The process of establishing the mapping relationship from image coordinates to ground coordinates includes: The midpoint of the bottom edge of the circumscribed rectangle of the personnel target is extracted as the reference point for the personnel ground projection, and the midpoint of the bottom edge of the outline of the hoisted target is extracted as the reference point for the hoisted ground projection. Obtain the image coordinates of the personnel ground projection reference point and the suspended ground projection reference point in the benchmark monitoring image; Based on the perspective transformation matrix, the image coordinates are transformed to correspond to the ground coordinates in a unified ground coordinate system. The perspective transformation matrix is ​​obtained by corresponding the image coordinates of the ground reference point in the reference monitoring image and the actual coordinates of the ground reference point in the ground coordinate system.

4. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 2, characterized in that, The target detection, contour segmentation, and cross-frame correlation of the reference monitoring image include: Target detection is performed on the baseline monitoring image to extract targets such as hooks, loads, personnel, and fixed obstacle boundaries. Contour segmentation is performed on hook targets, load targets, and personnel targets respectively to obtain hook contours, load contours, and personnel contours, and boundary contour point sets are extracted for fixed obstacle boundary targets; Based on the overlap of the bounding rectangles of similar targets between adjacent frames, a candidate matching relationship is established between the target in the current frame and the target in the previous frame. Based on the displacement of the center point of the same type of target between adjacent frames, candidate matching relationships are filtered and target matching pairs that do not meet the displacement constraint conditions are eliminated. For target matching pairs that meet the overlap and displacement constraints, cross-frame association is performed to generate hook trajectory, load trajectory and personnel trajectory respectively.

5. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 4, characterized in that, The specific steps for identifying the boundaries of the hook, load, personnel, and fixed obstacles to obtain the hook trajectory, load trajectory, and personnel trajectory are as follows: Extract the center of the hook outline and obtain the image coordinates of the hook center; Extract the center of the upper edge and the midpoint of the bottom edge of the load from the load profile, and obtain the image coordinates of the center of the upper edge and the image coordinates of the midpoint of the bottom edge of the load, respectively. Extract the midpoint of the bottom edge of the bounding rectangle of the person's outline and obtain the image coordinates of the midpoint of the bottom edge of the person; Extract the boundary contour point set from the fixed obstacle boundary target to obtain the fixed obstacle boundary contour data; The hook center image coordinates in consecutive frames are correlated across frames to obtain the hook trajectory; The image coordinates of the midpoint of the bottom edge of the hoist in consecutive frames are correlated across frames to obtain the hoist trajectory; The image coordinates of the midpoint of the bottom edge of a person in consecutive frames are correlated across frames to obtain the person's trajectory.

6. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 1, characterized in that, The specific steps for determining the hoisting monitoring status based on the positional correspondence between the hook center and the upper edge of the load, continuous frame stability, and load displacement status are as follows: Calculate the positional correspondence distance between the center image coordinates of the hook and the center image coordinates of the upper edge of the load, and determine that the hook and the load satisfy the positional correspondence relationship when the center of the hook is above the center of the upper edge of the load and the positional correspondence distance is less than the distance threshold. Based on the success of the association of the hook trajectory within the continuous analysis frame, the continuous frame stability coefficient of the hook target is calculated, and when the continuous frame stability coefficient is greater than the stability threshold, the hook trajectory is determined to meet the continuous frame stability. Based on the success of the association of the hoisting trajectory within the continuous analysis frame, the continuous frame stability coefficient of the hoisting target is calculated, and when the continuous frame stability coefficient is greater than the stability threshold, the hoisting trajectory is determined to meet the continuous frame stability. Calculate the displacement of the load between adjacent frames. If the displacement is greater than the displacement threshold, the load is determined to be in an effective displacement state. When the hook and the load satisfy the positional correspondence, the hook trajectory satisfies continuous frame stability, the load trajectory satisfies continuous frame stability, and the load is in an effective displacement state, the hoisting device is determined to be in monitoring status.

7. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 1, characterized in that, The calculation of personnel risk values ​​based on personnel trajectory, spatial distance between personnel and hoisting load, proximity relationship, and duration of stay in hazardous areas includes: Map the ground projection reference points of personnel in the personnel trajectory to the ground coordinate system to obtain the ground projection coordinates of personnel. Calculate the spatial distance between the personnel and the hoisting load based on the ground projection coordinates of the personnel and the center coordinates of the hoisting load's ground projection. Extract the personnel movement direction vector from the personnel trajectory, extract the hoisting movement direction vector from the hoisting trajectory, and calculate the personnel approach factor based on the angle between the personnel movement direction vector and the hoisting movement direction vector. Based on the positional relationship between the personnel's ground projection coordinates and the dynamic hazard zone, determine whether the personnel are in a hazard zone, and calculate the continuous residence time of the personnel in the dynamic hazard zone or within the boundary zone of the dynamic hazard zone to obtain the residence time normalization factor. The risk value of personnel is calculated based on spatial distance, proximity factor, and normalization factor of length of stay.

8. The image recognition-based dynamic risk classification and early warning system for hoisting operations as described in claim 7, characterized in that, The specific steps for continuously determining personnel risk values ​​over time and outputting tiered early warning information or intervention control information are as follows: Time smoothing is performed on personnel risk values ​​at consecutive time points to obtain smoothed risk values; The smoothed risk value is compared with the attention-level risk threshold, the alert-level risk threshold, and the intervention-level risk threshold to obtain the over-limit status of the corresponding risk threshold at each level; The system counts the number of times the system exceeds the risk threshold at each level within a statistical time window, and determines the risk level of the personnel based on the comparison between the number of times the system exceeds the threshold and the minimum number of valid frames exceeding the threshold at the corresponding level. When personnel reach the level of attention risk, an attention-level warning message is output. When the risk level of personnel reaches the alert level, an alert level warning message will be output; When personnel reach the intervention-level risk level, intervention and control information is output.

Images