A detection method, device and equipment for tower crane safety and a storage medium

Abstract

The application discloses a detection method, device and equipment for tower crane safety and a storage medium. The method comprises the following steps: acquiring a training sample set comprising a plurality of training sample images; constructing an initial YOLOv5 detection model; inputting the training sample set into the initial YOLOv5 detection model for iterative training; when a preset training termination condition is met, obtaining a multi-target detection model applied to the field of tower crane safety detection after training is completed; wherein the multi-target detection model is used for adding one or more corresponding labels to a detection target in a to-be-detected image, and determining a safety state based on the added labels. The application can accurately label labels for tower crane operation targets and determine safety states, thereby improving the accuracy and efficiency of tower crane safety detection.

Description

Technical Field

[0001] This application relates to the field of safety monitoring technology, and in particular to a detection method, device, equipment and storage medium for tower crane safety. Background Technology

[0002] In the construction industry, tower cranes, as core vertical transportation equipment, directly impact the safety of construction workers and project property. Statistics show that tower crane-related safety accidents account for 30%-40% of all construction safety accidents, with falling loads, personnel accidentally entering dangerous areas, and violations of regulations being the main causes. With the intelligent development of the construction industry, tower crane safety monitoring has evolved from traditional manual supervision to technology-based monitoring, becoming a core direction for industry safety management.

[0003] Currently, target detection technology based on the YOLO series models has been applied in the field of tower crane safety monitoring. However, existing YOLO-based detection methods can only output single-category information for each detected target when facing tower crane operation scenarios, such as identifying "personnel" or "loaded object". This single output information cannot accurately reflect the complex safety conditions in tower crane operation scenarios, such as whether personnel are in the danger zone around the load, or whether the load itself is of a high-risk type. Because it cannot effectively distinguish between operation scenarios with different safety conditions, existing methods are difficult to meet the actual needs of tower crane safety monitoring.

[0004] Therefore, how to enable the target detection model to output richer safety status information in order to accurately reflect the real safety situation of tower crane operation scenarios has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0005] The purpose of this application is to provide a method, device, equipment and storage medium for detecting tower crane safety, which can train a multi-target detection model that can simultaneously identify the basic category and associated extended attributes of the detection target, accurately label the tower crane operation target and determine its safety status, thereby improving the accuracy and efficiency of tower crane safety detection.

[0006] According to one aspect of this application, a method for detecting the safety of tower cranes is provided, the method comprising:

[0007] Obtain a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; Build an initial YOLOv5 detection model, set the corresponding basic output channel for each basic category, and set the corresponding extended output channel for each extended attribute; The training sample set is input into the initial YOLOv5 detection model for iterative training. During the training process, the prediction results of the model on each basic output channel are recorded by the first tensor, and the prediction results of the model on each basic output channel and each extended output channel are recorded by the second tensor. The first error between the prediction result in the first tensor and all basic category labels is calculated using the basic loss function. The second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels is calculated using the all-class loss function. The model parameters are then updated based on the first error and the second error. When the preset training termination condition is met, a multi-target detection model for tower crane safety inspection is obtained after training; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and to determine the safety status based on the added labels.

[0008] Optionally, the basic categories include: personnel, hooks, and suspended objects; the extended attributes include: personnel associated with personnel located around and not located around the suspended object, and long and short hooks, non-long and short hooks, and other suspended objects associated with the suspended object.

[0009] Optionally, before obtaining the training sample set containing multiple training sample images, the method further includes: Acquire raw tower crane operation images; A pre-trained segmentation model is used to perform pixel-level segmentation on the detected targets in the original tower crane operation image to obtain a segmentation mask corresponding to each detected target; Based on the segmentation mask, and according to the preset synthesis rules, the detected targets in the original tower crane operation image are recombined, replaced, and / or added to generate multiple synthesized images, and a label corresponding to each synthesized image is generated. The generated synthetic images are subjected to quality screening, and the screened synthetic images are used as training sample images to form the training sample set.

[0010] Optionally, the preset synthesis rules include at least one of the following: Personnel and suspended object scene compositing rules: The size of the suspended object is calculated using the segmentation mask of the suspended object, and a danger zone is delineated in the original tower crane operation image according to the size of the suspended object; the personnel target is determined using the segmentation mask of the personnel, and the personnel target is placed in the danger zone to generate a high-risk composite image, and the personnel target is placed outside the danger zone to generate a low-risk composite image; The rules for replacing and compositing suspended objects are as follows: The suspended object target is determined using a segmentation mask; the suspended object target is removed from the original tower crane operation image and the background is filled to obtain a background image; a target suspended object sample is selected from a pre-built suspended object sample library, which contains suspended object samples of different categories and / or different postures; the target suspended object sample is resized and its viewing angle is transformed according to the scene information of the original tower crane operation image; and the processed target suspended object sample is then merged into the background image to generate a composite image. Large model-assisted synthesis rules: Input the original tower crane operation image and preset scene description information into the multimodal large model to generate detailed scene description information that conforms to the tower crane operation scenario; input the detailed scene description information into the image generation model to generate a synthesized image.

[0011] Optionally, after obtaining the trained multi-target detection model for tower crane safety inspection when the preset training termination condition is met, the method further includes: Convert the trained multi-object detection model into an ONNX format model; Operator compatibility testing is performed on the ONNX format model to identify operators incompatible with the neural network processor in the target edge computing device; wherein, the target edge computing device is an edge computing device used to deploy the multi-target detection model; Based on the operator compatibility detection results, the model structure of the ONNX format model is adapted to obtain an adapted model structure including a first sub-model and a second sub-model. The adaptation process includes: simplifying the detection head of the ONNX format model to remove post-processing operations, thereby causing the simplified detection head to output an undecoded feature map; and dividing the ONNX format model into a first sub-model and a second sub-model. The first sub-model performs feature extraction and preliminary transformation, outputting an undecoded feature map. The second sub-model is a post-processing module used to post-process the undecoded feature map output by the first sub-model to obtain the final multi-target detection result. The first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor, and the second sub-model is deployed on the central processing unit in the form of program code.

[0012] Optionally, the step of converting the first sub-model into a deployment model format adapted to the neural network processor and deploying it on the neural network processor, and deploying the second sub-model in the form of program code on the central processing unit, includes: Obtain the raw accuracy of the first sub-model on the validation set; wherein, the raw accuracy is the accuracy of the first sub-model without any quantization processing; The process iterates through each network layer in the first sub-model, performing a quantization precision decision on the currently traversed target network layer to determine the quantization precision of each network layer. The quantization precision decision includes: firstly, quantizing the target network layer according to a first quantization precision and calculating a first precision loss before and after the quantization adjustment; wherein the first quantization precision is a quantization precision lower than the original precision; if the first precision loss exceeds a preset loss threshold, then quantizing the target network layer according to a second quantization precision and calculating a second precision loss before and after the quantization adjustment; wherein the second quantization precision is a quantization precision between the original precision and the first quantization precision; if the second precision loss still exceeds the preset loss threshold, then maintaining the original precision of the target network layer. Based on the quantization precision determined by the quantization precision decision for each network layer, a hybrid quantization strategy is formed; wherein, the hybrid quantization strategy is a combination scheme that uses different quantization precisions for different network layers. Based on the hybrid quantization strategy, the first sub-model is quantized and calibrated using a calibration image set to generate a quantized first sub-model; wherein, the quantization calibration is the process of determining the quantization parameters of each network layer through the calibration image set. The quantized first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor.

[0013] Optionally, after obtaining the trained multi-target detection model for tower crane safety inspection when the preset training termination condition is met, the method further includes: The real-time acquired tower crane operation monitoring video or image is used as the image to be detected and input into the trained multi-target detection model; The first sub-model in the multi-object detection model is used to detect objects in the image to be detected, and features are extracted based on the detection results to output an undecoded feature map. The undecoded feature map is post-processed by the second sub-model in the multi-object detection model to obtain the prediction result of each detected target in the image to be detected in each output channel. Based on the prediction results of each output channel, a corresponding label is added to each detected target, and the safety status is determined based on the added label.

[0014] To achieve the above objectives, this application also provides a detection device for tower crane safety, the device comprising: The acquisition module is used to acquire a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; The building module is used to build the initial YOLOv5 detection model, setting the corresponding basic output channel for each basic category and the corresponding extended output channel for each extended attribute; The training module is used to input the training sample set into the initial YOLOv5 detection model for iterative training. During the training process, the model's prediction results on each basic output channel are recorded through the first tensor, and the model's prediction results on each basic output channel and each extended output channel are recorded through the second tensor. The parameter tuning module is used to calculate the first error between the prediction result in the first tensor and all basic category labels using the basic loss function, calculate the second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels using the all-class loss function, and update the model parameters based on the first error and the second error. The processing module is used to obtain a trained multi-target detection model for tower crane safety detection when a preset training termination condition is met; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and determine the safety status based on the added labels.

[0015] To achieve the above objectives, this application also provides a computer device, which specifically includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the above-described detection method for tower crane safety.

[0016] To achieve the above objectives, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described detection method for tower crane safety.

[0017] The method, apparatus, equipment, and storage medium for tower crane safety detection provided in this application, by labeling training samples with basic category tags and associated extended attribute tags, enable the model to learn in a way that better meets the multi-dimensional recognition needs of tower crane safety detection. The model is provided with separate basic and extended output channels, allowing it to simultaneously output target basic category and associated attribute information, overcoming the limitations of single-category detection. During training, two tensors are used to record prediction results in different dimensions, and two loss functions are used to accurately calculate the corresponding errors and update parameters, making model training more targeted and ensuring the accuracy of the model in recognizing the basic category and associated attributes of tower crane operation targets. The finally trained model can quickly label targets in the image to be detected and directly determine their safety status without multiple detection inferences, significantly improving the efficiency and accuracy of tower crane safety detection and effectively adapting to the safety monitoring needs of tower crane operations. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of an optional process for a tower crane safety testing method provided in Embodiment 1; Figure 2 This is a schematic diagram of another optional process for the tower crane safety detection method provided in Example 1; Figure 3 This is a schematic diagram of an optional component structure of the tower crane safety detection device provided in Embodiment 2; Figure 4 This is a schematic diagram of an optional hardware structure for the computer device provided in Embodiment 3. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0020] Example 1 This application provides a method for detecting tower crane safety, such as... Figure 1 As shown, the method specifically includes the following steps: Step S101: Obtain a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have a relationship.

[0021] Each training sample image in the training sample set is collected from the actual operation scene of the tower crane, and one or more detection targets appearing in the training sample image have been pre-labeled. That is, the position range of each detection target in the training sample image is accurately marked by box selection, and one or more labels are marked for each box-selected detection target.

[0022] Specifically, the basic category label is used to characterize the core category attributes of the detection target, which can cover typical targets in tower crane operation scenarios such as personnel, hooks, suspended objects, construction vehicles, and building components. The extended attribute label is a further detailed description of the corresponding basic category, which is used to characterize additional information related to tower crane safety of the detection target. It needs to form a relationship with the basic category label. For example, for suspended objects, it can be marked whether they are long or short; for personnel, it can be marked whether they are in the danger zone of the suspended object; for hooks, it can be marked whether they are in the lifting operation state.

[0023] It's also important to note that there's a correlation between extended attribute labels and base category labels; some extended attributes are only meaningful for specific base categories. Therefore, during annotation, if a detected target is assigned an extended attribute label, it will necessarily be assigned its corresponding base category label simultaneously. If a detected target only has a base category label and no extended attribute labels, then only the base category label needs to be labeled. Through this annotation method, each detected target in each training sample image carries rich and hierarchical label information, providing accurate supervision signals for the subsequent model to learn the target's category and its safety status.

[0024] Step S102: Construct an initial YOLOv5 detection model, set the corresponding basic output channel for each basic category, and set the corresponding extended output channel for each extended attribute.

[0025] YOLOv5 (You Only Look Once version 5) is a high-efficiency real-time object detection deep learning model, characterized by fast detection speed, high accuracy, and flexible deployment, and is widely used in industrial object recognition. The output channel is the "judgment dimension" in the model's output layer used to predict a specific category or attribute. Each output channel corresponds to an independent prediction task, outputting a probability value between 0 and 1, representing the confidence that the detected target belongs to the category or attribute corresponding to that channel. In this embodiment, corresponding basic output channels are set for each basic category, and each basic output channel is used to predict whether the detected target belongs to the corresponding basic category. Simultaneously, corresponding extended output channels are set for each extended attribute, and each extended output channel is used to predict whether the detected target possesses the corresponding extended attribute. Through this division of output channels, the model can simultaneously output multi-dimensional prediction results of the detected target in both the basic category dimension and the extended attribute dimension during a single forward inference process.

[0026] Preferably, an independent Sigmoid activation function is configured for each output channel of the initial YOLOv5 detection model, so that each output channel independently outputs the corresponding prediction result.

[0027] It should also be noted that although this embodiment is described based on YOLOv5, the core improvement ideas are also applicable to subsequent YOLO series models such as YOLOv8 and YOLOv9, as well as other target detection models with similar architectures, and these should all be considered within the scope of protection of this application.

[0028] Step S103: Input the training sample set into the initial YOLOv5 detection model for iterative training. During the training process, the prediction results of the model on each basic output channel are recorded by the first tensor, and the prediction results of the model on each basic output channel and each extended output channel are recorded by the second tensor.

[0029] The first tensor and the second tensor are both represented as matrix tables, which can clearly and orderly store the predicted values ​​of the model on each output channel, facilitating subsequent error calculation and model parameter updates.

[0030] Step S104: Calculate the first error between the prediction result in the first tensor and all basic category labels using the basic loss function, calculate the second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels using the all-class loss function, and update the model parameters based on the first error and the second error.

[0031] In this embodiment, the design employing two tensors and two loss functions for separate calculations offers the following technical advantages: First, dedicated supervision of the first tensor and the basic loss function ensures the model's accuracy in the basic classification task, preventing interference with basic category recognition due to the introduction of extended attributes. Second, comprehensive supervision of the second tensor and the all-class loss function guides the model to learn the intrinsic relationship between basic categories and extended attributes, enabling the model to truly understand composite concepts rather than simply mechanically outputting multiple independent labels. Finally, the combined effect of the two loss functions allows the trained model to accurately identify the target's basic category and precisely determine its safety status attributes, ultimately achieving a multi-dimensional detection result containing both basic category labels and extended attribute labels in a single detection.

[0032] Preferably, the base loss function and the all-class loss function are both BCEWithLogitsLoss functions. This loss function incorporates a sigmoid activation function, which maps the model's original output values ​​to independent probability values ​​between 0 and 1, making it suitable for multi-label binary classification tasks. Compared to the Softmax loss function commonly used in traditional classification tasks (where the sum of probabilities for each class is 1, mutually restricting each other), BCEWithLogitsLoss allows each output channel to calculate its loss independently, without interference between channels. This means the model can activate multiple output channels for the same target simultaneously without suppressing another channel due to the activation of one. This characteristic enables the model to naturally output multiple labels, providing a foundation for subsequent multi-label parsing.

[0033] Step S105: When the preset training termination condition is met, a multi-target detection model for tower crane safety detection is obtained after training; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and to determine the safety status based on the added labels.

[0034] In this embodiment, by labeling training samples with basic category tags and associated extended attribute tags, the model's learning is made more aligned with the multi-dimensional recognition needs of tower crane safety detection. Separate basic and extended output channels are provided for the model, enabling it to simultaneously output target basic category and associated attribute information, overcoming the limitations of single-category detection. During training, two tensors are used to record prediction results for different dimensions, and two loss functions are used to accurately calculate the corresponding errors and update parameters, making model training more targeted and ensuring the model's accuracy in recognizing the basic category and associated attributes of tower crane operation targets. The finally trained model can quickly label targets in the image to be detected and directly determine their safety status without multiple detection inferences, significantly improving the efficiency and accuracy of tower crane safety detection and effectively adapting to the safety monitoring needs of tower crane operations.

[0035] In addition, as an optimal specific implementation, the basic categories include: personnel, hooks, and suspended objects; the extended attributes include: personnel associated with personnel located around and not located around the suspended object, as well as long and short hooks, non-long and short hooks, and other suspended objects associated with the suspended object.

[0036] In this embodiment, a two-dimensional output mechanism of "basic categories + extended attributes" is constructed in the `build_targets` function of YOLOv5. Basic categories 0 to 2 directly correspond to basic output channels 0 to 2, and extended attributes 3 to 7 directly correspond to extended output channels 3 to 7. Combined categories are jointly activated through "basic category channel + extended attribute channel", and the mapping rule is as follows: Basic categories (3): Basic category 0 (personnel), Basic category 1 (hook), Basic category 2 (lifted object); Hanging object combination categories (3): Activate basic category 2 (hanging object) + extended attributes 3 to 5 (long and short hanging objects / non-long and short hanging objects / other hanging objects); Personnel combination categories (2): Activate basic category 0 (personnel) + extended attributes 6 to 7 (personnel located around the suspended object / personnel not located around the suspended object).

[0037] This embodiment also uses two tensors to record the prediction results. The first tensor, t_orig_cl, has a shape of [n, 3] and is used to record the prediction result of each detected target on the three basic output channels, where n is the total number of detected targets in the current training sample image. The second tensor, t_new_cl, has a shape of [n, 8] and is used to record the prediction result of each detected target on all eight output channels. During the initialization phase, all prediction results of these two tensors are set to negative sample label values ​​(e.g., 0.05), indicating that by default, all detected targets do not belong to any category or attribute.

[0038] To improve the model's generalization ability and training stability, this embodiment also introduces label smoothing technology. Traditional label settings typically use "hard labels," where positive samples are labeled 1 and negative samples are labeled 0. Such extreme label values ​​can easily lead to the model becoming overconfident and overfitting. This embodiment adopts a "soft label" strategy, setting the positive sample label to 0.95 and the negative sample label to 0.05. By reducing the extreme values ​​of the labels, the model maintains a certain margin of error during the learning process, thereby reducing the risk of overfitting and improving generalization ability in unknown scenarios.

[0039] Specifically, prior to step S101, the method further includes: Step A1: Obtain the original tower crane operation image; Step A2: Use a pre-trained segmentation model to perform pixel-level segmentation on the detected targets in the original tower crane operation image to obtain a segmentation mask corresponding to each detected target; Step A3: Based on the segmentation mask, according to the preset synthesis rules, the detected targets in the original tower crane operation image are recombined, replaced and / or added to generate multiple synthesized images, and a label corresponding to each synthesized image is generated; Step A4: Perform quality screening on the generated synthetic images, and use the screened synthetic images as the training sample images to form the training sample set.

[0040] In the field of tower crane safety monitoring, training high-performance target detection models typically requires a large number of precisely labeled training sample images. However, in practical applications, obtaining a sufficient number of high-quality training samples faces numerous difficulties. First, tower crane operation scenarios are complex and varied, including factors such as changes in lighting, weather conditions, and differences in shooting angles, requiring a large number of samples covering different scenarios to ensure the model's generalization ability. Second, certain critical safety scenarios, such as "personnel being in a dangerous area around the suspended load," are strictly prohibited in actual construction, making it almost impossible to collect realistic sample images. To address the aforementioned difficulties in obtaining training samples, this embodiment provides a training sample enhancement method based on image synthesis technology. This method processes existing original tower crane operation images to automatically generate a large number of diverse synthetic images, and simultaneously generates precise labels corresponding to these synthetic images, thereby expanding the training sample set at low cost and high efficiency.

[0041] Specifically, a batch of original tower crane operation images is first acquired as the base material. Then, SAM2 (Segment Anything Model 2) is used as a pre-trained segmentation model to perform pixel-level accurate segmentation of the detected targets in the original tower crane operation images. Preferably, the sam2_hiera_large pre-trained model of SAM2 is loaded, the image predictor is initialized, and the input image resolution is set to 1024×1024 pixels, with the inference confidence threshold set to 0.75 to ensure the accuracy of the segmentation results. SAM2 supports three prompting methods: point prompts (user clicks on the core area of ​​the target), bounding box prompts (manually drawn target bounding boxes), and text prompts (inputting keywords such as "personnel" or "lifted object"), which can be flexibly selected to adapt to different scenario requirements, achieving a segmentation accuracy of over 95.2%. After obtaining the segmentation mask, to further enhance the realism of the synthesized image, edge optimization processing is performed on the mask: first, a Gaussian blur with a 3×3 kernel size and a standard deviation of 1.2 is used to smooth the edge contours; then, a bilateral filter with spatial sigma=50 and grayscale sigma=50 is used to feather hard boundaries while maintaining edge sharpness, eliminating harsh cutting marks and making the subsequent pasting of targets into the new background more natural. Through the above steps, high-quality independent target materials such as people and hanging objects, along with their corresponding segmentation masks, can be obtained, providing a reliable foundation for subsequent scene compositing.

[0042] In addition, the generated synthetic images undergo quality screening using a dual-threshold screening mechanism that combines sharpness scoring with scene reasonableness scoring based on semantic judgment from a large model. Both scores are required to be no less than 0.75 to filter out low-quality data. This approach significantly improves the diversity of synthetic scenes, increasing it by over 30% and effectively expanding the coverage of scarce scenes in the training sample set.

[0043] Furthermore, the preset synthesis rules include at least one of the following: (1) Human and hoisted object scene synthesis rules: The size of the hoisted object is calculated using the segmentation mask of the hoisted object, and a dangerous area is delineated in the original tower crane operation image according to the size of the hoisted object; the human target is determined using the segmentation mask of the human, and the human target is placed in the dangerous area to generate a high-risk synthesized image, and the human target is placed outside the dangerous area to generate a low-risk synthesized image; wherein, the human target in the high-risk synthesized image has an extended attribute label representing the dangerous state of the human, and the human target in the low-risk synthesized image has an extended attribute label representing the safe state of the human.

[0044] In the scene compositing rules for personnel and suspended objects, the dimensions of the suspended object are first calculated using a segmentation mask. Specifically, the diagonal length L of the bounding rectangle of the suspended object is used as a reference. A danger zone is delineated in the original tower crane operation image. This danger zone is a circular area with a radius of 1.5L centered on the center of the suspended object, corresponding to an actual distance of approximately 1-3 meters. Subsequently, the individual personnel targets are identified using a segmentation mask. During high-risk scene compositing, 1-3 personnel targets are randomly selected within the danger zone, and the posture and orientation of the personnel targets are adjusted using a pose estimation model to adapt to the shooting angle of the scene. At the same time, a lighting matching algorithm is applied to extract the RGB mean and variance of the background image to adjust the lighting parameters of the personnel targets. Then, the shadow offset is calculated based on the lighting direction of the background, and a semi-transparent shadow is generated to make the personnel targets blend naturally with the background. Finally, the bounding box of the personnel targets and the extended attribute label (i.e., category 6) for "personnel located around the suspended object" are automatically generated. When compositing low-risk scenes, personnel targets are placed outside the danger zone, specifically in a safe area greater than 2.5L (corresponding to an actual distance greater than 5 meters) from the center of the suspended object. The remaining processing logic is the same as for high-risk scenes, including posture adjustment, lighting matching, and shadow generation. Bounding boxes for personnel targets and extended attribute labels for "personnel not located around the suspended object" (i.e., category 7) are automatically generated. This method allows for the batch generation of composite images of high-risk and low-risk scenes with precise labels, effectively addressing the problem of scarce hazardous state samples in real-world scenarios.

[0045] (2) Sling replacement and synthesis rules: The sling target is determined by the segmentation mask of the sling, the sling target is removed from the original tower crane operation image and the background is filled to obtain a background image; the target sling sample is selected from the pre-built sling sample library, which contains sling samples of different categories and / or different postures; the target sling sample is adjusted in size and transformed in viewpoint according to the scene information of the original tower crane operation image, and the processed target sling sample is merged into the background image to generate a composite image.

[0046] In the object replacement synthesis rule, the original object to be replaced is first determined from the original tower crane operation image using the object's segmentation mask, and then removed using image inpainting technology. Specifically, the SAM2 segmentation model is used to accurately obtain the segmented region of the original object, and then the LaMa large mask inpainting model is called to fill the background of this region. The iteration count is set to 100 times and the inpainting radius to 5 pixels to ensure that there are no residual traces in the background area after removing the object, resulting in a clean background image. Simultaneously, a pre-built object sample library is constructed, divided according to size ranges from 0.5 meters to 5 meters, covering five categories of objects including long and short cranes, and non-long and short cranes, totaling more than 100 samples. All samples have undergone SAM2 segmentation preprocessing, meaning that each object sample has been accurately segmented from the original image and has a corresponding segmentation mask. When replacing suspended objects, a target suspended object sample is randomly selected from the suspended object sample library. Based on the position and size of the original suspended object in the original image, the target suspended object sample is proportionally adjusted to match its size. Simultaneously, a perspective transformation matrix is ​​calculated based on the shooting angle of the original image, and the perspective of the target suspended object sample is transformed to match the shooting angle of the background image. After size adjustment and perspective transformation, the processed target suspended object sample is fused into the previously obtained background image. To ensure a natural and realistic fusion effect, further optimization processing is performed: the RGB mean and variance of the background image are extracted using a lighting matching algorithm, and the lighting parameters of the target suspended object sample are adjusted to match the background lighting; simultaneously, the shadow offset is calculated based on the lighting direction of the background image to generate a semi-transparent projected shadow for the target suspended object sample. Through these steps, a single original tower crane operation image can generate 5 to 10 variant synthetic images containing different suspended object categories and postures, increasing the data volume by 5 to 10 times and effectively expanding the diversity and coverage of suspended object categories in the training sample set.

[0047] (3) Large model-assisted synthesis rules: Input the original tower crane operation image and preset scene description information into the multimodal large model to generate detailed scene description information that conforms to the tower crane operation scene; input the detailed scene description information into the image generation model to generate a synthesized image.

[0048] In the large model-assisted synthesis rule, the original tower crane operation image and / or simple preset scene description information (such as "rainy day scene, construction workers around the long and short cranes") are first input into the multimodal large model GPT-4V, which generates detailed scene description information that matches the tower crane operation scene. For example, the generated detailed scene description information includes: "Construction site, light rain, dim lighting, tower crane hook suspending long and short cranes (approximately 3 meters in length), two construction workers wearing safety helmets within 1.5 meters of the suspended load, the image is taken from a top-down angle, and the background includes scaffolding and construction equipment." Subsequently, this detailed scene description information is input into the image generation model Stable Diffusion v1.5, and the generated image resolution is set to 1024×768 pixels, the number of iterations is 50, and the CFG Scale is 7.5, thereby generating diverse synthetic images that match the detailed scene description information. Simultaneously with image generation, the multimodal large model can automatically output suggested annotations for the category and location of targets in the image, assisting in rapid manual verification.

[0049] It should also be noted that, in addition to the three deep learning-based intelligent synthesis rules mentioned above, this embodiment can also combine mature traditional data augmentation strategies from the YOLOv5 framework to further enhance the diversity of training samples. Specifically, these include the following categories: image stitching and blending augmentation, such as Mosaic augmentation which stitches four images together into a new image at random ratios, enabling the model to learn multiple targets and their contextual relationships simultaneously; and MixUp augmentation which blends two images pixel by pixel with weights of 0.3-0.7 to generate a synthetic image with transitional features, improving the model's robustness. Regarding target manipulation, Copy-Paste augmentation is used to randomly copy 1-3 detected targets and paste them to other locations in the image, maintaining the target ratio unchanged, thereby increasing the target density and distribution diversity in a single image. In terms of geometric transformations, images are subjected to random rotations (-30° to 30°), random scaling (0.5x to 1.5x), horizontal flipping (probability 0.5), and random cropping (cropping areas occupying 60% to 100% of the original image) to simulate target performance under different shooting angles, distances, and compositions. These traditional data augmentation strategies can be combined with the aforementioned three intelligent synthesis rules to further enrich the scene coverage and data diversity of the training sample set, effectively improving the model's generalization ability and robustness.

[0050] Specifically, after step S105, the method further includes: Step B1: Convert the trained multi-object detection model into an ONNX format model.

[0051] ONNX (Open Neural Network Exchange) is an open neural network exchange format used for model conversion and compatibility between different deep learning frameworks.

[0052] In this embodiment, after model training is complete, the trained PyTorch multi-object detection model first needs to be converted to ONNX format for subsequent deployment on edge computing devices. During the conversion process, the following key parameters are set: `opset_version=12` to ensure operator version compatibility; `do_constant_folding=True` is enabled for constant folding optimization to improve model inference efficiency; and `remove_unused_initializers=True` is enabled to remove redundant parameters and reduce model size. Before conversion, the model's detection head needs to be set to a mode compatible with Rockchip NPUs to ensure the complete preservation of the backbone, neck, and simplified detection head modules, laying the foundation for subsequent operator compatibility testing and model adaptation.

[0053] Step B2: Perform operator compatibility testing on the ONNX format model to identify operators incompatible with the neural network processor in the target edge computing device; wherein, the target edge computing device is an edge computing device used to deploy the multi-target detection model.

[0054] In this embodiment, after obtaining the ONNX format model, operator compatibility testing is performed to identify operators incompatible with the neural network processor in the target edge computing device. The target edge computing device is an edge computing device used to deploy multi-object detection models; in this embodiment, the RK3568 chip is preferably used. During the testing process, the model structure is first verified using ONNXRuntime's onnx.checker to ensure the model file itself is error-free. Then, the list of operators supported by the RK3568 chip is obtained using RKNNToolkit2's rknn.list_support_ops() (based on Rockchip V1.3 technical documentation, covering more than 200 core operators). Next, the computation graph nodes of the ONNX format model are traversed, and the operators used by each node are compared with the list of operators supported by RKNN to identify potentially incompatible operators, such as dynamic reshape and complex slicing operations. The operator type, input / output dimensions, and topological relationships of abnormal nodes are recorded to provide a basis for subsequent model adaptation.

[0055] Step B3: Based on the detection results of the operator compatibility test, the model structure of the ONNX format model is adapted to obtain an adapted model structure including a first sub-model and a second sub-model. The adaptation process includes: simplifying the detection head of the ONNX format model to remove post-processing operations, thereby causing the simplified detection head to output an undecoded feature map; and dividing the ONNX format model into a first sub-model and a second sub-model. The first sub-model is used to perform feature extraction and preliminary transformation, outputting an undecoded feature map. The second sub-model is a post-processing module used to post-process the undecoded feature map output by the first sub-model to obtain the final multi-target detection result.

[0056] In this embodiment, based on the detection results of operator compatibility testing, the model structure of the ONNX format model is adapted to obtain an adapted model structure including a first sub-model and a second sub-model. The adaptation process includes two aspects: First, the detection head of the ONNX format model is simplified by modifying the detection head structure based on the RKNN pattern, removing complex post-processing operations such as mesh generation and coordinate decoding, and retaining only convolutional layers and the Sigmoid activation function. This simplified detection head outputs undecoded feature maps. In this embodiment, it outputs feature maps at three scales, with dimensions of [1,255,80,80], [1,255,40,40], and [1,255,20,20]. Second, the entire ONNX format model is divided into a first sub-model and a second sub-model. The first sub-model contains a backbone network, a neck network, and a simplified detection head, used to perform feature extraction and preliminary transformation, outputting undecoded feature maps. The second sub-model is a post-processing module, implemented in program code, used to post-process the undecoded feature maps output by the first sub-model, including feature map decoding, non-maximum suppression (NMS), and multi-label parsing, ultimately obtaining complete detection results.

[0057] Step B4: Convert the first sub-model into a deployment model format adapted to the neural network processor and deploy it on the neural network processor, and deploy the second sub-model in the form of program code on the central processing unit.

[0058] In this embodiment, after model adaptation, the first sub-model is converted into a deployment model format adapted to the neural network processor (RKNN format in this embodiment) and deployed on the neural network processor (NPU) of the target edge computing device. Simultaneously, the second sub-model is deployed as program code on the central processing unit (CPU) of the target edge computing device. During actual inference, the NPU and CPU work together through a heterogeneous collaborative mechanism: the first sub-model performs forward inference on the input image on the NPU, outputting undecoded feature maps; these feature maps are transferred to the CPU post-processing module via memory mapping (DMA zero-copy technology), with a transmission latency controlled within 2ms; the CPU dynamically reconstructs the mesh based on the feature map size, performs coordinate inverse calculation and confidence level selection, and finally outputs the basic category label, extended attribute label, location information, and confidence level for each detected target. This dual-model collaborative architecture fully leverages the computational power advantage of the NPU in feature extraction while utilizing the flexibility of the CPU to complete complex post-processing logic, ensuring the integrity and accuracy of the detection function.

[0059] In this embodiment, a complete edge deployment optimization scheme is provided to address the technical challenges faced by existing deep learning models when deployed on edge computing devices, such as large model size, slow inference speed, severe loss of quantization accuracy, and poor operator compatibility. Through steps B1 to B4, this embodiment effectively solves the problem of model incompatibility with edge device operators, achieving a model size compression of over 75% (from 45MB to 11MB), an inference speed increase of over 10 times (from 150ms to 15ms), and a real-time detection speed of over 30 frames per second. Simultaneously, the loss of quantization accuracy is controlled within 1.2%, successfully overcoming the triangular contradiction between size, speed, and accuracy in edge deployment. This represents a leap from "difficult to deploy" to "efficient adaptation," meeting the real-time and reliability requirements of tower crane safety monitoring scenarios for edge devices.

[0060] Furthermore, step B4 specifically includes: Step B41: Obtain the raw accuracy of the first sub-model on the validation set; wherein the raw accuracy is the accuracy of the first sub-model without any quantization processing.

[0061] In this embodiment, before quantization, the raw accuracy of the first sub-model on the validation set is first obtained as the quantization benchmark. In this embodiment, a validation set containing 1000 images is prepared, covering six different tower crane operation scenarios, with high-risk scenarios accounting for 40%. The first sub-model without any quantization (i.e., the model using FP32 accuracy) is tested on this validation set, and its average accuracy (mAP) is 91.5%. This value is used as the benchmark accuracy in the subsequent quantization process to measure the impact of quantization on model performance.

[0062] Step B42: Iterate through each network layer in the first sub-model sequentially, and perform a quantization precision decision on the currently traversed target network layer to determine the quantization precision of each network layer; wherein, the quantization precision decision includes: firstly, quantizing the target network layer according to a first quantization precision, and calculating the first precision loss before and after the quantization adjustment; wherein, the first quantization precision is a quantization precision lower than the original precision; if the first precision loss exceeds a preset loss threshold, then quantizing the target network layer according to a second quantization precision, and calculating the second precision loss before and after the quantization adjustment; wherein, the second quantization precision is a quantization precision between the original precision and the first quantization precision; if the second precision loss still exceeds the preset loss threshold, then the target network layer maintains its original precision.

[0063] In this embodiment, after obtaining the baseline accuracy, each network layer (including convolutional layers, batch normalization layers, activation layers, etc.) in the first sub-model is traversed sequentially, and a quantization accuracy decision is made for the currently traversed target network layer. The specific decision process is as follows: First, the target network layer is quantized and adjusted according to the first quantization accuracy (INT8 accuracy in this embodiment), and the accuracy of the entire first sub-model on the validation set after quantization is tested. The first accuracy loss before and after quantization adjustment is calculated. If the first accuracy loss exceeds a preset loss threshold (set to 2% in this embodiment, this threshold is determined based on the minimum acceptable accuracy of 90% for edge deployment), the target network layer is reverted to the second quantization accuracy (INT16 accuracy in this embodiment) for quantization adjustment, and the second accuracy loss is calculated. If the second accuracy loss still exceeds the preset loss threshold, the target network layer retains its original accuracy (FP32) and no quantization adjustment is performed. Through the above layer-by-layer quantization and verification process, the most suitable quantization accuracy for each network layer is determined.

[0064] Step B43: Based on the quantization precision determined by the quantization precision decision for each network layer, a hybrid quantization strategy is formed; wherein, the hybrid quantization strategy is a combination scheme that uses different quantization precisions for different network layers.

[0065] In this embodiment, after determining the quantization precision for all network layers, a hybrid quantization strategy is formed based on the final determined quantization precision for each network layer. The hybrid quantization strategy refers to a combination of different quantization precisions applied to different network layers. In this embodiment, the decrease in mAP before and after quantization for each layer, recorded during the layer-by-layer quantization comparison, is used as a sensitivity score. High-sensitivity layers (such as classification heads and detection heads, with a sensitivity score greater than 1%) preferentially use INT16 or FP32 precision, while low-sensitivity layers (with a sensitivity score less than 0.5%) use INT8 precision. Specific configurations include: the first 10 convolutional layers of the backbone network use INT16 quantization, sensitive layers such as classification heads and detection heads use INT8 quantization, and batch normalization layers and activation layers maintain FP32 precision; simultaneously, channel-by-channel quantization is applied to the convolutional layers, and the optimal quantization parameters are solved by minimizing the KL divergence iteratively 50 times to reduce quantization error. This hybrid quantization strategy can maximize model accuracy while compressing the model size.

[0066] Step B44: Based on the hybrid quantization strategy, the first sub-model is quantized and calibrated using a calibration image set to generate a quantized first sub-model; wherein, the quantization calibration is the process of determining the quantization parameters of each network layer through the calibration image set.

[0067] In this embodiment, based on the hybrid quantization strategy formed in step B43, the first sub-model is quantized using a calibration image set. Quantization calibration is the process of determining the quantization parameters of each network layer through the calibration image set; that is, calculating the optimal scaling factor and zero-point quantization parameters based on the numerical distribution range of the actual image data in each layer. In this embodiment, 100 representative tower crane operation images are used as the calibration image set. Quantization calibration is performed on the truncated ONNX format model using RKNN Toolkit2, iterating 20 times to finally generate the quantized first sub-model. Quantization calibration makes the quantization parameters more closely match the distribution characteristics of the actual data, further reducing the accuracy loss caused by quantization.

[0068] Step B45: Convert the quantized first sub-model into a deployment model format adapted to the neural network processor, and deploy it on the neural network processor.

[0069] In this embodiment, after quantization calibration, the quantized first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor of the target edge computing device. In this embodiment, the quantized first sub-model is converted to RKNN format, a model format specific to Rockchip NPU, which can fully leverage the hardware acceleration capabilities of the NPU. After conversion, the RKNN model is loaded into the initialized NPU memory, with memory usage controlled within 1GB. Simultaneously, the second sub-model (post-processing module) is deployed to the central processing unit in program code form and bound to two A76 cores to optimize processing efficiency. After deployment, real-time inference can be performed in actual tower crane monitoring scenarios: the input image resolution is fixed at 896×896; the NPU performs forward inference and outputs feature maps, with a single-frame inference time not exceeding 12ms; after receiving the feature maps, the CPU performs non-maximum suppression (IoU threshold 0.45) and multi-label parsing, with post-processing time not exceeding 3ms. Furthermore, by reducing computation and scheduling overhead through operator fusion (such as Conv+BN+ReLU6 and ConvTranspose+BN+ReLU), and by using the TorchPrune tool to prune redundant layers and channels by no more than 20%, the computational load is further reduced. Ultimately, an inference speed of no less than 30 FPS is achieved on the RK3568 platform, meeting the requirements of real-time monitoring scenarios.

[0070] Specifically, after step S105, the method further includes: Step C1: Use the real-time acquired tower crane operation monitoring video or image as the image to be detected and input it into the trained multi-object detection model; Step C2: Target detection is performed on the image to be detected using the first sub-model in the multi-target detection model, and feature extraction is performed based on the detection results to output an undecoded feature map; wherein, the first sub-model is the model part deployed on the neural network processor of the edge computing device; Step C3: Post-process the undecoded feature map using the second sub-model in the multi-target detection model to obtain the prediction result of each detected target in the image to be detected under each output channel; wherein, the second sub-model is the model part deployed on the central processing unit of the edge computing device; Step C4: Based on the prediction results of each output channel, add a corresponding label to each detected target, and determine the safety status based on the added labels.

[0071] After the model is trained and deployed, real-time safety monitoring can be performed in actual tower crane operation scenarios. In practice, real-time acquired tower crane operation monitoring videos or images are used as the images to be detected and input into the trained multi-object detection model. First, the first sub-model, deployed on the neural network processor of the edge computing device, performs object detection and feature extraction on the image to be detected, outputting an undecoded feature map. Subsequently, the second sub-model, deployed on the central processing unit, performs post-processing on this feature map, including feature map decoding, non-maximum suppression, and multi-label parsing, obtaining prediction results for each detected target in eight output channels. Based on these prediction results, corresponding labels are added to each detected target: for personnel targets, if the probability of the 6th channel exceeds the threshold, an extended attribute label of "personnel located around the suspended object" is added, indicating that the personnel are in a danger zone; if the probability of the 7th channel exceeds the threshold, an extended attribute label of "personnel not located around the suspended object" is added, indicating that the personnel are in a safe zone. For suspended objects, if the probability of the third channel exceeds the threshold, an extended attribute label of "long / short suspended" is added; if the probability of the fourth channel exceeds the threshold, a label of "not long / short suspended" is added; and if the probability of the fifth channel exceeds the threshold, a label of "other suspended objects" is added. Based on these labels, the safety status can be determined: when a personnel target is detected with the label "personnel located around the suspended object," it is judged as a high-risk violation; when a suspended object target is detected with the label "long / short suspended," it is also judged as a high-risk violation. Finally, the basic category label, extended attribute label, location information, and confidence level of each detected target are output, providing a decision-making basis for subsequent audible and visual alarms, management personnel notifications, and tower crane control.

[0072] Furthermore, in actual use, after determining the safety status based on the labels output by the model, the system further performs tiered early warning and data recording operations. Specifically, when a personnel target is detected with the extended attribute label "personnel located around the suspended object" (i.e., category 6) or a suspended object target is detected with the extended attribute label "long and short suspension" (i.e., category 3), it is determined to be a high-risk violation event. The system immediately triggers a high-risk early warning mechanism: activating the on-site audible and visual alarm, simultaneously pushing the early warning information (including violation type, occurrence time, location image, etc.) to the management terminal, and sending a stop operation command to the tower crane control system to prevent safety accidents. For low-risk scenarios without violations, the system continuously records detection data and equipment status without triggering alarms. At the same time, the system stores all monitoring data in real time, including detection results (target information and attributes), violation events (type, time, location), early warning records (early warning level, response action), and sensor data. It supports both local storage and remote push output methods and can seamlessly integrate with the digital construction site platform to achieve centralized safety management and post-event traceability analysis.

[0073] Furthermore, to ensure the stability of detection accuracy during long-term deployment, this embodiment also includes a model iteration and update mechanism. Specifically, 10,000 real-world image data are collected every 3 months, and corresponding training samples are generated through the aforementioned intelligent data augmentation system. Incremental training is then performed on the deployed model, with a learning rate of 1e-5 and 200 iterations. Simultaneously, various validation metrics are continuously monitored, including mean accuracy (mAP), detection accuracy for each category, violation detection accuracy, and false positive rate. If the detection accuracy for a certain category falls below a preset threshold (e.g., 85%), an anomaly handling process is triggered, returning to optimize the multi-label loss function or specifically supplementing the scene training data for that category. Through continuous iterative optimization, the model maintains a consistently high level of detection performance.

[0074] like Figure 2As shown, the entire process from obtaining training samples to edge deployment application is fully demonstrated. The detection method for tower crane safety provided in this embodiment has achieved the following beneficial effects through the improvement of the YOLOv5 model and the corresponding data augmentation and edge deployment optimization: (1) The multi-target multi-label detection capability is significantly improved: By setting corresponding output channels for basic categories and extended attributes respectively, and using basic loss function and full-class loss function to supervise the prediction results in the first tensor and the second tensor respectively during the training process, the multi-target detection model after training can output basic category label and extended attribute label for each target in a single detection. The model can accurately identify basic categories such as personnel, hooks, and suspended objects, and accurately determine whether personnel are located around the suspended object and whether the suspended object is a long or short suspension, effectively distinguishing between high-risk scenarios (such as personnel being located around the suspended object) and low-risk scenarios (such as personnel not being located around the suspended object). (2) Significantly reduced data acquisition and labeling costs: By using the SAM2 pre-trained segmentation model to perform pixel-level accurate segmentation on the original tower crane operation images, and automatically generating synthetic images based on the segmentation mask according to various synthesis rules, and automatically generating accurate labels corresponding to the synthetic images, the technical problems of scarce high-risk samples and high labeling costs in real-world scenarios are effectively solved. The amount of real data required is reduced by more than 70%, and more than 5,000 high-quality synthetic images can be generated from 500 original images. The workload of manual labeling is reduced by more than 90%, and the labeling efficiency is improved by more than 10 times, which significantly reduces the cost of technology implementation and accelerates model iteration and scenario adaptation. (3) Extreme optimization of edge deployment performance: By converting the trained model to ONNX format and performing operator compatibility testing, the model structure is adapted according to the testing results. The model is divided into a first sub-model deployed on the neural network processor and a second sub-model deployed on the central processing unit. Layer-by-layer quantization accuracy decision and hybrid quantization strategy configuration are performed on the first sub-model, and finally, a deployment model adapted to the neural network processor is generated. The model size was compressed by more than 75% (from 45MB to 11MB), the inference speed was increased by more than 10 times (from 150ms to 15ms), and the real-time detection speed reached more than 30 frames / second. The quantization accuracy loss was controlled within 1.2%. At the same time, the compatibility problem of RKNN operator was solved, and seamless and efficient deployment on edge computing devices was achieved. (4) The environmental adaptability and scene generalization ability were significantly enhanced: through the combined application of various synthesis rules such as personnel and hanging object scene synthesis, hanging object replacement synthesis, and large model assisted synthesis, as well as the supplement of traditional data augmentation strategies, the generated synthetic images covered diverse scenes under different lighting, weather, and shooting angles, as well as various high-risk scenes and hanging object types.The model can adapt to complex and ever-changing construction environments, maintaining high detection accuracy even in harsh conditions. Its environmental adaptability is improved by more than 30%, and it can quickly adapt to the personalized monitoring needs of different construction sites. It does not require a large amount of on-site data for secondary training, significantly enhancing the model's generalization ability and practicality.

[0075] Example 2 This application provides a detection device for tower crane safety, such as... Figure 3 As shown, the device specifically includes the following components: The acquisition module 301 is used to acquire a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; Module 302 is used to build an initial YOLOv5 detection model, setting up corresponding basic output channels for each basic category and corresponding extended output channels for each extended attribute. Training module 303 is used to input the training sample set into the initial YOLOv5 detection model for iterative training. During the training process, the model's prediction results on each basic output channel are recorded by the first tensor, and the model's prediction results on each basic output channel and each extended output channel are recorded by the second tensor. The parameter tuning module 304 is used to calculate the first error between the prediction result in the first tensor and all basic category labels using the basic loss function, calculate the second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels using the full category loss function, and update the model parameters based on the first error and the second error. The processing module 305 is used to obtain a multi-target detection model applied to the field of tower crane safety detection when a preset training termination condition is met; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and determine the safety status based on the added labels.

[0076] Specifically, the basic categories include: personnel, hooks, and suspended objects; the extended attributes include: personnel associated with personnel located around and not located around the suspended object, as well as long and short hooks, non-long and short hooks, and other suspended objects associated with the suspended object.

[0077] Furthermore, the device also includes a sample module for: Acquire raw tower crane operation images; A pre-trained segmentation model is used to perform pixel-level segmentation on the detected targets in the original tower crane operation image to obtain a segmentation mask corresponding to each detected target; Based on the segmentation mask, and according to the preset synthesis rules, the detected targets in the original tower crane operation image are recombined, replaced, and / or added to generate multiple synthesized images, and a label corresponding to each synthesized image is generated. The generated synthetic images are subjected to quality screening, and the screened synthetic images are used as training sample images to form the training sample set.

[0078] Furthermore, the preset synthesis rules include at least one of the following: Personnel and suspended object scene compositing rules: The size of the suspended object is calculated using the segmentation mask of the suspended object, and a danger zone is delineated in the original tower crane operation image according to the size of the suspended object; the personnel target is determined using the segmentation mask of the personnel, and the personnel target is placed in the danger zone to generate a high-risk composite image, and the personnel target is placed outside the danger zone to generate a low-risk composite image; The rules for replacing and compositing suspended objects are as follows: The suspended object target is determined using a segmentation mask; the suspended object target is removed from the original tower crane operation image and the background is filled to obtain a background image; a target suspended object sample is selected from a pre-built suspended object sample library, which contains suspended object samples of different categories and / or different postures; the target suspended object sample is resized and its viewing angle is transformed according to the scene information of the original tower crane operation image; and the processed target suspended object sample is then merged into the background image to generate a composite image. Large model-assisted synthesis rules: Input the original tower crane operation image and preset scene description information into the multimodal large model to generate detailed scene description information that conforms to the tower crane operation scenario; input the detailed scene description information into the image generation model to generate a synthesized image.

[0079] Furthermore, the device also includes an adapter module for: Convert the trained multi-object detection model into an ONNX format model; Operator compatibility testing is performed on the ONNX format model to identify operators incompatible with the neural network processor in the target edge computing device; wherein, the target edge computing device is an edge computing device used to deploy the multi-target detection model; Based on the operator compatibility detection results, the model structure of the ONNX format model is adapted to obtain an adapted model structure including a first sub-model and a second sub-model. The adaptation process includes: simplifying the detection head of the ONNX format model to remove post-processing operations, thereby causing the simplified detection head to output an undecoded feature map; and dividing the ONNX format model into a first sub-model and a second sub-model. The first sub-model performs feature extraction and preliminary transformation, outputting an undecoded feature map. The second sub-model is a post-processing module used to post-process the undecoded feature map output by the first sub-model to obtain the final multi-target detection result. The first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor, and the second sub-model is deployed on the central processing unit in the form of program code.

[0080] Furthermore, the adapter module is specifically used for: Obtain the raw accuracy of the first sub-model on the validation set; wherein, the raw accuracy is the accuracy of the first sub-model without any quantization processing; The process iterates through each network layer in the first sub-model, performing a quantization precision decision on the currently traversed target network layer to determine the quantization precision of each network layer. The quantization precision decision includes: firstly, quantizing the target network layer according to a first quantization precision and calculating a first precision loss before and after the quantization adjustment; wherein the first quantization precision is a quantization precision lower than the original precision; if the first precision loss exceeds a preset loss threshold, then quantizing the target network layer according to a second quantization precision and calculating a second precision loss before and after the quantization adjustment; wherein the second quantization precision is a quantization precision between the original precision and the first quantization precision; if the second precision loss still exceeds the preset loss threshold, then maintaining the original precision of the target network layer. Based on the quantization precision determined by the quantization precision decision for each network layer, a hybrid quantization strategy is formed; wherein, the hybrid quantization strategy is a combination scheme that uses different quantization precisions for different network layers. Based on the hybrid quantization strategy, the first sub-model is quantized and calibrated using a calibration image set to generate a quantized first sub-model; wherein, the quantization calibration is the process of determining the quantization parameters of each network layer through the calibration image set. The quantized first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor.

[0081] In addition, the device also includes an application module for: The real-time acquired tower crane operation monitoring video or image is used as the image to be detected and input into the trained multi-target detection model; The first sub-model in the multi-object detection model is used to detect objects in the image to be detected, and features are extracted based on the detection results to output an undecoded feature map. The undecoded feature map is post-processed by the second sub-model in the multi-object detection model to obtain the prediction result of each detected target in the image to be detected in each output channel. Based on the prediction results of each output channel, a corresponding label is added to each detected target, and the safety status is determined based on the added label.

[0082] Example 3 This embodiment also provides a computer device, such as a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster composed of multiple servers), etc., capable of executing programs. Figure 4 As shown, the computer device 40 in this embodiment includes, but is not limited to, a memory 401 and a processor 402 that are communicatively connected to each other via a system bus. It should be noted that... Figure 4 Only a computer device 40 with components 401-402 is shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0083] In this embodiment, the memory 401 (i.e., the readable storage medium) includes flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 401 may be an internal storage unit of the computer device 40, such as the hard disk or memory of the computer device 40. In other embodiments, the memory 401 may also be an external storage device of the computer device 40, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the computer device 40. Of course, the memory 401 may include both the internal storage unit and the external storage device of the computer device 40. In this embodiment, the memory 401 is typically used to store the operating system and various application software installed on the computer device 40. In addition, the memory 401 may also be used to temporarily store various types of data that have been output or will be output.

[0084] In some embodiments, processor 402 may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. This processor 402 is typically used to control the overall operation of computer device 40.

[0085] Specifically, in this embodiment, the processor 402 is used to execute the program for the detection method of tower crane safety stored in the memory 401. When the program for the detection method of tower crane safety is executed, it performs the following steps: Obtain a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; Build an initial YOLOv5 detection model, set the corresponding basic output channel for each basic category, and set the corresponding extended output channel for each extended attribute; The training sample set is input into the initial YOLOv5 detection model for iterative training. During the training process, the prediction results of the model on each basic output channel are recorded by the first tensor, and the prediction results of the model on each basic output channel and each extended output channel are recorded by the second tensor. The first error between the prediction result in the first tensor and all basic category labels is calculated using the basic loss function. The second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels is calculated using the all-class loss function. The model parameters are then updated based on the first error and the second error. When the preset training termination condition is met, a multi-target detection model for tower crane safety inspection is obtained after training; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and to determine the safety status based on the added labels.

[0086] For a detailed description of the above method steps, please refer to Example 1. This example will not be repeated here.

[0087] Example 4 This embodiment also provides a computer-readable storage medium, such as flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, server, app store, etc., which stores a computer program. When the computer program is executed by a processor, it implements the following method steps: Obtain a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; Build an initial YOLOv5 detection model, set the corresponding basic output channel for each basic category, and set the corresponding extended output channel for each extended attribute; The training sample set is input into the initial YOLOv5 detection model for iterative training. During the training process, the prediction results of the model on each basic output channel are recorded by the first tensor, and the prediction results of the model on each basic output channel and each extended output channel are recorded by the second tensor. The first error between the prediction result in the first tensor and all basic category labels is calculated using the basic loss function. The second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels is calculated using the all-class loss function. The model parameters are then updated based on the first error and the second error. When the preset training termination condition is met, a multi-target detection model for tower crane safety inspection is obtained after training; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and to determine the safety status based on the added labels.

[0088] For a detailed description of the above method steps, please refer to the first embodiment. This embodiment will not repeat the details here.

[0089] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0090] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0091] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0092] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for detecting the safety of tower cranes, characterized in that, The method includes: Obtain a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; Build an initial YOLOv5 detection model, set the corresponding basic output channel for each basic category, and set the corresponding extended output channel for each extended attribute; The training sample set is input into the initial YOLOv5 detection model for iterative training. During the training process, the prediction results of the model on each basic output channel are recorded by the first tensor, and the prediction results of the model on each basic output channel and each extended output channel are recorded by the second tensor. The first error between the prediction result in the first tensor and all basic category labels is calculated using the basic loss function. The second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels is calculated using the all-class loss function. The model parameters are then updated based on the first error and the second error. When the preset training termination condition is met, a multi-target detection model for tower crane safety inspection is obtained after training; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and to determine the safety status based on the added labels.

2. The method for detecting tower crane safety according to claim 1, characterized in that, The basic categories include: personnel, hooks, and suspended objects; the extended attributes include: personnel associated with personnel who are located around the suspended object and personnel who are not located around the suspended object, as well as long and short hooks, non-long and short hooks, and other suspended objects associated with the suspended object.

3. The method for detecting tower crane safety according to claim 1, characterized in that, Before obtaining the training sample set containing multiple training sample images, the method further includes: Acquire raw tower crane operation images; A pre-trained segmentation model is used to perform pixel-level segmentation on the detected targets in the original tower crane operation image to obtain a segmentation mask corresponding to each detected target; Based on the segmentation mask, and according to the preset synthesis rules, the detected targets in the original tower crane operation image are recombined, replaced, and / or added to generate multiple synthesized images, and a label corresponding to each synthesized image is generated. The generated synthetic images are subjected to quality screening, and the screened synthetic images are used as training sample images to form the training sample set.

4. The method for detecting tower crane safety according to claim 3, characterized in that, The preset synthesis rules include at least one of the following: Personnel and suspended object scene composition rules: Calculate the size of the suspended object using the segmentation mask of the suspended object, and delineate the danger zone in the original tower crane operation image according to the size of the suspended object; Personnel targets are identified using personnel segmentation masks, and the personnel targets are placed within the danger zone to generate a high-risk composite image, while the personnel targets are placed outside the danger zone to generate a low-risk composite image. The rules for replacing and compositing suspended objects are as follows: The suspended object target is determined using a segmentation mask; the suspended object target is removed from the original tower crane operation image and the background is filled to obtain a background image; a target suspended object sample is selected from a pre-built suspended object sample library, which contains suspended object samples of different categories and / or different postures; the target suspended object sample is resized and its viewing angle is transformed according to the scene information of the original tower crane operation image; and the processed target suspended object sample is then merged into the background image to generate a composite image. Large model-assisted synthesis rules: Input the original tower crane operation image and preset scene description information into the multimodal large model to generate detailed scene description information that conforms to the tower crane operation scenario; input the detailed scene description information into the image generation model to generate a synthesized image.

5. The method for detecting tower crane safety according to claim 1, characterized in that, After obtaining the trained multi-target detection model for tower crane safety inspection when the preset training termination condition is met, the method further includes: Convert the trained multi-object detection model into an ONNX format model; Operator compatibility testing is performed on the ONNX format model to identify operators incompatible with the neural network processor in the target edge computing device; wherein, the target edge computing device is an edge computing device used to deploy the multi-target detection model; Based on the operator compatibility detection results, the model structure of the ONNX format model is adapted to obtain an adapted model structure including a first sub-model and a second sub-model. The adaptation process includes: simplifying the detection head of the ONNX format model to remove post-processing operations, thereby causing the simplified detection head to output an undecoded feature map; and dividing the ONNX format model into a first sub-model and a second sub-model. The first sub-model performs feature extraction and preliminary transformation, outputting an undecoded feature map. The second sub-model is a post-processing module used to post-process the undecoded feature map output by the first sub-model to obtain the final multi-target detection result. The first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor, and the second sub-model is deployed on the central processing unit in the form of program code.

6. The method for detecting tower crane safety according to claim 5, characterized in that, The step of converting the first sub-model into a deployment model format adapted to the neural network processor and deploying it on the neural network processor, and deploying the second sub-model in the form of program code on the central processing unit, includes: Obtain the raw accuracy of the first sub-model on the validation set; wherein, the raw accuracy is the accuracy of the first sub-model without any quantization processing; The process iterates through each network layer in the first sub-model, performing a quantization precision decision on the currently traversed target network layer to determine the quantization precision of each network layer. The quantization precision decision includes: firstly, quantizing the target network layer according to a first quantization precision and calculating a first precision loss before and after the quantization adjustment; wherein the first quantization precision is a quantization precision lower than the original precision; if the first precision loss exceeds a preset loss threshold, then quantizing the target network layer according to a second quantization precision and calculating a second precision loss before and after the quantization adjustment; wherein the second quantization precision is a quantization precision between the original precision and the first quantization precision; if the second precision loss still exceeds the preset loss threshold, then maintaining the original precision of the target network layer. Based on the quantization precision determined by the quantization precision decision for each network layer, a hybrid quantization strategy is formed; wherein, the hybrid quantization strategy is a combination scheme that uses different quantization precisions for different network layers. Based on the hybrid quantization strategy, the first sub-model is quantized and calibrated using a calibration image set to generate a quantized first sub-model; wherein, the quantization calibration is the process of determining the quantization parameters of each network layer through the calibration image set. The quantized first sub-model is converted into a deployment model format adapted to the neural network processor and deployed on the neural network processor.

7. The method for detecting tower crane safety according to any one of claims 1 to 6, characterized in that, After obtaining the trained multi-target detection model for tower crane safety inspection when the preset training termination condition is met, the method further includes: The real-time acquired tower crane operation monitoring video or image is used as the image to be detected and input into the trained multi-target detection model; The first sub-model in the multi-object detection model is used to detect objects in the image to be detected, and features are extracted based on the detection results to output an undecoded feature map. The undecoded feature map is post-processed by the second sub-model in the multi-object detection model to obtain the prediction result of each detected target in the image to be detected in each output channel. Based on the prediction results of each output channel, a corresponding label is added to each detected target, and the safety status is determined based on the added label.

8. A detection device for tower crane safety, characterized in that, The device includes: The acquisition module is used to acquire a training sample set containing multiple training sample images; wherein each training sample image contains one or more detection targets in the tower crane operation scene, and each detection target is labeled with a basic category label, or labeled with a basic category label and an extended attribute label that have an association relationship; The building module is used to build the initial YOLOv5 detection model, setting the corresponding basic output channel for each basic category and the corresponding extended output channel for each extended attribute; The training module is used to input the training sample set into the initial YOLOv5 detection model for iterative training. During the training process, the model's prediction results on each basic output channel are recorded through the first tensor, and the model's prediction results on each basic output channel and each extended output channel are recorded through the second tensor. The parameter tuning module is used to calculate the first error between the prediction result in the first tensor and all basic category labels using the basic loss function, calculate the second error between the prediction result in the second tensor and all labels including basic category labels and extended attribute labels using the all-class loss function, and update the model parameters based on the first error and the second error. The processing module is used to obtain a trained multi-target detection model for tower crane safety detection when a preset training termination condition is met; wherein, the multi-target detection model is used to add one or more corresponding labels to the detection targets in the image to be detected, and determine the safety status based on the added labels.

9. A computer device, the computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images