Safety Monitoring Method and System for Lifted Articles of Tower Crane

Abstract

The present invention discloses a safety monitoring method and system for the lifted objects of a tower crane. The method includes obtaining monitoring information in real time, where the monitoring information includes video monitoring information and sensor monitoring information; performing image recognition on the obtained monitoring images according to a pre-trained deep learning image recognition model to identify the target data information corresponding to each target in the monitoring images in real time; determining the inclination angle between the hook and the lifted object and the distance between the lifted object and the obstacle according to the pixel information of the hook and the lifted object and the sensor monitoring information; and determining whether there are potential hazards according to the inclination angle between the hook and the lifted object, the distance between the lifted object and the obstacle, and a preset warning threshold. The whole process of the present invention does not require manual observation and monitoring. When an abnormal situation is identified, a warning can be issued to ensure construction safety. It does not rely on cloud computing and is applicable to construction environments with unstable network conditions or high real-time requirements.

Description

Technical Field

[0001] The present invention relates to the technical field of tower crane safety monitoring, and particularly to a method and system for monitoring the safety of lifted objects by a tower crane. Background Art

[0002] In modern construction projects, tower cranes, as indispensable heavy equipment, bring many safety risks while ensuring the project progress. Especially in the high-altitude operation environment, due to factors such as blocked vision, limited viewing angles, and the diversity of lifted objects, it is very difficult for operators to monitor the status of lifted objects in real time and accurately. These challenges not only affect work efficiency, but more importantly, they pose great potential safety hazards to the safety management of the construction site. The number of personal injuries and property losses caused by accidents of lifted objects every year is astonishing, and effective technical means are urgently needed to prevent them.

[0003] Currently, the mainstream operation mode of lifted objects by tower cranes still involves the cooperation between the tower crane driver and the ground commander. In specific operations, the ground commander mainly remotely commands the tower crane driver to perform operations such as slewing the boom, moving the trolley, and controlling the up and down movement of the hook in a certain gear through a walkie-talkie and standard gestures. In addition, in some intelligent tower cranes equipped with hook cameras, the driver can also observe the situation near the hook in real time through the camera, but it is only limited to operator monitoring and does not achieve automatic recognition and intelligent support.

[0004] In some existing new technologies, there are some methods for identifying lifted objects based on 3D point clouds. By mounting a lidar on the tower crane to scan and model the lifted object, and classifying and identifying the modeled point cloud model through a 3D vision deep learning algorithm. However, for this method, additional lidar and 3D point cloud modeling equipment need to be installed. These equipment are costly and are easily damaged in a complex environment such as a construction site. At the same time, these equipment are also easily damaged during the frequent disassembly and assembly process, resulting in high maintenance costs and construction costs. And during the operation of the lidar and 3D point cloud modeling equipment, a large amount of storage space is required. Usually, it is necessary to communicate with a cloud server for calculation and storage. In a complex construction site environment, it is very easy to have poor signal conditions, resulting in difficult use. Summary of the Invention

[0005] Aiming at the deficiencies of the prior art, the present invention provides a method and system for monitoring the safety of lifted objects by a tower crane. It can automatically identify and judge the condition of the lifted object without manual observation and monitoring, improving the safety and operation efficiency of the hoisting operation.

[0006] According to the first aspect of the present invention, a method for monitoring the safety of lifted objects by a tower crane is provided, including:

[0007] Obtain monitoring information in real time, where the monitoring information includes video monitoring information and sensor monitoring information. Among them, the video monitoring information includes a monitoring screen with several targets, and the targets include a hook and a lifted object;

[0008] Perform image recognition on the obtained monitoring screen according to a pre-trained deep learning image recognition model to identify the target data information corresponding to each target in the monitoring screen in real time. The target data information includes the target category and the pixel information of the target, and the target categories include a hook, a lifted object, and an obstacle;

[0009] Determine the spatial coordinates of the lifted object and the inclination angle between the hook and the lifted object according to the pixel information of the hook and the lifted object and the sensor monitoring information;

[0010] Determine the distance between the lifted object and the obstacle according to the pixel information of the lifted object and the pixel information of the obstacle;

[0011] Determine whether there are potential hazards according to the inclination angle between the hook and the lifted object, the distance between the lifted object and the obstacle, and a preset warning threshold.

[0012] The monitoring devices required for the method of the present invention are only cameras and sensors. Then, through image recognition, the category of each target in the camera and the corresponding pixel information in the monitoring screen are determined. Furthermore, the spatial coordinates of the lifted object and the inclination angle between the hook and the lifted object are calculated through the information obtained by the sensors and the pixel information of each target, and the distance between the lifted object and the obstacle is determined, effectively reducing safety accidents caused by blocked vision or misjudgment. Thus, it can further quickly determine whether there are potential safety hazards in the current situation, automatically analyze the size, shape, and position of the lifted object, timely feedback the relative position and dynamic state of the lifted object and other objects, and reduce the risks of collision and falling. The whole process does not require manual observation and monitoring. When an abnormal situation (such as the lifted object being overweight, the sling being abnormally stressed, or the lifted object being tilted too much) is recognized, a warning can also be issued and real-time feedback can be provided to the operator to ensure construction safety and lay a foundation for the tower crane to further realize automated operations (such as path planning and automatic obstacle avoidance of the lifted object). And the present invention does not rely on cloud computing and can complete data processing and reasoning based on edge computing devices, which is suitable for construction environments with unstable network conditions or high real-time requirements.

[0013] In some embodiments, the monitoring screen includes a first monitoring screen obtained by a first camera module and a second monitoring screen obtained by a second camera module. The first camera module is arranged on the luffing trolley and obtains the first monitoring screen in the direction of the hook, and the second camera module is arranged at the end of the long section of the tower arm and obtains the second monitoring screen in the direction of the hook.

[0014] Thus, by setting it in this way, the position of the hook and the lifted object can be obtained from two monitoring pictures in different directions, so that the spatial position of the hook and the lifted object and the positional relationship with other obstacles can be better determined.

[0015] In some embodiments, the sensor monitoring information includes the vertical height data of the hook, the horizontal distance data of the hook relative to the center of the tower crane, and the rotation angle data of the tower arm.

[0016] Thus, by setting it in this way, the height, horizontal distance, and rotation angle of the current hook relative to the center of the tower crane can be determined from these data, so that the spatial coordinate position of the current hook can be determined from these data.

[0017] In some embodiments, the pre-trained deep learning image recognition model is a model based on YOLO11.

[0018] Thus, by setting it in this way, the YOLO11 model can be used to perform real-time image recognition on the monitoring picture. The YOLO11 model has the characteristics of high precision and low latency. After testing, the recognition rate of the trained model can reach 70% or more when performing real-time image recognition on the monitoring picture, and the hook recognition rate can reach 94%, thus effectively ensuring the accuracy and speed of the recognition detection.

[0019] In some embodiments, determining the spatial coordinates of the lifted object and the inclination angle between the hook and the lifted object according to the pixel information of the hook and the lifted object and the sensor monitoring information includes:

[0020] Determining the spatial coordinates of the hook according to the sensor monitoring information;

[0021] Determining the pixel space size ratio according to the calibrated size of the hook and the pixel information of the hook;

[0022] Determining the size of the lifted object according to the pixel space size ratio and the pixel information of the lifted object;

[0023] Determining the pixel offset between the hook and the lifted object according to the pixel information of the hook and the pixel information of the lifted object;

[0024] Determining the actual offset between the hook and the lifted object according to the pixel offset between the hook and the lifted object and the pixel space size ratio;

[0025] Determining the spatial coordinates of the lifted object according to the spatial coordinates of the hook and the actual offset between the hook and the lifted object;

[0026] Determine the inclination angle between the lifting hook and the lifted object according to the spatial coordinates of the lifting hook and the spatial coordinates of the lifted object.

[0027] Thus, by setting it like this, the spatial coordinates of the lifted object and the inclination angle between the lifting hook and the lifted object can be calculated, thereby forming a basis for judging potential safety hazards.

[0028] In some embodiments, the pixel information includes the pixel center position of the bounding box of the target and the bounding box size information;

[0029] Determine the distance between the lifted object and the obstacle according to the pixel information of the lifted object and the pixel information of the obstacle, including:

[0030] Determine the obstacle closest to the lifted object according to the pixel center position of the bounding box of the lifted object and the pixel center positions of the bounding boxes of the respective obstacles;

[0031] Determine the distance between the lifted object and the obstacle according to the pixel center position and the bounding box size information of the bounding box of the obstacle closest to the lifted object and the pixel center position and the bounding box size information of the bounding box of the lifted object.

[0032] Thus, by setting it like this, the distance between the lifted object and the obstacle can be calculated, thereby predicting the probability of collision between the lifted object and the obstacle.

[0033] According to a second aspect of the present invention, there is provided a safety monitoring system for a lifted object of a tower crane, including:

[0034] A monitoring module for real-time acquisition of monitoring information, the monitoring information including video monitoring information and sensor monitoring information, wherein the video monitoring information includes a monitoring screen with a number of targets, and the targets include a lifting hook and a lifted object;

[0035] An image recognition module for performing image recognition on the monitoring screen acquired by the monitoring module according to a pre-trained deep learning image recognition model to real-time identify the target data information corresponding to each target in the monitoring screen, the target data information including the target category and the pixel information of the target, and the target category including a lifting hook, a lifted object, and an obstacle;

[0036] A first data processing module for determining the spatial coordinates of the lifted object and the inclination angle between the lifting hook and the lifted object according to the pixel information of the lifting hook and the lifted object obtained by the image recognition module and the sensor monitoring information acquired by the monitoring module;

[0037] A second data processing module, configured to determine the distance between the lifted object and the obstacle according to the pixel information of the lifted object and the pixel information of the obstacle obtained by the image recognition module;

[0038] A hidden danger detection module, configured to determine whether there is a hidden danger according to the inclination angle between the hook and the lifted object obtained by the first data processing module, the distance between the lifted object and the obstacle obtained by the second data processing module, and a preset warning threshold.

[0039] The safety detection system for the lifted object of the tower crane according to the present invention obtains monitoring information through a monitoring module, uses an image recognition module to identify each target in the monitoring screen, and obtains the pixel information of each target, so that the condition of the lifted object and the situation between the lifted object and the obstacle can be obtained by using this information, and then it can be determined whether there is a hidden danger, automatically analyze the size, shape and position of the lifted object, timely feedback the relative position and dynamic state of the lifted object and other objects, and reduce the risks of collision and falling. The whole process does not require manual observation and monitoring. When an abnormal situation (such as the lifted object being overweight, the sling being stressed abnormally, or the lifted object being tilted too much) is identified, a warning can also be issued and real-time feedback can be provided to the operator to ensure construction safety and lay a foundation for the tower crane to further realize automated operations (such as path planning and automatic obstacle avoidance of the lifted object). And the present invention does not rely on cloud computing, and can complete data processing and reasoning based on edge computing devices, and is applicable to construction environments with unstable network conditions or high requirements for real-time performance.

[0040] In some embodiments, the monitoring screen includes a first monitoring screen and a second monitoring screen, and the monitoring module includes a first camera module disposed on the luffing trolley to obtain the first monitoring screen in the direction of the hook, and a second camera module disposed at the end of the long section of the tower arm to obtain the second monitoring screen in the direction of the hook.

[0041] In some embodiments, the sensor monitoring information includes the vertical height data of the hook, the horizontal distance data of the hook relative to the center of the tower crane, and the rotation angle data of the tower arm. The monitoring module further includes a first sensor disposed on the hoisting system of the luffing trolley, a second sensor disposed at the connection between the luffing trolley and the tower arm, and a third sensor disposed at the rotation center of the tower arm.

[0042] In some embodiments, the image recognition module is an edge computing device, and the edge computing device is deployed with a deep learning image recognition model pre-trained based on YOLO11. Description of the Drawings

[0043] Figure 1 It is the overall flowchart of the safety monitoring method for the lifted object of the tower crane according to an embodiment of the present invention;

[0044] Figure 2 The architecture diagram of the deep learning image recognition model in the safety monitoring method for the lifted objects of the tower crane according to an embodiment of the present invention;

[0045] Figure 3 The schematic diagram of the image categories of the dataset used in the training of the deep learning image recognition model in the safety monitoring method for the lifted objects of the tower crane according to an embodiment of the present invention;

[0046] Figure 4 The confusion matrix diagram of the trained deep learning image recognition model in the safety monitoring method for the lifted objects of the tower crane according to an embodiment of the present invention;

[0047] Figure 5 The step flow chart of step S13 in the safety monitoring method for the lifted objects of the tower crane according to an embodiment of the present invention;

[0048] Figure 6 The step flow chart of step S14 in the safety monitoring method for the lifted objects of the tower crane according to an embodiment of the present invention;

[0049] Figure 7 The principle block diagram of the safety monitoring system for the lifted objects of the tower crane according to an embodiment of the present invention;

[0050] Figure 8 The structural schematic diagram of an embodiment of the electronic device of the present invention. Detailed implementation manners

[0051] The present invention will be further described in detail below with reference to the accompanying drawings.

[0052] To make the objectives, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are some but not all of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0053] It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments may be combined with each other.

[0054] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including storage devices.

[0055] In the present invention, terms such as "module", "device", "system", etc. refer to related entities applied to a computer, such as hardware, a combination of hardware and software, software, or software in execution. Specifically, for example, a component may, but is not limited to, be a process running on a processor, a processor, an object, an executable component, an execution thread, a program, and / or a computer. Further, an application program or a script program running on a server, and the server may both be components. One or more components may be in an execution process and / or thread, and the components may be localized on one computer and / or distributed between two or more computers, and may be run by various computer-readable media. The components may also communicate through local and / or remote processes according to a signal having one or more data packets, for example, a signal from data that interacts with another component in a local system, a distributed system, and / or interacts with other systems through a network in the Internet.

[0056] Finally, it should also be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprise" and "include" not only include those elements, but also include other elements not expressly listed, or also include elements inherent to such a process, method, article, or device. Without further limitation, an element defined by the statement "comprising..." does not exclude the existence of additional identical elements in the process, method, article, or device comprising the element.

[0057] Figure 1 Schematically shows the overall process of the safety monitoring method for the lifted item of a tower crane according to an embodiment of the present invention. Refer to Figure 1 As shown, the safety monitoring method for the lifted item of the tower crane of the present invention is implemented to include the following steps:

[0058] Step S11: Obtain monitoring information in real time. The monitoring information includes video monitoring information and sensor monitoring information. Among them, the video monitoring information includes a monitoring screen having a plurality of targets, and the targets include a hook and a lifted item;

[0059] Step S12: Perform image recognition on the acquired surveillance video according to a pre-trained deep learning image recognition model to identify in real time the target data information corresponding to each target in the surveillance video. The target categories include a hook, a lifted item, and an obstacle, and the target data information includes the target category and the pixel information of the target.

[0060] Step S13: Determine the spatial coordinates of the lifted item and the inclination angle between the hook and the lifted item according to the pixel information of the hook and the lifted item and the sensor monitoring information.

[0061] Step S14: Determine the distance between the lifted item and the obstacle according to the pixel information of the lifted item and the pixel information of the obstacle.

[0062] Step S15: Determine whether there are potential hazards according to the inclination angle between the hook and the lifted item, the distance between the lifted item and the obstacle, and a preset warning threshold.

[0063] In step S11, the acquired monitoring information is the data information required in the overall process of the present invention. The monitoring information includes video monitoring information and sensor monitoring information. The video monitoring information is the surveillance video acquired by the camera module, and these surveillance videos contain several targets, including a hook, a lifted item, and some items or personnel near the hook and the lifted item that may exist on the construction site. The sensor monitoring information is the data information regarding the current position of the hook. Specifically, the sensor monitoring information includes the vertical height data of the hook, the horizontal distance data of the hook relative to the center of the tower crane, and the rotation angle data of the tower arm, so as to be able to determine the current position state of the hook relative to the crane based on these data information.

[0064] In some possible implementation manners, the acquired surveillance video may include a first surveillance video and a second surveillance video. Among them, the first surveillance video is the surveillance video acquired by a camera disposed on the luffing trolley and facing the direction of the hook, and the second surveillance video is the surveillance video acquired by a camera disposed at the end of the long section of the tower arm and facing the direction of the hook. Thus, the situations of the hook and the lifted item can be monitored by using the surveillance videos in two different directions, so as to more comprehensively determine the position situations of the hook and the lifted item, and to more comprehensively determine the position situations between the lifted item and other targets.

[0065] After obtaining the monitoring information in step S11, the target detection and recognition can be performed on the monitoring screen in the monitoring information according to step S12, so as to determine the target data information of each target in the monitoring screen. Among them, these target data information includes the target category and the pixel information of the target. The target category includes a hook, a lifted item, and an obstacle. Among them, the obstacle is all targets detected and recognized except the hook and the lifted item, and all are considered as obstacles. And the pixel information of the target may include the center position of the bounding box pixel and the size of the bounding box. The center position of the bounding box pixel is composed of x_center and y_center, which are the center coordinates of the bounding box pixel. The size of the bounding box is composed of x_width and y_heigh, which are the size of the bounding box. Specifically, in step S12, a pre-trained deep learning image recognition model is used to perform image recognition on the monitoring screen to determine the target data information of each target in the monitoring screen.

[0066] Specifically, in some possible implementation manners, the deep learning image recognition model may be a model based on YOLO11. Figure 2Schematically shows the complete architecture of the YOLO11 model, which is divided into three main parts: the Backbone, the Neck, and the Head. Each part contains multiple modules to improve the detection performance. Among them, the Backbone is responsible for extracting the basic features of the image; the Neck realizes multi-scale feature fusion through multiple upsampling and splicing operations; the Head performs object detection tasks on feature maps of different scales. Backbone: The Backbone is responsible for extracting the basic features from the input image. The Backbone of YOLO11 contains multiple convolutional layers (Conv) and the feature extraction module C3k2. The C3k2 module first passes through a convolutional layer (CBS), then splits the features into several branches, each branch contains multiple C3k units, and finally completes the fusion output through the feature splicing (Concat) operation. In addition, the Backbone also includes the SPPF module for further compressing the features, and the C2PSA module enhances the ability to capture different spatial scales through PSA (Pyramid Spatial Attention). Neck: The Neck part further processes the features extracted by the Backbone and supports the fusion of multi-scale features. This part contains multiple upsampling (Upsample) and splicing operations, uses multiple C3k2 modules to strengthen the consistency of feature expression, and at the same time retains the spatial information from the Backbone. The C2PSA module is also applied in the Neck part to further enhance the feature performance of the model at different spatial scales. Head part: The Head part contains multiple detection layers for object detection on feature maps of different scales. The Head part ensures the detection ability for multi-scale objects through multiple feature splicing and C3k2 modules. Finally, the detection layer outputs the category and location of the object. By training the YOLO11 model using the dataset, the model can be used to identify each target in the acquired surveillance footage at the construction site to obtain the target data information corresponding to each target in the surveillance footage.

[0067] Specifically, in this embodiment, when training the YOLO11 model, the dataset used is the construction site image dataset SODA (Site Object Detection Dataset) for deep learning in the construction field publicly available by Tsinghua University and South China University of Technology. This dataset has completed the processes of image acquisition, data cleaning, and data annotation. And this dataset contains 19,846 images from different construction sites, different weather conditions, and construction stages, and 286,201 target objects are annotated, with a total of 15 categories that can basically cover the recognition scenarios of lifted items. Examples of image categories are as Figure 3As shown, it includes 15 categories in total, namely person, board, brick, cutter, hook, vest, wood, scaffold, Electricbox, fence, helmet, rebar, handcart, hopper, and slogan. This dataset adopts the general VOC2007 format, and its key attributes mainly include the picture size (width, height), category name (name), and the bounding box position coordinates of the target , where is the upper left corner coordinate of the bounding box, and ( is the lower right corner coordinate). Since the VOC data format cannot be directly used for model training, when this invention uses this dataset to train the YOLO11 model, it is converted into the YOLO general format for model training, that is, the following format: class_id x_cente y_center width height. Among them, class_id is the target category number (starting from 0), and x_center and y_center are the coordinates of the center point of the bounding box, normalized to the range of the picture width and height [0,1], and width and height are the width and height of the bounding box respectively, also normalized to the range of the picture width and height [0,1]. When performing format conversion, it is necessary to map the categories that will appear in the VOC format to category numbers. Specifically, a category dictionary can be predefined in advance, and its elements are {name: class_id}. In addition, it is also necessary to convert the bounding box coordinates from the absolute coordinate values in the VOC format to the relative coordinate values in the YOLO format. The specific conversion formula can be as follows:

[0068]

[0069]

[0070] When using this dataset to train the model, to verify the model effect, the entire dataset is divided according to the division ratio of 9:1 for the training set and the validation set. To improve the model generalization rate, the initial training weights can adopt YOLO11n pre-trained on the COCO dataset, and the parameters of the model are optimized and adjusted through 1000 rounds to ensure its high precision and real-time performance in the task of detecting lifted items. After the finally trained model undergoes a test experiment, the confusion matrix corresponding to this model is as Figure 4As shown in the figure, the darker the color in the figure, the larger the value. The trained YOLO11 model can well distinguish the differences between various classes, but there are also certain missed detection situations, that is, the corresponding classes are recognized as the background, among which the missed detection situations of safety helmets and reflective vests are relatively serious, 46% and 34% respectively. This is because their pixels in the image are small and not easy to check, and they usually appear together with the main person. In addition, the recognition rates of other classes can reach 70% and above, and the recognition rate of the hook can reach 94%. The present invention can deploy the trained model to edge computing devices. The main operations involve completing model optimization and inference acceleration. Different model optimization methods may be involved according to different edge device series adopted. The edge device adopted in the present invention is the Huawei Ascend series processor. It is necessary to first convert the weight of the trained pt model into the onnx format, and then convert it into the om format that can be recognized by the Ascend AI processor, so as to ensure that the edge computing device can correctly run the trained model to analyze and process the image data transmitted by the tower crane hook camera.

[0071] After performing image recognition on the acquired monitoring screen by using the deep learning image recognition model in step S12 to obtain the target category and pixel information corresponding to each target in the monitoring screen, step S13 can be executed to determine the current state of the lifted item. Among them, the current state of the lifted item includes the spatial coordinates of the lifted item and the inclination angle between the hook and the lifted item. Figure 5 Schematically shows the step flow of step S13 in the method for monitoring the safety of the lifted item of the tower crane according to an embodiment of the present invention. Refer to Figure 5 As shown, step S13 can be specifically implemented as including the following steps:

[0072] Step S21: Determine the spatial coordinates of the hook according to the sensor monitoring information;

[0073] Step S22: Determine the pixel space size ratio according to the calibrated size of the hook and the pixel information of the hook;

[0074] Step S23: Determine the size of the lifted item according to the pixel space size ratio and the pixel information of the lifted item;

[0075] Step S24: Determine the pixel offset between the hook and the lifted item according to the pixel information of the hook and the pixel information of the lifted item;

[0076] Step S25: Determine the actual offset between the hook and the lifted item according to the pixel offset between the hook and the lifted item and the pixel space size ratio;

[0077] Step S26: Determine the spatial coordinates of the lifted object according to the spatial coordinates of the lifting hook and the actual offset between the lifting hook and the lifted object;

[0078] Step S27: Determine the inclination angle between the lifting hook and the lifted object according to the spatial coordinates of the lifting hook and the spatial coordinates of the lifted object.

[0079] First, calculate and determine the spatial coordinates of the lifting hook according to Step S21. According to the sensor monitoring information, the current position information of the lifting hook relative to the crane can be determined. Specifically, in the sensor monitoring information, the vertical height data of the lifting hook is the Z-axis coordinate data of the lifting hook, the horizontal distance data of the lifting hook relative to the center of the tower crane is defined as R, and the rotation angle data of the tower arm is defined as the angle θ. Furthermore, the coordinate data of the lifting hook on the X-axis and Y-axis can be calculated according to the following formula:

[0080]

[0081] Combining the above calculation results, finally determine the coordinates of the lifting hook in three-dimensional space ( X , Y , Z ).

[0082] Then execute Step S22. Specifically, first determine the calibration size of the lifting hook ( L real , W real , Z real ). The calibration size of the lifting hook can be directly obtained according to the relevant parameter data of the lifting hook and determined by manual pre-input. Then, determine the pixel size information of the lifting hook in the monitoring screen according to the pixel information of the lifting hook. Specifically, the boundary box size in the pixel information can be used to represent the pixel size information of the lifting hook in the monitoring screen. Exemplarily, since in the above embodiment, the monitoring screen includes a first monitoring screen and a second monitoring screen, the pixel length and width values of the boundary box in the first monitoring screen can be used to represent the pixel length and width values of the lifting hook ( L pixel , W pixel ), and the width of the pixel box in the second monitoring screen is used to represent the pixel height of the lifting hook ( H pixel ), so as to obtain the pixel size information of the lifting hook in the monitoring screen ( L pixel , W pixel , H pixel ). After determining the calibration size of the lifting hook and the pixel size information of the lifting hook in the monitoring screen, the pixel space size ratio can be determined according to the following formulaP L , P W with P H :

[0083]

[0084] Then execute step S23. After determining the pixel space size ratio, only the pixel size information of the lifted item in the monitoring screen needs to be obtained, and the actual size of the lifted item can be determined according to the pixel space size ratio and the pixel size information. Similarly, the pixel size information of the lifted item in the monitoring screen can also be represented by the pixel length and width values of the bounding box in the first monitoring screen for the pixel length and width values of the lifted item ( L 被起吊物品像素 , W 被起吊物品像素 ), and the pixel height of the lifting hook is represented by the width of the pixel box in the second monitoring screen ( H 被起吊物品像素 ). After that, the actual size of the lifted item can be determined only through the following formula L 被起吊物品 , W 被起吊物品 and H 被起吊物品 :

[0085]

[0086] Then execute step S24 to determine the pixel offset between the lifting hook and the lifted item. Specifically, it can be determined according to the center position of the bounding box pixel in the pixel information of both the lifting hook and the lifted item. The pixel plane offset between the lifting hook and the lifted item is determined by the center position of the bounding box pixel of the lifting hook and the center position of the bounding box pixel of the lifted item in the first monitoring screen ( △x 像素 , △y 像素 ), and the pixel height offset of the lifted item is determined by the center position of the bounding box pixel of the lifting hook and the center position of the bounding box pixel of the lifted item in the second monitoring screen △z 像素 , obtaining the pixel offset between the lifting hook and the lifted item ( △x 像素 , △y 像素 , △z 像素 )

[0087] Then, step S25 is executed to determine the actual offset between the lifting hook and the lifted object based on the pixel offset between the lifting hook and the lifted object and the pixel space size ratio determined in step S22 ( △X , △Y , △Z ), which can be specifically determined by the following formula:

[0088]

[0089] After obtaining the actual offset between the lifting hook and the lifted object and the spatial coordinates of the lifting hook, step S26 can be executed to calculate the spatial coordinates of the lifted object ( X 被起吊物品 , Y 被起吊物品 , Z 被起吊物品 ). Specifically, it can be determined according to the following formula:

[0090]

[0091] Meanwhile, after obtaining the actual offset between the lifting hook and the lifted object, step S27 can be executed to calculate the inclination angle between the lifting hook and the lifted object. Specifically, the inclination angle can be represented by α , and its calculation formula is as follows:

[0092]

[0093] Thus, step S13 is completed to determine the spatial coordinates of the lifted object and the inclination angle between the lifting hook and the lifted object. Among them, the inclination angle between the lifting hook and the lifted object is one of the factors for determining whether there are potential hazards. After completing step S13, step S14 can be executed to determine another factor for determining whether there are potential hazards, that is, the distance between the lifted object and the obstacle. Figure 6 Schematically shows the step flow of step S14 in the safety monitoring method of the lifted object of the tower crane according to an embodiment of the present invention. Referring to Figure 6 shown, step S14 can be specifically implemented as including the following steps:

[0094] Step S31: Determine the obstacle closest to the lifted object according to the pixel center position of the bounding box of the lifted object and the pixel center positions of the bounding boxes of each obstacle;

[0095] Step S32: Determine the distance between the lifted object and the obstacle according to the pixel center position and the bounding box size information of the obstacle closest to the lifted object and the pixel center position and the bounding box size information of the lifted object.

[0096] It can be understood that in the monitoring screen, there may be multiple monitored targets, and thus there may be multiple obstacles. If the distances between all obstacles and the lifted item are calculated, the computational workload will be very large and computational resources will be wasted. Therefore, in step S31, the obstacle closest to the lifted item is first determined. When determining the obstacle closest to the lifted item, it can be determined based on the center position of the bounding box pixels in the pixel sizes of the lifted item and the obstacle. In the implementation where the monitoring screen includes a first monitoring screen and a second monitoring screen, the obstacle with the center position of the bounding box pixels closest to the lifted item in the first monitoring screen and the obstacle with the center position of the bounding box pixels closest to the lifted item in the second monitoring screen can both be regarded as the obstacle closest to the lifted item.

[0097] After that, step S32 is executed to calculate the distance between the corresponding obstacle and the lifted item. The specific calculation method is implemented according to relevant methods in the prior art, and the present invention does not limit this. Exemplarily, the distance between the center position of the bounding box pixels of the corresponding obstacle and the center position of the bounding box pixels of the lifted item can be first determined, and then the distance between the center position of the corresponding obstacle and the center position of the lifted item can be calculated through the pixel space size ratio determined in step S22, so as to be used as the distance between the lifted item and the obstacle.

[0098] Finally, after determining the tilt angle between the hook and the lifted item and the distance between the lifted item and the obstacle, the warning threshold set in advance can be used to determine whether the tilt angle between the hook and the lifted item and the distance between the lifted item and the obstacle exceed the safety range, so as to determine whether there is a safety hazard. Among them, the warning threshold set in advance can include a tilt angle threshold and a distance threshold, so as to be able to compare with the tilt angle between the hook and the lifted item and the distance between the lifted item and the obstacle to determine whether there is a hazard. When it is determined that there is a hazard, a warning prompt can be issued. The issued warning prompt can be a sound warning linked to a warning module such as a buzzer, or a visual warning linked to a warning light module such as a warning light, or a prompt warning in the form of a pop-up prompt window on the user interface or locking the rotation of the crane, etc. The present invention does not limit this.

[0099] The monitoring devices required for the method of the present invention are only cameras and sensors. Then, through image recognition, the categories of each target in the camera and the corresponding pixel information in the monitoring screen are determined. Furthermore, based on the information obtained by the sensors and the pixel information of each target, the spatial coordinates of the lifted object, the tilt angle between the hook and the lifted object are calculated, and the distance between the lifted object and the obstacle is determined, effectively reducing safety accidents caused by blocked vision or misjudgment. Thus, it can further quickly determine whether there are potential safety hazards in the current situation, automatically analyze the size, shape and position of the lifted object, and timely feedback the relative position and dynamic state of the lifted object and other objects, reducing the risks of collision and falling. The whole process does not require manual observation and monitoring. When abnormal situations (such as the lifted object being overweight, the sling being abnormally stressed or the lifted object tilting too much) are identified, warnings can be issued and real-time feedback can be provided to the operator to ensure construction safety, laying a foundation for the tower crane to further realize automated operations (such as path planning and automatic obstacle avoidance of the lifted object). And the present invention does not rely on cloud computing, and can complete data processing and reasoning based on edge computing devices, which is applicable to construction environments with unstable network conditions or high requirements for real-time performance.

[0100] Figure 7 Schematically shows a principle block diagram of a safety monitoring system for a lifted object of a tower crane according to an embodiment of the present invention. Referring to Figure 7 as shown, the safety monitoring system for a lifted object of the tower crane of the present invention includes:

[0101] A monitoring module 1 for real-time acquisition of monitoring information, where the monitoring information includes video monitoring information and sensor monitoring information. Among them, the video monitoring information includes a monitoring screen with several targets, and the targets include a hook and a lifted object;

[0102] An image recognition module 2 for performing image recognition on the monitoring screen obtained by the monitoring module according to a pre-trained deep learning image recognition model to real-time identify the target data information corresponding to each target in the monitoring screen. The target data information includes target categories and pixel information of the target, and the target categories include a hook, a lifted object and an obstacle;

[0103] A first data processing module 3 for determining the spatial coordinates of the lifted object and the tilt angle between the hook and the lifted object according to the pixel information of the hook and the lifted object obtained by the image recognition module and the sensor monitoring information obtained by the monitoring module;

[0104] A second data processing module 4 for determining the distance between the lifted object and the obstacle according to the pixel information of the lifted object and the pixel information of the obstacle obtained by the image recognition module;

[0105] The hidden danger detection module 5 is used to determine whether there is a hidden danger according to the inclination angle between the hook and the lifted object obtained by the first data processing module, the distance between the lifted object and the obstacle obtained by the second data processing module, and a preset warning threshold.

[0106] Among them, the monitoring screen includes a first monitoring screen and a second monitoring screen. The monitoring module includes a first camera module arranged on the luffing trolley to obtain the first monitoring screen in the direction of the hook, and a second camera module arranged at the end of the long section of the tower arm to obtain the second monitoring screen in the direction of the hook.

[0107] The sensor monitoring information includes the vertical height data of the hook, the horizontal distance data of the hook relative to the center of the tower crane, and the rotation angle data of the tower arm. The monitoring module further includes a first sensor arranged on the hoisting system of the luffing trolley, a second sensor arranged at the connection between the luffing trolley and the tower arm, and a third sensor arranged at the center of rotation of the tower arm.

[0108] The image recognition module is an edge computing device. The edge computing device deploys a deep learning image recognition model based on YOLO11 and trained using the construction site image dataset SODA (Site Object Detection Dataset) for deep learning in the construction field publicly disclosed by Tsinghua University and South China University of Technology as the dataset.

[0109] It should be noted that the implementation process and principle of the safety monitoring system for the lifted object of the tower crane in the embodiments of the present invention can be specifically referred to the corresponding descriptions in the above method embodiments. For example, the corresponding descriptions in the method embodiment part regarding the acquisition of the target data information corresponding to each target in the monitoring screen, the calculation of the inclination angle between the hook and the lifted object, and the calculation of the distance between the lifted object and the obstacle, etc., will not be elaborated herein. Exemplarily, the safety monitoring system for the lifted object of the tower crane in the embodiments of the present invention can be any intelligent device with a processor, including but not limited to computers, smartphones, personal computers, robots, cloud servers, etc.

[0110] In some embodiments, the present invention provides a non - volatile computer - readable storage medium. One or more programs including execution instructions are stored in the storage medium. The execution instructions can be read and executed by an electronic device (including but not limited to a computer, a server, or a network device, etc.) to execute the safety monitoring method for the lifted object of the tower crane in any of the above embodiments of the present invention.

[0111] In some embodiments, the embodiments of the present invention further provide a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by a computer, cause the computer to execute the method for safely monitoring the lifted items of a tower crane according to any one of the above embodiments.

[0112] In some embodiments, the embodiments of the present invention further provide an electronic device, which includes: at least one processor, and a memory communicatively connected to the at least one processor. Wherein, the memory stores instructions executable by the at least one processor, and when the instructions are executed by the at least one processor, the at least one processor is enabled to execute the method for safely monitoring the lifted items of a tower crane according to any one of the above embodiments.

[0113] In some embodiments, the embodiments of the present invention further provide a storage medium, on which a computer program is stored. It is characterized in that when the program is executed by a processor, the method for safely monitoring the lifted items of a tower crane according to any one of the above embodiments is implemented.

[0114] Figure 8 is a schematic hardware structure diagram of an electronic device for executing the method for safely monitoring the lifted items of a tower crane provided by another embodiment of the present application. As Figure 8 shown, the device includes:

[0115] One or more processors 610 and a memory 620. Figure 8 Here, one processor 610 is taken as an example.

[0116] The device for executing the method for safely monitoring the lifted items of a tower crane may further include: an input device 630 and an output device 640.

[0117] The processor 610, the memory 620, the input device 630 and the output device 640 may be connected through a bus or other means. Figure 8 Here, connection through a bus is taken as an example.

[0118] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the method for safely monitoring the lifted items of a tower crane in the embodiments of the present application. The processor 610 executes various functional applications and data processing of the server by running the non-volatile software programs, instructions, and modules stored in the memory 620, that is, implements the method for safely monitoring the lifted items of a tower crane in the above method embodiments.

[0119] The memory 620 may include a program storage area and a data storage area. The program storage area may store an operating system and application programs required for at least one function. The data storage area may store data created according to the use of the safety monitoring method for the lifted objects of the tower crane, etc. In addition, the memory 620 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other non-volatile solid-state storage devices. In some embodiments, the memory 620 may optionally include a memory remotely disposed relative to the processor 610, and these remote memories may be connected to the electronic device through a network. Examples of the above networks include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0120] The input device 630 may receive input digital or character information and generate signals related to user settings and function control of the image processing device. The output device 640 may include a display device such as a display screen.

[0121] The one or more modules are stored in the memory 620 and, when executed by the one or more processors 610, execute the safety monitoring method for the lifted objects of the tower crane in any of the above method embodiments.

[0122] The above product may execute the method provided in the embodiments of the present application, and has corresponding functional modules and beneficial effects for executing the method. Technical details not described in detail in this embodiment may be referred to the method provided in the embodiments of the present application.

[0123] The electronic device in the embodiments of the present application exists in various forms, including but not limited to:

[0124] (1) Mobile communication devices: Such devices are characterized by having mobile communication functions and mainly aim to provide voice and data communication. Such terminals include: smart phones (such as iPhone), multimedia phones, functional phones, and low-end phones, etc.

[0125] (2) Ultra-mobile personal computer devices: Such devices belong to the category of personal computers, have computing and processing functions, and generally also have the characteristic of mobile Internet access. Such terminals include: PDAs, MIDs, and UMPC devices, etc., such as iPad.

[0126] (3) Portable entertainment devices: Such devices can display and play multimedia content. Such devices include: audio and video players (such as iPod), handheld game consoles, e-books, and smart toys and portable vehicle navigation devices.

[0127] (4) Server: A device that provides computing services. The server consists of a processor, hard disk, memory, system bus, etc. The server is similar to a general computer architecture, but due to the need to provide highly reliable services, it has higher requirements in terms of processing power, stability, reliability, security, scalability, manageability, etc.

[0128] (5) Other electronic devices with data interaction functions.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0130] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution, in essence, or the part that makes contributions to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and are not intended to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of each embodiment of the present application.

Claims

1. A method for monitoring the safety of objects hoisted by a tower crane, characterized in that: include: Acquire monitoring information in real time, the monitoring information including video monitoring information and sensor monitoring information, wherein the video monitoring information includes a monitoring screen with several targets, the targets including a hook and a hoisted object, and the sensor monitoring information includes vertical height data of the hook, horizontal distance data of the hook relative to the center of the tower crane, and rotation angle data of the tower arm; Perform image recognition on the acquired monitoring screen according to the pre-trained deep learning image recognition model to identify the target data information corresponding to each target in the monitoring screen in real time, wherein the target data information includes the target category and the pixel information of the target, and the target category includes the hook, the hoisted object and the obstacle; Determine the spatial coordinates of the hoisted object and the tilt angle between the hook and the hoisted object according to the pixel information of the hook and the hoisted object and the sensor monitoring information; Determine the distance between the hoisted object and the obstacle according to the pixel information of the hoisted object and the pixel information of the obstacle; Determine whether there is a hidden danger based on the tilt angle between the hook and the hoisted object, the distance between the hoisted object and the obstacle, and the pre-set warning threshold; The method of determining the spatial coordinates of the hoisted object and the inclination angle between the hook and the hoisted object according to the pixel information of the hook and the hoisted object and the sensor monitoring information includes: Determine the spatial coordinates of the hook based on the sensor monitoring information; Determine the hook according to the calibrated size of the hook and the pixel information of the hook to determine the pixel space size ratio; Determine the size of the hoisted object according to the pixel space size ratio and the pixel information of the hoisted object; Determine the pixel offset between the hook and the hoisted object according to the pixel information of the hook and the pixel information of the hoisted object; Determine the actual offset between the hook and the hoisted object according to the pixel offset and pixel space size ratio between the hook and the hoisted object; Determine the spatial coordinates of the hoisted object according to the spatial coordinates of the hook and the actual offset between the hook and the hoisted object; Determine the inclination angle between the hook and the object being hoisted according to the spatial coordinates of the hook and the spatial coordinates of the object being hoisted; The acquired monitoring screen is subjected to image recognition according to a pre-trained deep learning image recognition model, so as to identify the target data information corresponding to each target in the monitoring screen in real time, and the process is executed by edge computing equipment arranged on the construction site.

2. The tower crane hoisted object safety monitoring method according to claim 1 is characterized in that: The monitoring screen includes a first monitoring screen obtained by a first camera module and a second monitoring screen obtained by a second camera module. The first camera module is arranged on the boom trolley to obtain the first monitoring screen in the direction of the hook, and the second camera module is arranged at the end of the long section of the tower arm to obtain the second monitoring screen in the direction of the hook.

3. The method for safety monitoring of objects hoisted by a tower crane according to claim 1, characterized in that: The pre-trained deep learning image recognition model is a model based on YOLO11.

4. The method for safety monitoring of objects hoisted by a tower crane according to claim 1, characterized in that: The pixel information includes the pixel center position and size information of the bounding box of the target; Determining the distance between the hoisted object and the obstacle according to the pixel information of the hoisted object and the pixel information of the obstacle includes: Determine the obstacle closest to the hoisted object according to the pixel center position of the bounding box of the hoisted object and the pixel center position of the bounding box of each obstacle; The distance between the hoisted object and the obstacle is determined according to the bounding box pixel center position and bounding box size information of the obstacle closest to the hoisted object and the bounding box pixel center position and bounding box size information of the hoisted object.

5. A tower crane hoisted object safety monitoring system, characterized in that: include: A monitoring module, used for acquiring monitoring information in real time, wherein the monitoring information includes video monitoring information and sensor monitoring information, wherein the video monitoring information includes a monitoring screen with a plurality of targets, wherein the targets include a hook and a hoisted object, and the sensor monitoring information includes vertical height data of the hook, horizontal distance data of the hook relative to the center of the tower crane, and rotation angle data of the tower arm; An image recognition module is used to perform image recognition on the monitoring screen acquired by the monitoring module according to a pre-trained deep learning image recognition model, so as to identify in real time the target data information corresponding to each target in the monitoring screen, wherein the target data information includes the target category and the pixel information of the target, and the target category includes the hook, the hoisted object and the obstacle, wherein the image recognition module is an edge computing device arranged on the construction site; The first data processing module is used to determine the spatial coordinates of the hoisted object and the inclination angle between the hook and the hoisted object according to the pixel information of the hook and the hoisted object obtained by the image recognition module and the sensor monitoring information obtained by the monitoring module, including: Determine the spatial coordinates of the hook based on the sensor monitoring information; Determine the hook according to the calibrated size of the hook and the pixel information of the hook to determine the pixel space size ratio; Determine the size of the hoisted object according to the pixel space size ratio and the pixel information of the hoisted object; Determine the pixel offset between the hook and the hoisted object according to the pixel information of the hook and the pixel information of the hoisted object; Determine the actual offset between the hook and the hoisted object according to the pixel offset and pixel space size ratio between the hook and the hoisted object; Determine the spatial coordinates of the hoisted object according to the spatial coordinates of the hook and the actual offset between the hook and the hoisted object; Determine the inclination angle between the hook and the object being hoisted according to the spatial coordinates of the hook and the spatial coordinates of the object being hoisted; A second data processing module is used to determine the distance between the hoisted object and the obstacle according to the pixel information of the hoisted object and the pixel information of the obstacle obtained by the image recognition module; The hidden danger detection module is used to determine whether there is a hidden danger based on the inclination angle between the hook and the hoisted object obtained by the first data processing module, the distance between the hoisted object and the obstacle obtained by the second data processing module, and a pre-set warning threshold.

6. The tower crane hoisted object safety monitoring system according to claim 5 is characterized in that: The monitoring screen includes a first monitoring screen and a second monitoring screen, and the monitoring module includes a first camera module arranged on the boom trolley to obtain the first monitoring screen in the direction of the hook, and a second camera module arranged at the end of the long section of the tower arm to obtain the second monitoring screen in the direction of the hook.

7. The tower crane hoisted object safety monitoring system according to claim 6 is characterized in that: The monitoring module also includes a first sensor arranged on the hoisting system of the luffing trolley, a second sensor arranged at the connection between the luffing trolley and the tower arm, and a third sensor arranged at the rotation center of the tower arm.

8. The tower crane hoisted object safety monitoring system according to claim 5 is characterized in that: The image recognition module is an edge computing device, and the edge computing device is deployed with a deep learning image recognition model pre-trained based on YOLO11.

Images