Tower crane teleoperation intelligent hoisting object identification system based on cloud side end architecture

Abstract

The invention belongs to the technical field of intelligent construction, and particularly discloses a tower crane teleoperation intelligent hoisting object identification system based on a cloud side end architecture, which comprises an element acquisition module, an element processing module and a cloud service platform, the data acquisition module is used for acquiring hoisted object state element data and hoisting operation image data during tower crane construction; the element processing module is connected with the cloud service platform and is used for processing the real-time hoisting object state element data and the hoisting operation image data at the edge end position based on a target detection model to obtain processed data; and the cloud service platform is used for identifying the hanging object based on a target classification model and the processed data to obtain an identification result, and is also used for training the target detection model and the target classification model and sending the target detection model to the element processing module.

Description

Technical Field

[0001] This application belongs to the technical field of intelligent construction, and more specifically, relates to a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture. Background Art

[0002] Currently, the research on intelligent unmanned tower cranes provides support for the digital transformation of the construction industry. Intelligent unmanned tower cranes require tower crane drivers to remotely operate. The driver remotely controls the tower crane in a remote operation room to complete the hoisting operation. During this process, the real-time positioning and recognition of the lifted object by the tower crane are the keys to realizing intelligent unmanned tower cranes, which are related to many downstream tasks, such as the intelligent realization of material statistics, hoisting efficiency analysis, active safety, construction progress analysis, and refined management of hoisting operations at the construction site.

[0003] Traditional material statistics rely on workers to visually identify the lifted objects and manually record them. This not only requires arranging special personnel but also is difficult to avoid the situation of missing records and misrecords, with low accuracy and efficiency. Summary of the Invention

[0004] Aiming at the defects of the prior art, the purpose of this application is to provide a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture, aiming to solve the problem of low accuracy and efficiency caused by the existing material statistics relying on workers to visually identify the lifted objects and manually record them.

[0005] To achieve the above purpose, in the first aspect, this application provides a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture, including an element acquisition module, an element processing module, and a cloud service platform: The element acquisition module is connected to the element processing module and is used to collect real-time lifted object state element data and hoisting operation image data during tower crane construction; The element processing module is connected to the cloud service platform and is used to process the real-time lifted object state element data and the hoisting operation image data based on an object detection model at the edge end to obtain processed data; The cloud service platform is used to identify the lifted object based on an object classification model and the processed data to obtain an identification result, and is also used to train the object detection model and the object classification model, and send the object detection model to the element processing module.

[0006] In this application, the element acquisition module collects real-time lifting object state element data and lifting operation image data during tower crane construction. The element processing module processes the data at the edge end closer to the data source to reduce data transmission latency. By using the collaborative computing of the edge end and the cloud service platform, distributed training and updating of the model are achieved. On the cloud service platform, the lifting object is identified based on the target classification model and the processed data to obtain the recognition result, improving the accuracy and efficiency of material statistics.

[0007] According to a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-end architecture provided by this application, the element acquisition module includes a lifting operation image information acquisition unit and a lifting object state perception unit: The lifting operation image information acquisition unit is configured at the bottom of the trolley walking beam and is used to obtain the real-time video stream directly below the tower crane hook. The lifting object state perception unit is used to obtain the lifting object state element data.

[0008] According to a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-end architecture provided by this application, the lifting operation image information acquisition unit includes a first intelligent digital camera. The first intelligent digital camera is deployed at the bottom of the trolley walking beam and is used to detect and real-time locate the hook and the lifting object based on a deep learning algorithm.

[0009] This application uses the first intelligent digital camera to detect and real-time locate the hook and the lifting object based on a deep learning algorithm, which can eliminate background noise interference and reduce the amount of image processing data. By reducing the computational complexity of image feature extraction, the overall processing latency of the system is reduced, and the real-time processing efficiency of the system is improved.

[0010] According to a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-end architecture provided by this application, the lifting operation image information acquisition unit further includes a spherical network camera. The spherical network camera is used to real-time obtain the tower crane operation image data based on an adaptive optimization algorithm for complex lighting scenarios.

[0011] According to a tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-end architecture provided by this application, the element acquisition module collaboratively transmits the lifting object state element data and the lifting operation image data through a multi-process transmission method and a 5G communication network.

[0012] This application combines multi-threaded queue management and a 5G communication network to reduce data transmission latency and ensure the real-time performance and accuracy of image information and lifting object state element information during the lifting object recognition process.

[0013] A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture provided by the present application, wherein the element processing module includes a second intelligent digital camera and an edge computing unit: The second intelligent digital camera is used to collect hoisting operation image data and also to preprocess the hoisting operation image data; The edge computing unit is deployed in the control cabinet at the root of the tower crane boom and is used to process the real-time lifting object state element data and the hoisting operation image data based on a target detection model to obtain processed data.

[0014] A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture provided by the present application, wherein the cloud service platform includes a cloud server and a remote operation visualization terminal: The cloud server and the remote operation visualization terminal are connected to each other and are used to build a federated learning framework based on a distributed computing cluster, dynamically send a lifting object recognition model to the element processing module through a cloud-edge collaboration mechanism, and are also used to periodically aggregate hoisting data from multiple construction sites and train and optimize the model parameters of the lifting object recognition model; The remote operation visualization terminal is used to visually display the hoisting data and / or the recognition result.

[0015] A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture provided by the present application, wherein the cloud server includes a lifting object recognition cloud server, and the lifting object recognition cloud server is used to train a YOLOv5 target detection model and a MobileNetV3 target classification model, and based on the MobileNetV3 target classification model and the processed data, identify the lifting object to obtain a recognition result, and is also used to send the YOLOv5 target detection model to the element processing module.

[0016] A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture provided by the present application, wherein the cloud server further includes a tower crane monitoring cloud server, and the tower crane monitoring cloud server is used to periodically aggregate hoisting data from multiple construction sites and send the hoisting data and the recognition result to the remote operation visualization terminal.

[0017] A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture provided by the present application, wherein the remote operation visualization terminal includes a main display area and a control interaction area: The main display area is used to display the hoisting data and / or the recognition result in a split screen; The control interaction area is used to receive a first input for the user to select the display content or a second input for controlling the tower crane, and in response to the first input or the second input, set the display content of the main display area or generate a control instruction to control the tower crane.

[0018] Generally speaking, compared with the prior art, the above technical solutions conceived by the present application have the following beneficial effects: (1) The element acquisition module collects real-time hoisted object state element data and hoisting operation image data during the tower crane construction. The element processing module processes the data at the edge end closer to the data source, reducing the delay of data transmission. By using the collaborative computing of the edge end and the cloud service platform, distributed training and updating of the model are realized. The cloud service platform identifies the hoisted object based on the target classification model and the processed data, and obtains the recognition result, improving the accuracy and efficiency of material statistics.

[0019] (2) The tower crane remote operation intelligent hoisted object recognition system based on the cloud-edge architecture can be integrated into the intelligent tower crane hoisting site and used in cooperation with the intelligent tower crane remote operation control center, solving the problems of large manual input and low accuracy rate in the manual hoisted object recognition process, and improving the real-time recognition rate of materials during hoisting.

[0020] (3) The intelligent digital camera processes data and updates the model locally, and sends the updated model to the cloud server, thus realizing the rapid iteration and optimization of the model, improving the recognition accuracy and efficiency. At the same time, the edge end calculates the moment balance equation in real time, and triggers an emergency stop instruction when it detects that the hoisting weight exceeds the limit by ≥5%. Combined with the cloud security situation analysis, the high-altitude hoisting accident rate is reduced.

[0021] (4) Adopting the idea of federated learning, by processing data and updating the model at the edge end, the data is retained locally, avoiding the transmission and centralized storage of sensitive data, and effectively protecting the privacy of users. At the same time, the intelligent digital camera in the element processing module processes and analyzes data locally, reducing the dependence on cloud resources and improving the response speed and efficiency of the system. Description of the Drawings

[0022] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following will briefly introduce the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0023] Figure 1 is a schematic structural diagram of a tower crane remote operation intelligent hoisted object recognition system based on the cloud-edge architecture provided by an embodiment of the present application; Figure 2 is a schematic framework diagram of a tower crane remote operation intelligent hoisted object recognition system based on the cloud-edge architecture provided by an embodiment of the present application; Figure 3It is the hardware architecture diagram of the intelligent lifting object recognition system for tower crane remote operation based on the cloud-edge-end architecture provided by the embodiments of the present application; Figure 4 It is the framework diagram of the implementation case of intelligent digital camera lifting object recognition provided by the embodiments of the present application; Figure 5 It is the framework diagram of the implementation case of spherical network camera lifting object recognition provided by the embodiments of the present application. Detailed implementation manners

[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0025] The term "and / or" in this article is an association relationship describing associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist simultaneously, and B exists alone. The symbol " / " in this article represents an "or" relationship between associated objects. For example, A / B represents A or B.

[0026] In the embodiments of the present application, words such as "exemplary" or "for example" are used to represent examples, illustrations or explanations. Any embodiment or design solution described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or having more advantages than other embodiments or design solutions. Exactly speaking, using words such as "exemplary" or "for example" aims to present relevant concepts in a specific manner.

[0027] In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" refers to two or more. For example, a plurality of processing units refers to two or more processing units, etc.; a plurality of elements refers to two or more elements, etc.

[0028] Next, in combination with Figures 1-5 An introduction is made to the intelligent lifting object recognition system for tower crane remote operation based on the cloud-edge-end architecture provided in the embodiments of the present application.

[0029] Figure 1 It is the structural schematic diagram of the intelligent lifting object recognition system for tower crane remote operation based on the cloud-edge-end architecture provided by the embodiments of the present application. As Figure 1 shown, the system 100 includes an element collection module 110, an element processing module 120, and a cloud service platform 130, where: The element collection module 110 is connected to the element processing module 120 and is used to collect real-time lifting object state element data and lifting operation image data during tower crane construction; Optionally, the data of the lifted object state elements may be data such as the lifted weight, the hook height, and the boom amplitude.

[0030] Optionally, the hoisting operation image data may be real-time construction video image data.

[0031] The element processing module 120 is connected to the cloud service platform 130 and is used to process the real-time lifted object state element data and the hoisting operation image data based on the object detection model at the edge side to obtain the processed data; Optionally, the processing may include sensing signal parsing, dynamic region of interest adaptive cropping, and providing safety warnings through calculation of the lifted weight and amplitude, etc.

[0032] The cloud service platform 130 is used to identify the lifted object based on the object classification model and the processed data to obtain the identification result, and is also used to train the object detection model and the object classification model, and send the object detection model to the element processing module.

[0033] Optionally, the cloud service platform constructs a lifted object identification federated learning framework, sends the object detection model to the element processing module, realizes lifted object identification and analysis, cross-site hoisting data aggregation analysis and safety situation visualization, assists in real-time monitoring of tower crane remote operation and provides safety guarantee.

[0034] A tower crane remote operation intelligent lifted object identification system based on a cloud-edge-end architecture provided by the present application collects the real-time lifted object state element data and the hoisting operation image data during the construction of the tower crane through the element collection module, processes the data at the edge side closer to the data source through the element processing module, reduces the data transmission delay, and utilizes the collaborative computing of the edge side and the cloud service platform to realize the distributed training and update of the model. The lifted object is identified based on the object classification model and the processed data on the cloud service platform to obtain the identification result, improving the accuracy and efficiency of material statistics.

[0035] In some embodiments, the element collection module 110 includes a hoisting operation image information collection unit and a lifted object state sensing unit: The hoisting operation image information collection unit is arranged at the bottom of the traveling beam of the hoisting trolley and is used to obtain the real-time video stream directly below the tower crane hook; The lifted object state sensing unit is used to obtain the data of the lifted object state elements.

[0036] Figure 2 It is a framework schematic diagram of the tower crane remote operation intelligent lifted object identification system provided by the embodiments of the present application, as Figure 2As shown in the figure, the element acquisition module consists of a hoisting operation image information acquisition unit and a lifted object state perception unit. Among them, the hoisting operation image information acquisition unit is configured at the bottom of the traveling beam of the hoisting trolley to obtain the real-time video stream directly below the tower crane hook and output the encoded video data to the element processing module.

[0037] Optionally, the lifted object state perception unit realizes precise measurement through a variety of sensor devices and a data processing module. Specifically, an absolute encoder installed at the flange end of the hoisting drum and a horizontal displacement encoder arranged along the boom track transmit the RS485 output signal to the data processing module. The single-wheel axle pin sensor integrated in the hoisting wire rope guiding wheel quantizes its 4-20mA analog signal through a 24-bit Σ-Δ analog-to-digital converter. The multi-sensor collaborative algorithm built into the data processing module first calculates the hook height based on the encoder pulse count at the hoisting drum and the wire rope winding circumference, and then processes the horizontal displacement signal to convert it into the hoisting trolley amplitude. At the same time, it processes the original data of the single-wheel axle pin sensor to generate dynamic load information. The specific formulas are as follows: (1) (2) In the formula, is the hoisting height or lowering height, is the rotation angle of the hoisting drum, that is, the measured value obtained by the encoder at the hoisting drum end through formula (1), is the diameter of the wire rope.

[0038] In some embodiments, the hoisting operation image information acquisition unit includes a first intelligent digital camera. The first intelligent digital camera is deployed at the bottom of the traveling beam of the hoisting trolley and is used to detect and real-time locate the hook and the lifted object based on the deep learning algorithm.

[0039] Optionally, the first intelligent digital camera is deployed at the bottom of the traveling beam of the hoisting trolley, equipped with a 64-bit hardware processing system based on the ARMv8-A architecture and running the Linux operating system, and realizes the precise detection and real-time location of the hook and the lifted object through the embedded deep learning algorithm.

[0040] Optionally, the deep learning algorithm uses a target detection model to continuously track the movement trajectory of the hook and completes the precise positioning of the lifted object in combination with the spatial position relationship between the hook and the lifted object.

[0041] Specifically, based on the relative position characteristics of the hook and the lifted object, the first intelligent digital camera dynamically crops the key area including the hook and the associated lifted object to generate a local image. The height of the cropping area is determined by the lower edge coordinate of the hook detection frame, and horizontally, it extends ±1.5 times the width of the lifted object based on the center of the hook, outputting structured metadata containing target position and size information, eliminating background noise interference and reducing the amount of image processing data, reducing the overall processing delay of the system by reducing the computational complexity of image feature extraction, improving the real-time processing efficiency of the system, and providing technical support for the rapid response in the remote operation scenario of tower cranes.

[0042] In some embodiments, the image information acquisition unit for hoisting operations further includes a spherical network camera, which is used to obtain the tower crane operation image data in real time based on an adaptive optimization algorithm for complex illumination scenarios.

[0043] Optionally, the spherical network camera obtains the tower crane operation image data in real time and obtains the tower crane operation image data in real time based on an adaptive optimization algorithm for complex illumination scenarios.

[0044] Specifically, in the low-illumination area caused by the occlusion of the hoisting path, the brightness and contrast of the image are dynamically adjusted through a backlight compensation algorithm based on histogram equalization to improve the signal-to-noise ratio of the dark part details and avoid image blurring under backlight; for the strong reflection area, a strong light suppression algorithm guided by region segmentation is adopted to dynamically compress the high-brightness area to an 8-12bit effective gray scale range to ensure the texture retention rate of the high-contrast edges. At the same time, a 3D noise reduction module based on joint spatio-temporal domain analysis is integrated. Through the collaborative processing of motion vector estimation and spatial non-local mean filtering, the clarity of the image is improved, supporting the high-precision target detection and recognition of the lifted object recognition system under complex illumination conditions, meeting the image quality requirements of the remote operation scenario, and ensuring the high-precision recognition requirements in the remote operation scenario.

[0045] In some embodiments, the element acquisition module collaboratively transmits the data of the lifted object state elements and the hoisting operation image data through a multi-process transmission method and a 5G communication network.

[0046] The element acquisition module collaboratively realizes the efficient transmission of the data of the lifted object state elements and the hoisting operation image data through a multi-process transmission method and a 5G communication network.

[0047] Specifically, the image acquisition thread reads the video stream of the spherical network camera in real time based on the Real-Time Streaming Protocol (RTSP) and uses a circular queue with a depth of ≤ 3 frames for dynamic caching. When queue redundancy is detected, historical frames are actively discarded. At the same time, the status element acquisition thread polls the hook height, load weight, and boom amplitude data through the Modbus RTU protocol at a cycle of 10 ms and encapsulates them into Controller Area Network (CAN) messages. The above raw data is transmitted to the element processing module in parallel through Gigabit Ethernet and the CAN bus. This module performs adaptive region of interest cropping to reduce the video stream bandwidth requirement and conducts cyclic redundancy check (CRC) and sliding window filtering on the sensor data. The processed data is transmitted to the 5G user equipment through 5G New Radio, accessed to the 5G core network through the base station backhaul link, and finally routed to the cloud service platform through ultra-reliable low-latency communication. Among them, 256QAM modulation is used between the 5G user equipment and the base station to ensure that the uplink rate is ≥ 500 Mbps. By combining multi-threaded queue management and the 5G communication network, the data transmission delay is reduced, and the real-time and accuracy of the image information and the lifting object status element information during the lifting object recognition process are ensured.

[0048] Figure 3 is the hardware architecture diagram of the tower crane remote operation intelligent lifting object recognition system provided by the embodiment of the present application. As Figure 3 shown, the information collected by the element acquisition module is transmitted to the local area network switch through an Ethernet cable. The switch forwards the data to the 5G switch, and the data is connected to the 5G user equipment through 5G New Radio. The 5G user equipment wirelessly transmits the data to the 5G base station. The 5G base station transmits the data to the 5G core network through a high-speed optical fiber. The core network is responsible for data routing, forwarding, and related management and control tasks. Finally, the data reaches the cloud server in the cloud service platform.

[0049] The element acquisition module realizes data feedback through a multi-process transmission architecture: Status element data including hook height, lifting weight, and boom amplitude parameters are collected through the Modbus RTU protocol, quantized by a 24-bit Σ-Δ analog-to-digital converter (ADC), and encapsulated into CAN 2.0B standard messages; Video data: Intelligent digital cameras and spherical network cameras output H.264 encoded video streams based on the RTSP protocol and are dynamically cached through a circular queue with a depth of ≤3 frames; The above heterogeneous data is transmitted in parallel to the element processing module through Gigabit Ethernet (video stream) and CAN bus (sensing signal), and preprocessing such as adaptive region of interest cropping and sliding window filtering (window length 500ms) is performed; The processed structured data is transmitted through the 5G New Radio uplink, accessed to the base station through a 5G user equipment, and 256QAM modulation technology is used to ensure an uplink rate ≥500Mbps; The base station is connected to the 5G core network through a low-latency fiber optic backhaul network; The core network routes the data to the cloud service platform through an ultra-reliable low-latency communication channel to meet the real-time control requirements in the tower crane operation scenario.

[0050] In some embodiments, the element processing module includes a second intelligent digital camera and an edge computing unit: The second intelligent digital camera is used to collect hoisting operation image data and also to preprocess the hoisting operation image data; The edge computing unit is deployed in the control cabinet at the root of the tower crane boom and is used to process the real-time hoisted object status element data and hoisting operation image data based on the target detection model to obtain the processed data.

[0051] The element processing module integrates and mounts the second intelligent digital camera and the edge computing unit. The edge computing unit is deployed in the control cabinet at the root of the tower crane boom and synchronously receives the hoisting operation image data (including video stream) and hoisted object status element data (including hook height, lifting weight, amplitude) collected by the first intelligent digital camera and the second intelligent digital camera, performs adaptive region of interest cropping on the video stream, and at the same time uses sliding window normalization and extended Kalman filtering to clean the sensor signals and eliminate instantaneous noise.

[0052] The edge computing unit also loads the lightweight target detection model issued by the cloud service platform, realizes data processing locally, uploads the processed data to the cloud service platform, reduces the original video transmission bandwidth requirement from 20Mbps to 4Mbps, and effectively solves the wireless network bandwidth bottleneck problem in the high-altitude operation of tower cranes.

[0053] The load signal is collected in real time through the unicycle axle pin sensor. Combining the boom amplitude and the hook height, the moment balance equation is dynamically solved based on the tower crane structure parameters. When the detected actual load overload rate ≥ 5%, a localized safety response is triggered. The element processing module processes the load overload judgment locally, compresses the safety response delay, and significantly reduces the risk of lifting accidents during the high-altitude operation of the tower crane.

[0054] In some embodiments, the cloud service platform includes a cloud server and a remote operation visualization terminal: The cloud server and the remote operation visualization terminal are connected to each other, and are used to build a federated learning framework based on a distributed computing cluster, dynamically send the object recognition model to the element processing module through the cloud-edge collaboration mechanism, and are also used to periodically aggregate the lifting data from multiple construction sites to train and optimize the model parameters of the object recognition model; The remote operation visualization terminal is used to visually display the lifting data and / or the recognition results.

[0055] The cloud service platform operates in cooperation with the cloud server and the remote operation visualization terminal. The cloud server builds a federated learning framework based on a distributed computing cluster, dynamically sends a lightweight object detection model to the element processing module through the cloud-edge collaboration mechanism, and periodically aggregates the lifting data from multiple construction sites. The model parameters are jointly trained and optimized using cross-site data to achieve the adaptive ability of the model to complex working conditions.

[0056] Optionally, the cloud server integrates safety situation analysis at the same time, and fuses multi-source data (including load, amplitude, height and environmental parameters) in real time to generate a safety assessment index, which is pushed to the remote operation visualization terminal through the Open Platform Communications Unified Architecture (OPC UA) protocol.

[0057] The remote operation visualization terminal is used to visually display the lifting data and / or the recognition results.

[0058] In some embodiments, the cloud server includes an object recognition cloud server, which is used to train the YOLOv5 object detection model and the MobileNetV3 object classification model, identify the object based on the MobileNetV3 object classification model and the processed data to obtain the recognition result, and is also used to send the YOLOv5 object detection model to the element processing module.

[0059] The lifting object recognition cloud server processes the tower crane operation video stream uploaded by the element processing module in real time through a collaborative architecture that deploys an improved YOLOv5 object detection model and a lightweight MobileNetV3 object classification model. The YOLOv5 model detects the position of the hook, and the MobileNetV3 model identifies the type of the lifted object based on the spatial position relationship between the hook and the lifted object. Every 24 hours, an optimized model is generated through knowledge distillation technology and sent to the second intelligent digital camera of the element processing module.

[0060] Figure 4 It is the framework diagram of the implementation case of lifting object recognition for the intelligent digital camera provided by the embodiment of the present application. As Figure 4 shown, the ARMv8-A embedded system built inside the second intelligent digital camera uses the YOLOv5 model to achieve hook tracking, accurately locates the hook and the lifted object based on the position relationship between the hook and the lifted object, and automatically crops the regional image containing the hook and the lifted object. For the image preprocessed by the intelligent camera, the MobileNetV3 model is deployed in the cloud service platform to identify the type of the lifted object. At the same time, the lifting object recognition cloud server trains the YOLOv5 model and sends the trained model to the second intelligent digital camera of the element processing module.

[0061] Figure 5 It is the framework diagram of the implementation case of lifting object recognition for the spherical network camera provided by the embodiment of the present application. As Figure 5 shown, the spherical network camera performs backlight compensation to improve the imaging effect in low-light areas, applies strong light suppression to reduce interference in high-brightness areas, and removes image noise through 3D noise reduction technology. The lifting object recognition cloud server deploys the YOLOv5 model to process the image data transmitted by the spherical network camera to achieve lifting object recognition and classification.

[0062] In some embodiments, the cloud server further includes a tower crane monitoring cloud server, which is used to periodically aggregate the lifting data from multiple construction sites and send the lifting data and recognition results to the remote operation visualization terminal.

[0063] The tower crane monitoring cloud server receives the sensor data (lifting weight, amplitude, height) preprocessed at the edge end and the lifting object recognition result in real time through the OPC UA protocol, and displays the real-time pose and safety assessment parameters of the lifted object through the human-machine interaction interface; after loading the optimized model, the intelligent digital camera performs real-time detection locally and only uploads the structured metadata, forming a technical link of "edge real-time perception-cloud global optimization-monitoring closed-loop decision-making".

[0064] The tower crane monitoring cloud server aggregates multi-angle video streams and sensor data of tower cranes at multiple construction sites, initiates real-time data requests to the lifting object recognition cloud server based on the HTTP / 2 protocol, and after obtaining the lifting object recognition results, synchronously pushes the video streams and structured data to the remote operation visualization terminal; the remote operation visualization terminal adopts a microservices architecture to achieve cross-platform remote access, supports real-time interaction between mobile devices (iOS / Android) and personal computer (PC) browsers through two-way communication of RESTful API and WebSocket. Operators can dynamically retrieve multi-perspective videos, lifting object recognition results, and cross-site lifting statistical charts of any tower crane, and generate remote operation control instructions to achieve cloud-based collaborative monitoring and precise remote control of tower crane groups across regions.

[0065] In some embodiments, the remote operation visualization terminal includes a main display area and a control interaction area: The main display area is used for split-screen display of lifting data and / or recognition results; The control interaction area is used to receive a first input for the user to select the displayed content or a second input for controlling the tower crane, and in response to the first input or the second input, set the displayed content of the main display area or generate a control instruction to control the tower crane.

[0066] The remote operation visualization terminal includes: a main display area that split-screens and displays multi-perspective real-time video streams and real-time safety heat maps, and windows and displays cross-site lifting statistical charts (failure rate, efficiency curve), current lifting object recognition results, and the iterative progress of the federated learning model; a control interaction area: provides a remote operation console with built-in one-key emergency stop, lifting path planning, and multi-site situation comparison views; assisting operators to quickly obtain the state of the lifted object, construction environment parameters, and warning information, so as to achieve precise remote operation of the tower crane.

[0067] In an embodiment of the present application, a method for intelligent recognition of lifted objects using the tower crane remote operation intelligent lifted object recognition system provided by the present application based on a cloud-edge-terminal architecture includes the following steps: 1) Initialize the spherical network camera and the intelligent digital camera, activate the storage array, and receive and store multiple video streams in real time; 2) Start the spherical network camera, collect real-time panoramic video images at the lifting trolley of the tower crane, perform strong light suppression to eliminate the reflection interference of metal components, and enable a 3D noise reduction module for spatio-temporal domain joint filtering in low-light rain and fog environments to effectively reduce the image noise level; 3) Start the intelligent digital camera, based on the relative position characteristics of the hook above the lifted object, use a deep learning object detection algorithm to real-time locate the specific positions of the hook and the lifted object, and by analyzing the relative geometric relationship between the hook and the lifted object, dynamically crop out the key image area containing the hook and the lifted object; 4) After receiving the real-time signals from the encoder and the unicycle axle pin sensor, the element processing module first performs sliding window normalization and extended Kalman filtering to eliminate the deformation error of the tower body, and real-time calculates the hook height, the lifting weight, and the boom amplitude.

[0068] 5) The element processing module dynamically calculates the safety threshold based on the moment balance equation. When the wind speed ≥ level 6, it fuses the wind speed data to dynamically adjust the parameters of the moment equation. When it detects that the lifting weight exceeds the threshold by 5%, it triggers a local safety response, stops urgently and prevents subsequent operations; if the lifting weight is within the safe range, it proceeds to step 6); 6) The preprocessed data is transmitted to the lifted object recognition cloud server through a 5G Ultra Reliable Low Latency Communication (URLLC) channel. The YOLOv5 model is used to detect the bounding box of the lifted object in the video stream of the spherical network camera, and the positioning and classification of the lifted object are completed; the MobileNetV3 model is used to classify the type of the lifted object in the cropped image of the intelligent camera; 7) The lifted object recognition cloud server aggregates the incremental data of multiple construction sites every 24 hours to train the global model, optimizes the YOLOv5 detection model through the federated learning framework, and distributes the optimized model to the intelligent digital camera of the element processing module to improve the adaptability and accuracy of the model to complex working environments; 8) Transmit the recognition results and safety parameters to the tower crane monitoring cloud server, and display the type of the lifted object, the weight parameters, and the safety assessment results in real time on the remote operation visualization terminal to assist the operator in accurately remotely operating the tower crane. Remote users can access the cloud platform through a PC or a mobile device and obtain the real-time video stream and safety status data based on the RESTful API to ensure the convenience and safety of remote operations.

[0069] 9) When an unrecognized type of lifted object is detected, it automatically captures the key frame and marks the spatio-temporal information, triggering incremental training in the cloud, and completing the edge deployment of the new model within 12 hours.

[0070] 10) Establish a full-cycle digital archive for the hoisting process, record the recognition results such as the concrete hopper, the start and end time stamps of hoisting, and the sequence of spatial trajectory coordinates, and construct a structured construction log.

[0071] It can be understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0072] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer may be a general-purpose computer, a dedicated computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center in a wired manner (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wirelessly (such as infrared, wireless, microwave, etc.). The computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more integrated available media. The available medium may be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)), etc.

[0073] It can be understood that the various digital numbers involved in the embodiments of the present application are only for the convenience of description and are not used to limit the scope of the embodiments of the present application.

[0074] Those skilled in the art can easily understand that the above are only the preferred embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, and improvements made within the spirit and principle of the present application should be included in the protection scope of the present application.

Claims

1. A tower crane remote operation intelligent lifting object recognition system based on a cloud-edge-terminal architecture, characterized in that, It includes an element acquisition module, an element processing module, and a cloud service platform: The element acquisition module is connected to the element processing module and is used to acquire the data of the hoisted object state elements and the hoisting operation image data during the tower crane construction; The element processing module is connected to the cloud service platform and is used to process the real-time hoisted object state element data and the hoisting operation image data based on the object detection model at the edge end position to obtain the processed data; The cloud service platform is used to identify the hoisted object based on the object classification model and the processed data to obtain the identification result, and is also used to train the object detection model and the object classification model, and send the object detection model to the element processing module.

2. The tower crane remote operation intelligent load identification system based on the cloud-edge-end architecture according to claim 1, characterized in that, The element acquisition module includes a hoisting operation image information acquisition unit and a hoisted object state perception unit: The hoisting operation image information acquisition unit is used to obtain the real-time video stream directly below the tower crane hook; The hoisted object state perception unit is used to obtain the data of the hoisted object state elements.

3. The tower crane remote operation intelligent load identification system based on the cloud-edge-terminal architecture according to claim 2, characterized in that, The hoisting operation image information acquisition unit includes a first intelligent digital camera, which is deployed at the bottom of the traveling beam of the tower crane's hoisting trolley and is used to detect and real-time locate the hook and the hoisted object based on the deep learning algorithm.

4. The tower crane remote operation intelligent load recognition system based on the cloud-edge-end architecture according to claim 3, characterized in that, The hoisting operation image information acquisition unit also includes a spherical network camera, which is used to obtain the tower crane operation image data in real time based on the adaptive optimization algorithm for complex illumination scenarios.

5. The tower crane remote operation intelligent load recognition system based on the cloud-edge-end architecture according to any one of claims 1-4, characterized in that, The element acquisition module transmits the data of the hoisted object state elements and the hoisting operation image data through the multi-process transmission method and the 5G communication network in coordination.

6. The tower crane remote operation intelligent load identification system based on the cloud-edge-end architecture according to claim 1, characterized in that The element processing module includes a second intelligent digital camera and an edge computing unit: The second intelligent digital camera is connected to the edge computing unit and is used to acquire the hoisting operation image data and also to preprocess the hoisting operation image data; The edge computing unit is deployed in the control cabinet at the root of the tower crane boom and is used to process the real-time hoisted object state element data and the hoisting operation image data based on the object detection model to obtain the processed data.

7. The tower crane remote operation intelligent load recognition system based on the cloud-edge-end architecture according to claim 1, characterized in that, The cloud service platform includes a cloud server and a remote operation visualization terminal: The cloud server is connected to the remote operation visualization terminal and is used to build a federated learning framework based on the distributed computing cluster, dynamically send the object recognition model to the element processing module through the cloud-edge collaboration mechanism, and is also used to periodically aggregate the hoisting data from multiple construction sites to train and optimize the model parameters of the hoisted object recognition model; The remote operation visualization terminal is used to visually display the hoisting data and / or the recognition result.

8. The tower crane remote operation intelligent lifting object recognition system based on the cloud-edge-terminal architecture according to claim 7, wherein, The cloud server includes a hoisted object recognition cloud server, which is used to train the YOLOv5 object detection model and the MobileNetV3 object classification model, identify the hoisted object based on the MobileNetV3 object classification model and the processed data to obtain the recognition result, and is also used to send the YOLOv5 object detection model to the element processing module.

9. The tower crane remote operation intelligent load identification system based on the cloud-edge-end architecture according to claim 8, wherein, The cloud server further includes a tower crane monitoring cloud server, which is used to periodically aggregate the lifting data from multiple construction sites and send the lifting data and the recognition result to the remote operation visualization terminal.

10. The tower crane remote operation intelligent load recognition system based on the cloud-edge-terminal architecture according to claim 7, characterized in that, The remote operation visualization terminal includes a main display area and a control interaction area: The main display area is connected to the control interaction area and is used to display the lifting data and / or the recognition result in a split screen; The control interaction area is used to receive a first input for the user to select the display content or a second input for controlling the tower crane, and in response to the first input or the second input, set the display content of the main display area or generate a control instruction to control the tower crane.

Images