Edge calculation real-time early warning system and method for illegal behaviors of crane operator

Abstract

The invention discloses a crane operator behavior monitoring system and method based on AI vision, and belongs to the technical field of industrial safety monitoring. In order to solve the problem that the crane accident proportion caused by crane operators is the highest nowadays, the system collects operator video streams in real time through an AI camera, an edge processor executes a multi-stage behavior recognition model, and dangerous behaviors such as dozing, calling, smoking and frequent departure are accurately recognized; when an abnormity is detected, triggering a local sound-light alarm and synchronously uploading the sound-light alarm to a remote management platform; the platform integrates data of multiple devices through a wireless transmission network, and realizes alarm record tracing, behavior trend analysis and visual report generation. The method solves the hysteresis problem of traditional manual supervision, is high in recognition accuracy and short in response time, and remarkably reduces the operation safety accident rate of the crane.

Description

Technical Field

[0001] This invention relates to an edge computing real-time early warning system for crane operator violations, which belongs to the technical field of real-time early warning. Background Technology

[0002] Nowadays, there are many types of cranes, and they are used in large quantities and on a wide range of applications. Therefore, it is necessary to develop a method for monitoring dangerous behaviors of operators.

[0003] Existing monitoring systems rely on ordinary video cameras and manual supervision. However, ordinary cameras lack specialized behavioral analysis capabilities for crane scenarios, making timely intervention impossible within the window of time between behavioral occurrence and accident formation (typically 3-5 seconds). Furthermore, they exhibit a high false alarm rate in environments with strong backlighting, such as the operator's cab. Manual supervision faces challenges in identifying and responding to subtle operator movements, often resulting in delays. Additionally, the current practice of storing alarm information from individual cranes in isolation hinders cross-device comparative analysis, and historical behavioral data remains untapped for accident prevention – all issues that urgently require solutions. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides an edge computing real-time early warning system for crane operator violations, ensuring the safety of crane operations. Furthermore, it solves the problem of real-time identification of dangerous behaviors in enclosed spaces; compensates for the lack of intelligence in existing monitoring systems; and allows for cross-device comparative analysis of alarm information from multiple cranes, using historical behavior data for accident prevention.

[0005] The technical solution adopted in this invention is: an edge computing real-time early warning system for crane operator violations, which includes three parts: a local AI video monitoring system, a wireless transmission network, and a remote video fusion platform; The local AI video surveillance system includes front-end acquisition equipment, data processing equipment, response equipment, and transmission equipment, which are responsible for the acquisition and processing of on-site data; the remote video fusion platform receives on-site data through a wireless transmission network to enable remote management and viewing. The front-end acquisition device is an AI camera with AI function, used for intelligent identification and tracking of targets, target detection and task classification. The data processing device includes an edge processor, a switch, and a memory; the edge processor interfaces with the main display terminal and the AI ​​camera, and is responsible for AI analysis of the on-site video; the switch is the on-site network hub, connecting the edge processor, the memory, and the AI ​​camera to achieve network interconnection between devices; The response device includes an audible and visual alarm and an in-vehicle display. The audible and visual alarm is used to receive signals from the edge processor and triggers an audible and visual warning when the AI ​​identifies an anomaly. The in-vehicle display is used to receive signals from the edge processor and display the results of data processing and matching from the camera. The transmission device is a 4G wireless router, which transmits on-site data to the remote platform layer via a wireless network. The remote video fusion platform includes a cloud server, a video fusion platform, and terminal devices; the cloud server is used to receive on-site video / data transmitted by the 4G router, serving as the computing power and storage support for the remote platform; the video fusion platform, based on the data from the cloud server, enables real-time projection of on-site monitoring functions to the office or control center.

[0006] Furthermore, the AI ​​camera features 2 megapixels, 120dB wide dynamic range, and 1.5 TOPS computing power.

[0007] Furthermore, the AI ​​camera supports backlight compensation and strong light suppression.

[0008] Furthermore, the AI ​​camera supports the detection of 16 types of targets and classifies one of them.

[0009] A method for operating an edge computing real-time early warning system for crane operator violations includes the following steps: Step 1, Image Acquisition: The camera acquires images of the driver's behavior at a set frame rate; Step 2. Image preprocessing: Preprocess the acquired image data; Step 3. Feature Extraction: Extract key features from the image; Step 4. Behavior recognition: Identify the driver's behavior based on the extracted features.

[0010] Furthermore, in step 2, the preprocessing of the image data includes grayscale conversion, binarization, and denoising operations.

[0011] Furthermore, in step 3, the key features are the face, eyes, and hands.

[0012] Furthermore, in step 4, the high-risk behaviors defined for behavior recognition include dozing off, making phone calls, smoking, and obstructing the camera.

[0013] Compared with existing technologies, this invention has the following advantages: Addressing the issue that crane operators currently account for the largest proportion of crane accidents, the system uses a customized AI camera (2 megapixels, 1.5 TOPS computing power) to collect real-time video streams from operators. An edge processor executes a multi-level behavior recognition model (supporting 16 target detections + 64 classifications) to accurately identify dangerous behaviors such as dozing off, making phone calls, smoking, and frequent absences from duty. When an anomaly is detected, a local audible and visual alarm is triggered and simultaneously uploaded to a remote management platform. The platform integrates data from multiple devices via a wireless transmission network, enabling alarm record traceability (stored for 3-5 years), behavior trend analysis, and visualization report generation. This invention solves the problem of the lag in traditional manual supervision, achieving an accuracy rate of ≥90% and a response time of ≤2 seconds, significantly reducing the accident rate in crane operations. Attached Figure Description

[0014] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of the workflow of an edge computing real-time early warning system for crane operator violations.

[0016] Figure 2 This is a structural diagram of an edge computing real-time early warning system for crane operator violations. Detailed Implementation

[0017] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0018] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit the present invention or its application or use. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0020] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps described in these embodiments do not limit the scope of the invention. It should also be understood that, for ease of description, the dimensions of the various parts shown in the drawings are not drawn to actual scale. Techniques, methods, and devices known to those skilled in the art may not be discussed in detail, but where appropriate, such techniques, methods, and devices should be considered part of the specification. In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0021] In the description of this invention, it should be understood that the orientation or positional relationship indicated by directional terms such as "front, back, up, down, left, right", "horizontal, vertical, horizontal" and "top, bottom" is generally based on the orientation or positional relationship shown in the accompanying drawings, and is only for the convenience of describing this invention and simplifying the description. Unless otherwise stated, these directional terms do not indicate or imply that the device or element referred to must have a specific orientation or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation on the scope of protection of this invention. The directional terms "inner" and "outer" refer to the inner and outer contours relative to the outline of each component itself.

[0022] For ease of description, spatial relative terms such as "above," "over," "on the upper surface of," "above," etc., are used herein to describe the spatial positional relationship of a device or feature as shown in the figures to other devices or features. It should be understood that spatial relative terms are intended to encompass different orientations in use or operation besides the orientation of the device as described in the figures. For example, if the device in the figures is inverted, a device described as "above" or "above" other devices or structures would subsequently be positioned as "below" or "under" other devices or structures. Thus, the exemplary term "above" can include both "above" and "below." The device may also be positioned in other different ways (rotated 90 degrees or in other orientations), and the spatial relative descriptions used herein will be interpreted accordingly.

[0023] Furthermore, it should be noted that the use of terms such as "first" and "second" to define components is merely for the purpose of distinguishing the corresponding components. Unless otherwise stated, the above terms have no special meaning and therefore should not be construed as limiting the scope of protection of this invention.

[0024] This system consists of three parts based on location: a local AI video surveillance system, a wireless transmission network, and a remote video fusion platform. (1) The local AI video surveillance system includes components such as AI cameras, edge processors, memory, and displays. The local configuration can stand alone as a system, capable of acquiring, displaying, analyzing, alarming, and storing operator behavior video image data.

[0025] Functions: Provides real-time monitoring and recording of the control panel and operator work status. It can detect dangerous behaviors such as: making phone calls, inattention, smoking, frequent absences from duty, and obstructing cameras. Upon detecting any of these behaviors, it immediately provides an audio alert to the operator. The local AI can be trained and upgraded.

[0026] Implementation Method: Utilizing the VS-2XA2146F-IS AI camera. It supports the HEOP (Head-Up Platform), allowing for the installation, uninstallation, and upgrading of intelligent applications through an online application store and management platform, and the import of third-party intelligent applications. It supports 1.5 TOPS computing power, 60MB system memory, 400MB intelligent memory, and 2GB eMMC storage resource sharing. It supports the AIOP (AI Platform) with deep learning frameworks such as Caffe, PyTorch, and TensorFlow, enabling the deployment and execution of AI models, and the generation and uploading of detection results. It includes BASE, BSC, and HIKFLOW libraries. Features include backlight compensation, strong light suppression, 3D digital noise reduction, and 120dB wide dynamic range to adapt to different environments. It supports the detection of 16 types of targets and the classification of one of them, with 64 classification categories supported. It also supports image capture and face capture. It supports the storage of four model packages, each containing one detection model and two classification models. The local AI video surveillance system is equipped with an edge processor to clean and match the data provided by the camera.

[0027] (2) Wireless transmission network: The channel for data transmission between the local system and the office or control center. The network is divided into two types: local area network or public network, which can be selected according to the needs.

[0028] (3) Remote video fusion platform: mainly realizes remote management of local AI monitoring system. After configuring the management host (server) and installing the platform software, the local monitoring function can be projected to the office or control center in real time. The large screen can be configured to realize screen projection display, which facilitates user cluster management and reduces workload.

[0029] Functions: The system can synchronously transmit and display monitoring images from the control room of each crane via the network, allowing operators to observe their status on the office screen; it records alarm information pushed by the crane video system connected to this platform, and can query, filter, and export stored data for secondary use; it has alarm data analysis capabilities, which can analyze operational behavior by device, time, and alarm type, and provide historical alarm trend curve analysis views.

[0030] Algorithm flow: 1. Image Acquisition: The camera captures images of the driver's behavior at a set frame rate.

[0031] 2. Image preprocessing: Preprocess the acquired image data, including grayscale conversion, binarization, and noise reduction.

[0032] 3. Feature extraction: Extract key features from the image, such as faces, eyes, and hands.

[0033] 4. Behavior recognition: Based on the extracted features, identify the driver's behavior, such as driving while fatigued, dozing off, or using a mobile phone.

[0034] The detailed steps for using the above technical solution are as follows: Step 1: System Initialization and Parameter Configuration Start the field device layer and the remote platform layer, complete the network interconnection of field devices through the switch, and establish a wireless network connection between the field and the remote device through the 4G router.

[0035] The remote platform sends configuration parameters to the AI ​​camera: set the image acquisition frame rate, which is recommended to be 15-30 frames / second to balance real-time performance and bandwidth; the monitoring area ROI should focus on the operator area inside the crane cab; and the image resolution should match the 2-megapixel hardware specifications. The edge processor loads a pre-set AI model and imports a rule library for violations, clearly defining the criteria for judging high-risk behaviors such as dozing off, making phone calls, smoking, and obstructing cameras.

[0036] Test equipment connectivity: The AI ​​camera captures images and transmits them to the vehicle display and remote platform in real time. The sound and light alarm triggers normally. The memory begins to initialize the storage directory: distinguishing between raw video, pre-processed video, and abnormal event segments.

[0037] Step 2: Image acquisition, real-time image capture stage: The AI ​​camera continuously captures real-time footage of the crane's cab at a preset frame rate of 15-30 frames per second, focusing on the operator's upper body area: face, hands, and upper body movements. During the acquisition process, hardware-level image optimization is automatically enabled: 120dB wide dynamic range balances backlight / strong light environment inside the operating cabin, such as strong light outside the window and reflection of equipment indicator lights; backlight compensation enhances the brightness of shadow areas (such as the operator's side face); strong light suppression avoids overexposure of the image; 3D noise reduction processes noise in low light environment to ensure that the image is clear and distinguishable. The acquired raw image data is synchronized in real time to two terminals: ① a local edge processor for real-time analysis; and ② a storage device for backing up the raw data and supporting subsequent backtracking.

[0038] Step 3: Image preprocessing and data optimization stage; After receiving the raw image stream from the AI ​​camera, the edge processor performs preprocessing in the order of grayscale conversion, binarization, and noise reduction: Grayscale conversion: Convert the RGB color image to a single-channel grayscale image to reduce the amount of data while retaining key information such as operator contours and feature points; Binarization: Set an adaptive threshold to convert the grayscale image into a black and white binary image, highlighting the contrast between the target operator and the background; Denoising: Use a median filtering algorithm to remove isolated noise points in the binary image to avoid noise interfering with subsequent feature extraction.

[0039] Step 4: Key feature extraction, target information extraction stage; The edge processor calls a pre-defined AI object detection model to extract features from the pre-processed image, focusing on three key features: facial features: extracting key facial points such as eyes, mouth, eyebrows, and cheek contours, and recording the degree of eye opening and closing and head posture; eye features: focusing on identifying the eyelid closure status, duration of closure, and frequency of closure; eyeball movement trajectory, and whether the eye is focused on the control panel; and hand features: extracting hand contours, finger posture, hand position, whether the hand is close to the head or ear, and whether the hand is holding items such as cigarettes or mobile phones.

[0040] During feature extraction, the AI ​​model automatically filters out irrelevant background features, retains only feature data related to operator behavior, and converts the feature data into an AI-recognizable vector format; the extracted feature vectors are transmitted to the AI ​​analysis module of the edge processor in real time, and are also backed up to the storage.

[0041] Step 5: Behavior matching and identification, violation determination stage; After receiving the feature vector, the edge processor's AI behavior recognition model performs real-time matching with a pre-set high-risk behavior rule library to determine whether any violations exist.

[0042] During the recognition process, the AI ​​model supports real-time correction: if the light in the control cabin changes suddenly, the feature matching threshold is automatically adjusted to avoid misjudgment; the recognition results of each frame are output in real time and stored in association with the corresponding preprocessed image and feature vector.

[0043] Step 6: Anomaly Response and Early Warning, Violation Handling Stage; If the recognition result is normal: the edge processor continues to receive the next batch of image data, repeats steps 2-5, and the vehicle display shows the operator's normal screen in real time without any additional response; If the identification result is invalid: The edge processor immediately triggers a level three response: Local alert: Sends a signal to the audible and visual alarm, which is activated to remind the operator to stop the violation; Local display: A red violation warning box pops up on the vehicle display, and the violation scene is frozen for 3 seconds before the real-time scene is restored; Remote reporting: The violation type, violation time, violation scene and characteristic data are packaged and transmitted to the cloud server of the remote platform layer through the 4G wireless router, and pushed to the video fusion platform simultaneously. Step 7: Remote platform processing and management, remote control phase; After receiving the violation data reported on-site, the cloud server classifies and stores it: archives it by crane number, violation type, and timestamp, and provides computing power to support the secondary analysis of the video fusion platform. The video fusion platform visualizes violation data: it simultaneously displays violation footage, violation type, violation time, and crane number on remote large screens, client devices, and mobile devices, allowing managers to view it in real time. Managers can operate remotely via terminals to: ① view the scene of the violation in real time; ② review the original video before and after the violation; ③ issue instructions to the site; ④ update the rule library for violation behavior; the video fusion platform automatically counts violation data (generating reports by violation type, crane number, and time period) for managers to analyze and optimize later.

[0044] Step 8: Continuous system operation and iteration, cyclical optimization phase; After the system completes a violation response, it continuously monitors operator behavior without manual intervention; it builds a multi-source alarm data fusion analysis engine: generating risk trend reports based on three dimensions: device / time / behavior type; and predicting the probability of accident risk for individual operators based on historical data.

[0045] The system features an innovative hardware architecture. The dedicated AI monitoring unit in the driver's cab comprises a customized AI camera (2MP / 120dB wide dynamic range / 1.5TOPS computing power) and an edge processor, forming the local processing core. This achieves a closed-loop video acquisition, analysis, and alarm process within a physically enclosed space in less than 2 seconds, resolving the response latency issue. Simultaneously, a multi-level behavior recognition model for crane scenarios has been developed: supporting simultaneous operation of 16 target detection tasks and 64 classification tasks; specifically identifying high-risk behaviors such as dozing off, making phone calls, lack of concentration, smoking, frequent absence from duty, and obstructing the camera.

[0046] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An edge computing real-time early warning system for crane operator violations, characterized in that: The system consists of three parts: a local AI video surveillance system, a wireless transmission network, and a remote video fusion platform. The local AI video surveillance system includes front-end acquisition equipment, data processing equipment, response equipment, and transmission equipment, which are responsible for the acquisition and processing of on-site data; the remote video fusion platform receives on-site data through a wireless transmission network to enable remote management and viewing. The front-end acquisition device is an AI camera with AI function, used for intelligent identification and tracking of targets, target detection and task classification. The data processing device includes an edge processor, a switch, and a memory; the edge processor interfaces with the main display terminal and the AI ​​camera, and is responsible for AI analysis of the on-site video; the switch is the on-site network hub, connecting the edge processor, the memory, and the AI ​​camera to achieve network interconnection between devices; The response device includes an audible and visual alarm and an in-vehicle display. The audible and visual alarm is used to receive signals from the edge processor and triggers an audible and visual warning when the AI ​​identifies an anomaly. The in-vehicle display is used to receive signals from the edge processor and display the results of data processing and matching from the camera. The transmission device is a 4G wireless router, which transmits on-site data to the remote platform layer via a wireless network. The remote video fusion platform includes a cloud server, a video fusion platform, and terminal devices; The cloud server is used to receive live video / data transmitted from the 4G router, serving as the computing power and storage support for the remote platform; The video fusion platform uses data from cloud servers to enable real-time projection of on-site monitoring functions to offices or control centers.

2. The edge computing real-time early warning system for crane operator violations according to claim 1, characterized in that: The AI ​​camera features 2 megapixels, 120dB wide dynamic range, and 1.5 TOPS computing power.

3. The edge computing real-time early warning system for crane operator violations according to claim 2, characterized in that: The AI ​​camera supports backlight compensation and strong light suppression.

4. The edge computing real-time early warning system for crane operator violations according to claim 3, characterized in that: The AI ​​camera supports the detection of 16 types of targets and classifies one of them.

5. The working method of the edge computing real-time early warning system for crane operator violations according to claim 4, characterized in that, Includes the following steps: Step 1, Image Acquisition: The camera acquires images of the driver's behavior at a set frame rate; Step 2. Image preprocessing: Preprocess the acquired image data; Step 3. Feature Extraction: Extract key features from the image; Step 4. Behavior recognition: Identify the driver's behavior based on the extracted features.

6. The working method of the edge computing real-time early warning system for crane operator violations according to claim 5, characterized in that: In step 2, the preprocessing of image data includes grayscale conversion, binarization, and noise reduction.

7. The working method of the edge computing real-time early warning system for crane operator violations according to claim 6, characterized in that: In step 3, the key features are the face, eyes, and hands.

8. The working method of the edge computing real-time early warning system for crane operator violations according to claim 7, characterized in that: In step 4, the high-risk behaviors defined for behavior recognition include dozing off, making phone calls, smoking, and obstructing the camera.

Images