Crane obstacle avoidance function online monitoring method and system based on computer vision and digital twinning

Abstract

The invention relates to the technical field of computer vision, in particular to a crane obstacle avoidance function online monitoring method and system based on computer vision and digital twinning, and the method comprises the steps: 1, collecting image data of a crane working area and crane motion state data; 2, processing the image data to extract obstacle information; 3, a digital twinborn model of the crane is constructed and model parameters are updated, the digital twinborn model of the crane comprises a three-dimensional geometric model and a physical behavior model of the crane, the obstacle information and the crane motion state data are mapped into the digital twinborn model in the model parameter updating process, and the updating frequency is consistent with the data acquisition frequency; 4, simulating the movement of the crane in the digital twinborn model and carrying out collision detection; and 5, generating an obstacle avoidance monitoring result and triggering an early warning mechanism. The potential collision risk is found in advance by monitoring the obstacles and the motion state of the working area of the crane in real time, and collision accidents in crane operation are avoided.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an online monitoring method and system for crane obstacle avoidance function based on computer vision and digital twins. Background Technology

[0002] Cranes, as heavy industrial equipment, are widely used in ports, construction sites, and factories, and their safe operation is crucial. Obstacle avoidance is a core component of crane safety systems, designed to prevent collisions with obstacles during operation. In existing technologies, crane obstacle avoidance primarily relies on physical sensors, such as ultrasonic sensors, lidar, or infrared sensors. These sensors detect obstacles by emitting and receiving signals and trigger alarms or stop mechanisms based on preset thresholds. However, these methods have several limitations. First, physical sensors have limited detection ranges and are easily affected by environmental factors. For example, in rainy, snowy, or dusty environments, sensor accuracy significantly decreases, leading to false alarms or missed alarms. Second, existing obstacle avoidance systems are typically based on static thresholds and cannot adapt to dynamically changing environments. For instance, when obstacles move or the crane's working path is complex, the system struggles to adjust its obstacle avoidance strategy in real time. Furthermore, existing technologies lack comprehensive monitoring of the crane's overall operating status, often focusing only on local obstacle detection while neglecting the interaction between the crane's structural movement and the environment. This can potentially lead to collisions during high-speed or large-scale movements.

[0003] Another problem is that existing obstacle avoidance systems often rely on offline data or pre-programmed paths, failing to achieve true online monitoring. Online monitoring requires the system to collect data, process information, and make decisions in real time, but existing methods are insufficient in data processing speed and analytical depth. For example, computer vision-based obstacle avoidance methods are gaining popularity in industrial applications, using cameras to capture images and identify obstacles. However, relying solely on computer vision is susceptible to changes in lighting, occlusion, and viewing angle limitations, leading to inaccurate identification. Meanwhile, digital twin technology, as an emerging approach, can simulate the behavior of physical entities through virtual models; however, in crane obstacle avoidance applications, digital twins are often used for post-event analysis or design phases, rather than real-time monitoring. In existing technologies, the combined application of computer vision and digital twins is limited, and there is a lack of optimization schemes for the dynamic environment of cranes. This results in shortcomings in obstacle avoidance functionality in terms of real-time performance, accuracy, and adaptability, failing to meet the needs of complex industrial scenarios.

[0004] Furthermore, existing obstacle avoidance systems suffer from deficiencies in parameter acquisition and processing. For example, obstacle distance calculations typically rely on single-sensor data, lacking multi-source data fusion, leading to measurement errors. Crane motion parameters are often acquired via encoders or GPS, but this data is not effectively integrated with visual information, causing delays in obstacle avoidance decisions. The low update frequency of digital twin models fails to reflect real-time changes, further reducing system reliability. Overall, existing technologies lack an online monitoring method integrating computer vision and digital twins, hindering the achievement of efficient and accurate crane obstacle avoidance. Therefore, an innovative technical solution is urgently needed that combines real-time data acquisition, advanced image processing, and dynamic digital twin simulation to address these issues. Summary of the Invention

[0005] To achieve the above objectives, this invention provides an online monitoring method and system for crane obstacle avoidance function based on computer vision and digital twins. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twins includes the following steps: Step 1: Collect image data of the crane's working area and crane motion status data. The image data is collected by multiple cameras deployed on the crane structure, including color images and depth images. The acquisition frequency is dynamically adjusted according to the crane's working speed. The crane motion status data includes the crane's current position, speed, and direction of movement, which is acquired through encoders and position sensors. The timestamps of the image data and the crane motion status data are synchronized. Step 2: Process image data to extract obstacle information, including image preprocessing, obstacle detection, and obstacle feature extraction. Preprocessing includes denoising and enhancement. Obstacle detection is achieved through feature matching and contour analysis. Obstacle feature extraction includes calculating the position coordinates, size, and movement trend of obstacles. Step 3: Construct a digital twin model of the crane and update the model parameters. The digital twin model of the crane includes a three-dimensional geometric model and a physical behavior model of the crane. The model parameter update is achieved by mapping obstacle information and crane motion state data into the digital twin model. The update frequency is consistent with the data acquisition frequency. Step 4: Simulate the crane's movement and perform collision detection in the digital twin model. The motion simulation calculates the crane's future trajectory based on the physical behavior model. Collision detection is achieved by calculating the spatial distance between the crane structure and obstacles. The distance calculation method adopts geometric intersection test. If the distance is lower than the safety threshold, it is marked as a potential collision risk. Step 5: Generate obstacle avoidance monitoring results and trigger the early warning mechanism. The obstacle avoidance monitoring results include the crane's obstacle avoidance status assessment. The assessment outputs a safe, warning, or dangerous status based on the risk level and risk location. The early warning mechanism triggers an audible and visual alarm or sends a stop signal based on the assessment results.

[0006] Preferably, the specific process of acquiring image data of the crane's working area and crane motion status data in step 1 includes: Multiple cameras are installed on the crane boom, hook, and cab to cover the main areas of the crane's working range and avoid blind spots. The image data collected by the cameras includes high-resolution color images and depth images. The depth images are acquired through stereo vision or time-of-flight camera technology. The acquisition frequency is dynamically adjusted according to the crane's working speed. The adjustment process is based on real-time calculation of the crane's movement speed. When the crane's movement speed increases, the acquisition frequency increases accordingly, and when the crane's movement speed decreases, the acquisition frequency decreases accordingly. The crane's motion status data is collected by encoders to obtain the rotation angle and displacement of the crane's winch and traveling mechanism, and by position sensors to obtain the crane's global positioning system coordinates or indoor positioning system coordinates. The motion speed in the crane's motion status data is calculated by the rate of change of displacement, and the motion direction is calculated by the direction vector of the coordinate sequence. The timestamp synchronization of image data and crane motion status data is achieved through a high-precision clock module. The clock module is deployed in the data acquisition unit to ensure that the time deviation between the image data frame and the crane motion status data point is less than the preset tolerance. The data is sent to the central processing unit through a network transmission protocol, and data compression and encryption measures are adopted during the transmission process.

[0007] Preferably, the specific process of processing image data to extract obstacle information in step 2 includes: Image preprocessing first uses Gaussian filtering to remove noise and reduce ambient light and dust interference, and then uses histogram equalization to enhance the image and improve its contrast. Image segmentation uses a thresholding method to distinguish between foreground and background. The foreground includes potential obstacle areas, while the background includes crane structures and fixed environments. Obstacle detection is achieved through feature matching, which uses a scale-invariant feature transformation algorithm to extract key points in the image and match them with a predefined obstacle template. Obstacle detection is also achieved through contour analysis, which uses an edge detection algorithm to extract obstacle boundaries and combines morphological operations to fill the internal regions of the boundaries. In obstacle feature extraction, the position coordinates are obtained by transforming the pixel coordinates to the world coordinate system. The transformation process is based on camera calibration parameters and depth image data. The size is calculated by the image scale, which is determined based on the camera focal length and object distance. The motion trend is calculated by the displacement of the obstacle in consecutive frame images, and the displacement is analyzed by optical flow or block matching methods.

[0008] Preferably, obstacle feature extraction in step 2 also includes obstacle type identification and obstacle motion prediction: Obstacle type recognition is achieved through a convolutional neural network model, which is pre-trained on various obstacle datasets, including categories of people, vehicles, and equipment. The recognition results are used to adjust the obstacle avoidance strategy. Obstacle motion prediction is achieved through time series analysis, which uses an autoregressive integral moving average model to predict future positions based on historical location coordinates. The prediction results are used to update obstacle behavior in the digital twin model. Obstacle feature extraction also involves multi-camera data fusion. Multi-camera data fusion integrates image data from different perspectives through triangulation to improve the accuracy of position coordinates. The fusion process includes data association and error correction.

[0009] Preferably, the specific process of constructing the digital twin model of the crane and updating the model parameters in step 3 includes: The three-dimensional geometric model of the crane is created based on computer-aided design drawings, including the geometry and connection relationships of the crane boom, hook, support and traveling mechanism. The physical behavior model is constructed based on multibody dynamics theory to simulate the kinematic and dynamic characteristics of the crane, including mass, inertia and friction parameters. The model parameter update is driven by real-time data. The obstacle position coordinates and size in the obstacle information are mapped to the digital twin model as obstacle entities. The crane's current position, speed and direction of movement in the crane motion state data are mapped to the digital twin model to update the crane state. The update frequency is consistent with the data acquisition frequency to ensure that the digital twin model is synchronized with the physical world. The model parameter update also includes environmental parameters, which are obtained through background analysis of image data, including the effects of ground flatness and wind speed. Environmental parameters are integrated into the physical behavior model to adjust the simulation accuracy.

[0010] Preferably, the model parameter update in step 3 also includes real-time calibration and error compensation: Real-time calibration is achieved by comparing the output of the digital twin model with the actual sensor data. If the deviation exceeds a preset threshold, the model parameters are adjusted. The calibration process uses Kalman filtering to reduce the impact of noise. Error compensation targets environmental factors, including temperature changes and mechanical wear. Error compensation is achieved through lookup tables or linear regression models. Lookup tables are built based on experimental data, while linear regression models are trained based on historical error data. Model parameter updates also involve dynamic load processing. The dynamic load is obtained through the lifting weight information in the crane's motion state data. The lifting weight information is calculated through force sensors or motor current. The dynamic load is integrated into the physical behavior model and affects the motion simulation.

[0011] Preferably, the specific process of simulating crane movement and performing collision detection in the digital twin model in step 4 includes: The motion simulation is based on a physical behavior model and uses a numerical integration method to calculate the trajectory of the crane in the future time step. The time step is determined according to the crane's speed; the higher the speed, the smaller the time step. Collision detection employs a hierarchical bounding box method for geometric intersection testing. The hierarchical bounding box includes axis-aligned bounding boxes and orientation bounding boxes, which are used to simplify the complex geometry of crane structures and obstacles. Spatial distance calculation is achieved by solving the minimum Euclidean distance between the crane structure and the obstacle. If the minimum Euclidean distance is lower than the safety threshold, it is marked as a potential collision risk. The safety threshold is dynamically adjusted based on the crane type and working environment. The adjustment process is based on statistical analysis of historical collision data, which includes records of close-range events in previous operations. The safety threshold is also corrected for environmental complexity, which is evaluated by a combination of obstacle density and movement speed.

[0012] Preferably, the specific process of generating obstacle avoidance monitoring results and triggering the early warning mechanism in step 5 includes: Obstacle avoidance status assessment is based on the risk level and risk location in the collision detection results. The risk level is divided according to the ratio of spatial distance to safety threshold. The smaller the ratio, the higher the risk level. The risk location is output through the coordinates in the digital twin model. The assessment output is a safe state, a warning state, or a dangerous state. A safe state indicates no collision risk, a warning state indicates a potential collision risk that needs attention, and a dangerous state indicates an immediate collision risk that requires intervention. The early warning mechanism is triggered based on the assessment results. If the status is warning or danger, visual and auditory signals are issued through the sound and light alarm, or a stop signal is sent to the crane drive unit through the control system. The warning signal includes the risk location and recommended obstacle avoidance actions. The recommended obstacle avoidance actions are calculated based on the simulation results in the digital twin model. The simulation results generate the optimal obstacle avoidance path through a path planning algorithm, which takes into account the crane's motion constraints and the obstacle's motion trend.

[0013] Preferably, the early warning mechanism in step 5 also includes multi-level response and log recording: The multi-level response is triggered according to the risk level. In the warning state, the early warning mechanism only issues a warning signal without interrupting the crane operation. In the dangerous state, the early warning mechanism immediately sends a stop signal and initiates emergency braking. The log records include obstacle avoidance monitoring results, warning signals, and operation history. The log data is used for subsequent analysis and system optimization. The storage format adopts a structured database and supports querying and visualization. The early warning mechanism is also integrated into the remote monitoring platform, which receives real-time data via a wireless network, allowing operators to remotely intervene and adjust obstacle avoidance parameters.

[0014] Accordingly, embodiments of the present invention also provide an online monitoring system for crane obstacle avoidance function based on computer vision and digital twin, including a memory configured to store instructions, a processor configured to call the instructions from the memory, and capable of implementing the online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in any embodiment of the present invention when executing the instructions.

[0015] The beneficial effects of this invention are: 1. This invention, combining computer vision and digital twin technology, enables more accurate perception of the dynamic environment surrounding a crane. Computer vision overcomes issues related to lighting, viewing angle, and occlusion, while digital twin technology, through real-time updates of the virtual model, allows the system to operate efficiently even in complex environmental conditions, significantly improving the reliability and accuracy of the obstacle avoidance system.

[0016] 2. This invention employs dynamic digital twin simulation, enabling the system to continuously adapt to environmental changes by updating the virtual model in real time. Digital twin technology can reflect structural changes in real time during crane movement and, combined with computer vision for obstacle detection, achieves a more intelligent dynamic obstacle avoidance strategy.

[0017] 3. This invention achieves comprehensive monitoring of the overall operating status of the crane by introducing digital twin technology and real-time monitoring. The digital twin model can simulate the overall structure and movement of the crane, and detect potential risks in the movement state in a timely manner, thereby effectively avoiding collision accidents.

[0018] 4. The technical solution of this invention, combining real-time data acquisition and advanced image processing, enables online monitoring and real-time decision-making. Computer vision combined with digital twin technology allows the system to acquire environmental data in real time, process and analyze it rapidly, providing cranes with immediate obstacle avoidance strategies and effectively preventing collision risks caused by data delays or inapplicable pre-programmed paths.

[0019] 5. This invention employs multi-source data fusion technology, combining data from sensors and motion parameters, to accurately calculate the position of obstacles and the trajectory of the crane. This multi-source data fusion technology significantly improves data accuracy and reduces the risk of false alarms and missed alarms.

[0020] 6. This invention employs a high-frequency digital twin model update mechanism to ensure the system can reflect the crane's status and changes in the surrounding environment in real time. This high-frequency update improves the system's real-time performance and adaptability, making obstacle avoidance decisions more accurate and efficient. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0022] Figure 1 This is a flowchart of the steps of the method of the present invention; Figure 2 This is a flowchart of step 4 of the method of the present invention, which involves simulating the movement of a crane in a digital twin model and performing collision detection. Figure 3 This is a flowchart of step 5 of the method of the present invention, which generates obstacle avoidance monitoring results and triggers an early warning mechanism. Detailed Implementation

[0023] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should also be noted that, to make the embodiments more comprehensive, the following embodiments are the best and preferred embodiments, and those skilled in the art can use other alternative methods to implement some well-known technologies; moreover, the accompanying drawings are only for more specific description of the embodiments and are not intended to specifically limit the present invention.

[0024] Please see Figures 1-3 This invention provides an online monitoring method for crane obstacle avoidance based on computer vision and digital twins. In step 1, multiple cameras are installed on the crane structure to collect real-time image data of the crane's working area. The image data includes color images and depth images. These images help the system capture obstacle information around the crane. Furthermore, the crane's motion status data (including position, speed, and direction) is acquired in real-time through encoders and position sensors. This data not only helps in understanding the crane's trajectory but also ensures synchronization with the timestamps of the image data, thus providing an accurate spatiotemporal context for subsequent image processing and collision detection.

[0025] By dynamically adjusting the image acquisition frequency, the system can optimize data acquisition in real time according to the crane's operating speed, enabling the system to maintain a high-efficiency working state at any speed. Furthermore, the position data is synchronized with the image data, ensuring the accuracy and timeliness of subsequent processing.

[0026] In step 2, after preprocessing steps (such as denoising and enhancement), the image data undergoes obstacle detection and feature extraction. Through feature matching and contour analysis methods, potential obstacles can be identified from color and depth images, and their location coordinates, size, and motion trends can be further extracted. This information provides crucial input for the establishment of the digital twin model and collision detection.

[0027] This process can accurately detect and extract obstacles in complex working environments, avoiding misidentification caused by environmental complexity. The combination of preprocessing and obstacle detection improves detection accuracy and provides reliable data support for subsequent collision risk assessment.

[0028] In step 3, the digital twin model consists of a three-dimensional geometric model and a physical behavior model of the crane. The three-dimensional geometric model describes the structure and components of the crane, while the physical behavior model simulates the crane's dynamic motion behavior. After each acquisition of image data and motion status data, the system updates the parameters of the digital twin model in real time to ensure that the virtual model reflects the real-time state of the current environment.

[0029] By updating the digital twin model in real time, the system can continuously track and simulate the dynamic state and environmental changes of the crane, providing the latest data support for the next step of motion simulation and collision detection, thereby improving the system's real-time performance and accuracy.

[0030] In step 4, the crane's motion is simulated based on the physical behavior model within the digital twin model to calculate its future trajectory. A geometric intersection test is used to calculate the spatial distance between the crane structure and obstacles. If this distance is below a preset safety threshold, it is marked as a potential collision risk.

[0031] This step enables the system to predict potential collision risks between the crane and obstacles and take early warning measures. Through spatial distance calculation and geometric intersection testing, the collision detection has high accuracy and can accurately identify potential risks in complex working environments.

[0032] In step 5, the system generates obstacle avoidance monitoring results and assesses the crane's obstacle avoidance status based on risk level and location. According to the assessment results, the system outputs a "Safe," "Warning," or "Danger" status. When the risk reaches a certain level, the system will trigger an audible and visual alarm or send a stop signal to remind the operator to take appropriate countermeasures.

[0033] The early warning mechanism can respond promptly to collision risks and provide real-time feedback, effectively preventing accidents caused by operators' failure to detect obstacles in time. Through multi-level risk assessment and timely alarm response, this invention improves the safety and reliability of cranes in complex working environments.

[0034] This invention combines computer vision with digital twin technology to create a comprehensive, real-time online crane obstacle avoidance monitoring system. This system accurately senses and predicts environmental changes, quickly detects obstacles, simulates the crane's movement trajectory, and performs collision risk assessments. This method overcomes the limitations of existing technologies, such as the constraints of physical sensors and the incompatibility of static thresholds with dynamic environments. It significantly improves the real-time performance, accuracy, and adaptability of crane obstacle avoidance, effectively ensuring the safe operation of cranes.

[0035] In one possible implementation, multiple cameras are strategically mounted on the crane boom, hook, and cab to cover the main working area of ​​the crane and avoid blind spots. These cameras acquire high-resolution color and depth images of the crane's working area. The depth images are obtained using two technologies: stereo vision and time-of-flight camera technology. Stereo vision uses image data from multiple cameras at different angles to calculate the depth information of objects; the time-of-flight camera directly acquires depth information by measuring the reflection time of light signals. Thus, by combining high-resolution and depth images, the system can more accurately identify and analyze obstacles.

[0036] The image acquisition frequency is dynamically adjusted based on the crane's movement speed. When the crane moves faster, the system needs to acquire images more frequently to capture fast-moving obstacles; conversely, when the crane moves slower, the image acquisition frequency decreases accordingly. This adjustment process is accomplished by calculating the crane's movement speed in real time, ensuring effective image data acquisition under different operating conditions and preventing image information loss or untimely updates due to excessively low frequency.

[0037] The crane's motion data is collected in two ways: first, encoders are used to monitor the rotation angle and displacement of the hoist and traveling mechanism; second, position sensors are used to acquire the crane's Global Positioning System (GPS) coordinates or Indoor Positioning System (IPS) coordinates. This data helps the system accurately determine the crane's current position and further calculate its speed (via the rate of change of displacement) and direction of motion (via the direction vector of the coordinate sequence). This motion data, along with image data, provides crucial spatiotemporal information for subsequent collision detection and obstacle avoidance decisions.

[0038] To ensure time synchronization between image data and crane motion status data, the system employs a high-precision clock module. This clock module, deployed in the data acquisition unit, ensures that the time deviation between image data frames and motion status data points is less than a preset tolerance, which is crucial for real-time collision detection and obstacle avoidance. After data acquisition, the image data and motion status data are transmitted to the central processing unit via a network transmission protocol. To ensure the stability and security of data transmission, data compression and encryption measures are employed during transmission, effectively reducing bandwidth pressure while protecting data privacy and integrity.

[0039] Through efficient and accurate data collection, synchronization and transmission, the crane's obstacle avoidance and monitoring capabilities in complex environments have been greatly improved, effectively enhancing work safety and enabling timely response to changes, thus avoiding risks caused by environmental complexity or operational errors.

[0040] In one possible implementation, the purpose of image preprocessing is to eliminate environmental interference and improve image quality, thereby providing more accurate input for subsequent obstacle detection. First, Gaussian filtering is used to denoise the image. Gaussian filtering smooths each pixel in the image with a weighted average of its neighboring pixels, reducing interference from ambient lighting variations, dust, or camera noise. Second, histogram equalization is used to enhance the image. This method improves image contrast by adjusting the brightness distribution, making details in the image more prominent, especially in complex environments, effectively improving the visibility of obstacles.

[0041] The goal of image segmentation is to separate the foreground (potential obstacle regions) from the background (such as the structure of a crane or a fixed environment). Thresholding segmentation methods divide pixels into two categories based on image grayscale values: foreground (obstacles) and background. By adjusting the threshold setting, relatively accurate foreground extraction can be achieved under different lighting conditions, allowing potential obstacles to be clearly separated.

[0042] Obstacle detection employs two main methods: feature matching and contour analysis. In feature matching, the Scale Invariant Feature Transform (SIFT) algorithm is used to extract key points in the image and match them against a predefined obstacle template. This method is robust to scale, rotation, and illumination changes, enabling accurate obstacle identification. Secondly, edge detection algorithms (such as the Canny algorithm) are used to extract obstacle boundaries, and morphological operations (such as dilation and erosion) are applied to fill the obstacle's interior region, thus obtaining the complete obstacle contour.

[0043] Obstacle feature extraction is crucial to the system's decision-making process. First, location coordinates are calculated by converting pixel coordinates to the world coordinate system. This conversion relies on camera calibration parameters and depth image data, which allow for the calculation of the obstacle's precise location in 3D space. Second, the obstacle's size is calculated using an image scale determined by the camera's focal length and object distance. Finally, the obstacle's motion trend is calculated through its displacement across consecutive frames. Optical flow or block matching methods are used to analyze the obstacle's movement within the image, thereby inferring its direction and velocity.

[0044] The embodiments of the present invention effectively improve the real-time performance, accuracy and robustness of the obstacle avoidance function of cranes, which is of great significance for improving operational safety and reducing collision risks.

[0045] In one possible implementation, a three-dimensional geometric model of the crane is first created based on its computer-aided design (CAD) drawings. This model includes the structural shapes and connections of key components such as the boom, hook, support, and traveling mechanism. During model creation, constraints such as dimensions, connection nodes, revolute joints, and sliding joints for each component must be defined to ensure that the virtual model maintains consistency with the actual equipment in terms of spatial structure and motion characteristics. After modeling is complete, a data format compatible with the simulation engine is exported using 3D modeling software (such as SolidWorks, CATIA, or UG), providing a geometric foundation for subsequent physical modeling.

[0046] The physical behavior model, built upon the principles of multibody dynamics, is used to simulate the kinematics and dynamics of a crane. Within a digital twin environment, the model defines the mass distribution, moment of inertia, center of gravity position, and frictional characteristics of each structural component. Constraint equations describe the driving relationships of the boom rotation, hook lifting, and traveling mechanisms, realizing the physical response of the virtual crane. The model can also automatically adjust dynamic parameters according to different operating conditions (such as heavy load, no load, or tilting operation) to ensure simulation accuracy.

[0047] The core of a digital twin model is real-time synchronization. The system updates model parameters using real-time data collected by sensors and a vision system. The position coordinates and dimensions of obstacles are mapped into virtual space to generate obstacle entities for dynamic obstacle avoidance analysis. Simultaneously, the crane's motion status data, including position, speed, and direction, is continuously input to update the virtual model's attitude and trajectory. This process is automatically executed by the data interface module, with an update frequency consistent with the data acquisition frequency, thus achieving real-time correspondence between the digital twin model and the physical crane.

[0048] Updating environmental parameters is a crucial step in enhancing the realism of the simulation. The system analyzes background information from images to obtain environmental parameters such as ground flatness, wind speed, and lighting conditions. Ground flatness data is used to correct the support status of the crane chassis, while wind speed information is used to simulate the stress changes on the boom and load. All environmental parameters are input into the physical behavior model in real time, dynamically correcting the dynamic equations to ensure that the simulation results are highly consistent with actual working conditions.

[0049] Through multi-dimensional methods such as geometric modeling, dynamic simulation, data-driven updates, and environmental integration, a high-precision construction and dynamic synchronization of a crane digital twin system has been achieved, providing a solid technical foundation for intelligent monitoring of crane obstacle avoidance functions.

[0050] In one possible implementation, motion simulation is based on a physical behavior model of the crane, employing a numerical integration method to predict the crane's future trajectory. By dividing time into several small time steps, the system determines the length of each time step based on the current speed of the crane. Specifically, the higher the crane's speed, the smaller the time step, ensuring higher computational precision and accurate trajectory prediction. In this way, the simulated trajectory reflects the crane's true dynamic behavior, providing accurate motion data for subsequent collision detection.

[0051] In the collision detection process, a hierarchical bounding box method is used for geometric intersection testing. Hierarchical bounding boxes simplify the modeling of the crane structure and surrounding obstacles by using multiple nested bounding boxes to enclose the object's geometry, reducing computational complexity. Bounding boxes are divided into axis-aligned bounding boxes (AABB) and oriented bounding boxes (OBB). The former simplifies calculations but may not be accurate enough for complex shapes, while the latter can more accurately fit the object's shape and is suitable for handling complex obstacles and crane structures.

[0052] The key to collision detection lies in spatial distance calculation, specifically solving for the minimum Euclidean distance between the crane structure and obstacles. If this distance is below a set safety threshold, the system marks the area as a potential collision risk. Calculating the minimum Euclidean distance allows for an accurate assessment of the relative position of the crane and obstacles, thereby determining whether a collision is possible.

[0053] The safety threshold is a key parameter for avoiding collisions, and it needs to be dynamically adjusted based on the crane type and working environment. In practical applications, the safety threshold is not fixed but adjusted based on historical collision data. Historical collision data contains records of close-range events that occurred during past operations, which help the system understand the crane's performance in different environments. Through statistical analysis of historical data, the system can automatically adjust the safety threshold based on past experience. Furthermore, the safety threshold also considers environmental complexity, which is comprehensively assessed through obstacle density and crane speed. For example, when obstacle density is high or speed is high, the safety threshold will be appropriately lowered to improve collision detection sensitivity and ensure higher safety.

[0054] Through precise motion simulation and intelligent risk assessment, the safety and efficiency of crane operations have been significantly improved.

[0055] In one possible implementation, obstacle avoidance status assessment is based on the aforementioned collision detection results, determining the crane's safety status by analyzing risk level and risk location. Risk level is determined by the ratio of spatial distance to a safety threshold. Specifically, the smaller the distance between the crane and the obstacle, or the smaller the ratio of distance to a set safety threshold, the higher the collision risk, and the higher the risk level of the assessment result. This ratio can be updated in real time to ensure timely monitoring and identification of collision risks. Risk location is determined using coordinate information in a digital twin model. The relative positions of the crane and the obstacle are accurately displayed in this model, facilitating real-time monitoring of potential hazardous areas by monitoring personnel.

[0056] Based on the risk level and location assessment, the system will output the following three statuses: Safe status: This indicates that there is currently no risk of collision, and the crane can continue operating.

[0057] Warning status: This indicates a potential collision risk, requiring operators to exercise special caution and be prepared to take action.

[0058] Hazardous condition: This indicates an immediate risk of collision, requiring immediate intervention to avoid an accident.

[0059] Once the status is assessed as a warning or hazard, the system will activate an early warning mechanism, issuing visual and audible signals via sound and light alarms to alert operators to take immediate action. If the risk is severe, the system can also automatically send a stop signal to the crane's drive unit via the control system, ensuring the crane stops quickly in dangerous situations and preventing collisions. This early warning mechanism not only increases the response time for manual intervention but also enables a degree of automated intervention, improving overall safety.

[0060] To further reduce collision risk, the system also generates recommended obstacle avoidance actions based on simulation results from the digital twin model. The simulation results are derived through a path planning algorithm, which calculates the optimal obstacle avoidance path for the crane to avoid collisions with obstacles. During path planning, the algorithm considers the crane's motion constraints, such as maximum speed and minimum turning radius, to ensure the recommended path is feasible and efficient. Furthermore, the algorithm considers the movement trends of obstacles, such as their speed and direction, thereby dynamically adjusting the obstacle avoidance path to ensure the crane can safely navigate even in complex and changing environments.

[0061] This obstacle avoidance monitoring and early warning mechanism, through precise status assessment and intelligent path planning, not only improves the safety of crane operations but also provides operators with effective decision support.

[0062] In one possible implementation, obstacle type recognition is achieved through a convolutional neural network (CNN) model. This model is pre-trained using training data containing various obstacle datasets, covering different categories of obstacles such as people, vehicles, and equipment. Through the CNN model, the system can automatically identify the type of obstacle in an image, providing crucial information for adjusting obstacle avoidance strategies. For example, if an obstacle is identified as a person, the system might prioritize a strategy of slow stopping and minor adjustments; if it's identified as a vehicle or equipment, it might adopt a more direct avoidance path. In this way, the system can flexibly adjust its operational strategy based on the specific category of the obstacle, thereby improving operational efficiency and safety.

[0063] Obstacle motion prediction is achieved through time series analysis, specifically using an Autoregressive Integral Moving Average (ARIMA) model. This model analyzes the trajectory of obstacles and predicts their future positions based on historical location coordinate data. This is particularly important for dynamic obstacles (such as moving vehicles or people) because their motion states change over time, and accurate predictions help the system update obstacle behavior in the digital twin model in a timely manner. For example, if the system predicts that an obstacle will enter the crane's working area in the next few seconds, it can adjust in advance to avoid a collision. Such predictions significantly improve the real-time performance and accuracy of obstacle avoidance strategies, reducing the need for human intervention.

[0064] To accurately determine the location of obstacles, the system employs multi-camera data fusion technology. By capturing images from multiple cameras at different angles, the system integrates the data from these diverse perspectives using triangulation. This method improves the accuracy of obstacle location coordinates, especially in complex environments, where image data from multiple perspectives can complement each other, reducing blind spots or errors from a single camera's viewpoint. The fusion process includes data association and error correction. Data association involves matching data from different cameras to ensure the consistent position of the same obstacle in different images; error correction corrects potential errors in the images to improve the accuracy of the final calculation results.

[0065] By conducting multi-dimensional analysis of obstacle types, movement behavior, and locations, the accuracy and real-time performance of the crane obstacle avoidance system are improved, and strong technical support is provided for safety management.

[0066] In one possible implementation, real-time calibration is achieved by comparing the output of the digital twin model with actual sensor data. The system collects sensor data from various parts of the crane in real time, including information on position, attitude, and speed, and compares it with the results simulated by the digital twin model. If the deviation exceeds a preset threshold, the system automatically adjusts the parameters of the digital twin model to make the simulation results closer to reality. To improve calibration accuracy and reduce the impact of sensor noise on parameter updates, a Kalman filter method is used during the calibration process. By fusing historical data and current measurements, noise is effectively smoothed and more reliable parameter corrections are obtained.

[0067] Error compensation addresses model deviations caused by environmental factors, primarily temperature variations and mechanical wear. Temperature changes can lead to thermal expansion of mechanical components, while long-term mechanical wear can affect the crane's motion accuracy. Therefore, the system employs two methods for error compensation: Lookup Table: Data on model errors under different temperatures and wear conditions are obtained through experiments and compiled into a lookup table. The system queries the lookup table based on the current environmental conditions to quickly correct the model output.

[0068] Linear regression model: The linear regression model is trained using historical error data, and compensation is made by predicting the impact of environmental factors on the model, thereby improving the adaptability and continuity of error correction.

[0069] Dynamic load processing is used to account for changes in the crane's motion characteristics due to variations in the load during actual operation. The system measures the load information using force sensors or motor current and converts it into dynamic load data. This dynamic load is then integrated into the physical behavior module of the digital twin model, affecting the results of the motion simulation. For example, an increase in load will cause changes in the crane's acceleration and sway amplitude; the system adjusts its obstacle avoidance strategy by simulating these changes to ensure operational safety.

[0070] Through multi-dimensional calibration and compensation, the digital twin model can reflect the actual state of the crane in real time, improving the reliability and intelligence level of the obstacle avoidance system.

[0071] In one possible implementation, the multi-level response mechanism triggers different early warning measures based on different risk levels, aiming to dynamically adjust the crane's response mode according to the proximity and hazard of the obstacle. Specific steps include: When the system detects an approaching obstacle or other potential hazard, the warning mechanism only issues a warning signal if the risk is low. At this time, crane operation is unaffected, and the operator can assess the situation and take appropriate action based on the warning signal. For example, the system may issue a visual or audible alarm to alert the operator to changes in the surrounding environment. The response during the warning state does not interrupt operation, ensuring that work progress is not affected.

[0072] When the system determines that an obstacle has approached to a point where a collision or other serious consequences may occur, it enters a danger state. At this time, the warning mechanism immediately triggers a stop signal and activates the emergency braking system to stop the crane's movement as quickly as possible. This rapid response is to avoid serious accidents that may be caused by delays in human reaction and to ensure the safety of equipment and personnel.

[0073] The logging function is used to store and analyze key data throughout the alerting process. The system will record the following: Obstacle avoidance monitoring results: Monitoring data for each obstacle avoidance task performed by the system, including key information such as obstacle type, location, and movement trajectory. This data provides a basis for subsequent analysis and optimization.

[0074] Warning signals: Every warning signal issued by the system includes the triggering time, triggering conditions, and response measures for warnings and dangerous states.

[0075] Operational history: Records every operator intervention, including adjustments to obstacle avoidance parameters and execution of operational instructions. This data helps to retrospectively analyze problems and evaluate the operator's decision-making process.

[0076] All log data will be stored in a structured database in a standardized format to facilitate subsequent querying, analysis, and visualization. Through these records, the system can not only perform real-time diagnostics but also analyze the operational process afterward, identify potential optimization opportunities, and improve overall system performance through data-driven approaches.

[0077] To improve system operability and response efficiency, the early warning mechanism is also integrated into the remote monitoring platform. The remote monitoring platform receives real-time data from the crane via a wireless network, allowing operators to monitor the system's operational status remotely, regardless of the on-site environment. Operators can view real-time obstacle avoidance monitoring results and early warning signals through the platform, and remotely intervene when an alarm occurs, adjusting obstacle avoidance parameters or initiating an emergency response. This function greatly enhances the operator's decision-making capabilities, especially in large or hazardous environments, enabling operators to adjust strategies promptly and prevent accidents.

[0078] The integration of multi-level response, logging, and remote monitoring in the early warning mechanism not only improves the system's security and intelligence but also enhances operators' control over equipment and their ability to cope with complex situations.

[0079] Accordingly, embodiments of the present invention also provide an online monitoring system for crane obstacle avoidance function based on computer vision and digital twin, including a memory configured to store instructions, a processor configured to call the instructions from the memory, and capable of implementing the online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in any embodiment of the present invention when executing the instructions.

[0080] The implementation of this method relies on the collaborative work of hardware and software components. The hardware includes multiple cameras deployed on the crane (such as industrial-grade color and depth cameras), encoders, position sensors (such as GPS receivers), audible and visual alarms, a central processing unit, and a data transmission network. The software includes a computer vision processing module, a digital twin modeling platform (such as physics engine-based simulation software), data synchronization algorithms, and early warning control logic. All components are integrated through standard industrial protocols to ensure real-time performance and reliability.

[0081] Example In this embodiment, the application scenario is a port container crane that moves containers in a terminal area, where there are moving vehicles, personnel, and other equipment as potential obstacles. The crane's working area covers an area of ​​approximately 100 meters × 50 meters, and the crane includes a boom, hook, and traveling mechanism.

[0082] Step 1: Acquire image data of the crane's working area and crane motion status data. This step involves acquiring image data using multiple cameras deployed on the crane structure, and collecting crane motion status data using encoders and position sensors. The specific implementation is as follows: Camera Deployment: Multiple cameras are installed at the end of the crane boom, above the hook, and on the top of the cab, covering the main areas of the crane's working range and avoiding blind spots. The cameras are industrial-grade high-definition models capable of simultaneously capturing color and depth images. The color image resolution is set to 1920x1080 pixels, and the depth image is acquired using stereo vision technology, employing a dual-camera system to calculate a disparity map to generate depth information. Acquisition Frequency Dynamically Adjusted Based on Crane Operating Speed: The adjustment process is based on real-time calculations of the crane's movement speed, obtained through encoder data. When the crane's speed exceeds 1 meter per second, the acquisition frequency increases to 30 frames per second; when the speed is below 0.5 meters per second, the acquisition frequency decreases to 15 frames per second. This dynamic adjustment ensures real-time data transmission while optimizing processing resources.

[0083] Crane motion status data acquisition: Crane motion status data includes the crane's current position, speed, and direction of movement. The current position is acquired via a GPS receiver with centimeter-level accuracy; the speed is obtained by collecting the rotation angle and displacement of the crane's winch and traveling mechanism using a rotary encoder, and then calculating the rate of displacement change; the direction of movement is calculated using the direction vector from the GPS coordinate sequence, determined based on the position difference between consecutive time points. All sensor data is transmitted via the CAN bus protocol.

[0084] Data synchronization: Timestamp synchronization of image data and crane motion status data is achieved through a high-precision clock module deployed in the data acquisition unit. This module uses Network Time Protocol (NTP) to ensure a time deviation of less than a preset tolerance of 10 milliseconds. Data is transmitted to the central processing unit via the network, employing JPEG compression and AES encryption during transmission to reduce bandwidth consumption and ensure security.

[0085] Step 2: Process image data to extract obstacle information; This step receives the raw image dataset output from step 1, processes the images using computer vision algorithms, and extracts obstacle information. The specific implementation is as follows: Image preprocessing: First, Gaussian filtering is used to denoise the image. The kernel size of the Gaussian filter is adaptively selected according to the image resolution. For example, for a 1920x1080 image, the kernel size is set to 5x5 pixels, and the standard deviation is set to 1.5 to reduce ambient lighting variations and dust interference. Then, histogram equalization is used to enhance the image. Histogram equalization improves image contrast by redistributing pixel intensity values, making obstacles easier to identify.

[0086] Image segmentation: A threshold segmentation method is used to distinguish between the foreground and the background. The threshold segmentation is based on the Otsu algorithm to automatically calculate the optimal threshold. The foreground includes potential obstacle areas (such as people and vehicles), and the background includes crane structures and fixed environments (such as the ground and container stacks).

[0087] Obstacle detection is achieved through feature matching and contour analysis. Feature matching uses the Scale Invariant Feature Transform (SIFT) algorithm to extract key points in the image. The SIFT algorithm detects extreme points in scale space and generates descriptors, which are then matched against a predefined obstacle template. The template includes an image library of people, vehicles, and equipment. Contour analysis uses the Canny edge detection algorithm to extract obstacle boundaries. The threshold for the Canny algorithm was determined experimentally, with a low threshold set to 50 and a high threshold set to 150. Morphological operations (such as dilation and erosion) are combined to fill the internal regions of the boundaries to generate complete obstacle contours.

[0088] Obstacle feature extraction includes calculating the obstacle's position coordinates, size, and motion trend. Position coordinates are obtained by transforming pixel coordinates to the world coordinate system. This transformation is based on camera calibration parameters (such as camera intrinsic and extrinsic parameters) and depth image data, implemented using a perspective transformation matrix. Size is calculated using the image scale, which is determined by the camera's focal length and object distance. For example, with a focal length of 10 mm and an object distance of 20 m, the scale represents 0.05 meters per pixel. Motion trend is calculated by the obstacle's displacement in consecutive frames. Displacement is analyzed using optical flow methods (such as the Lucas-Kanade method), which estimate the obstacle's velocity by tracking the motion vectors of feature points.

[0089] Obstacle type recognition is achieved using a convolutional neural network (CNN) model. The CNN model is pre-trained on the COCO dataset, including categories for people, vehicles, and equipment. The training process uses a stochastic gradient descent optimizer with a learning rate of 0.001. The recognition results are used to adjust obstacle avoidance strategies (e.g., applying a higher safety threshold for people obstacles). Obstacle motion prediction is achieved through time series analysis, employing an autoregressive integral moving average (ARIMA) model to predict future locations based on historical location coordinates. The parameters (p, d, q) of the ARIMA model are determined using the Akaike information criterion, and the prediction results are used to update obstacle behavior in the digital twin model. Multi-camera data fusion integrates image data from different perspectives using triangulation. Triangulation is based on stereo vision principles, calculating three-dimensional coordinates, and using least squares correction to improve the accuracy of location coordinates.

[0090] Step 3: Build a digital twin model of the crane and update the model parameters; This step receives the obstacle information dataset output from step 2 and the crane motion state dataset output from step 1, constructs a digital twin model of the crane, and updates the model parameters. The specific implementation is as follows: Digital twin model construction: The 3D geometric model of the crane was created based on computer-aided design (CAD) drawings. The geometry and connection relationships of the crane's boom, hook, support, and traveling mechanism were imported using SolidWorks software, and the model format was STEP file. The physical behavior model was constructed based on multibody dynamics theory. The kinematic and dynamic characteristics of the crane, including mass, inertia, and friction parameters, were simulated using the Matlab / Simulink platform. The mass parameters were obtained from the crane design manual, the inertia parameters were obtained by calculating the geometric inertia tensor, and the friction parameters were determined through experimental measurements.

[0091] Model parameter updates: Driven by real-time data, the obstacle position coordinates and dimensions from obstacle information are mapped into the digital twin model as obstacle entities, represented using a 3D mesh. The crane's current position, speed, and direction of motion from crane motion data are mapped into the digital twin model to update the crane's state, with the update frequency consistent with the data acquisition frequency (e.g., 15-30Hz). Environmental parameters are obtained through background analysis of image data, including ground flatness (slope calculated from depth images) and wind speed influence (through additional wind speed sensor data). Environmental parameters are integrated into the physical behavior model to adjust simulation accuracy.

[0092] Real-time calibration and error compensation: Real-time calibration is achieved by comparing the output of the digital twin model with actual sensor data. For example, comparing the crane position predicted by the model with the GPS position. If the deviation exceeds a preset threshold (e.g., 0.1 meters), the model parameters are adjusted. The calibration process uses the Kalman filtering method, which reduces the influence of noise through state equations and observation equations. The state equations are based on the kinematic model, and the observation equations are based on sensor data. Error compensation addresses environmental factors such as temperature changes and mechanical wear. Error compensation is achieved through lookup tables, which are constructed based on experimental data. For example, encoder errors are measured at different temperatures and compensation values ​​are stored. Dynamic load processing obtains the load information from the crane's motion state data. The load information is calculated using force sensors or motor current. The relationship between motor current and load is determined through calibration curves. Dynamic load is integrated into the physical behavior model and affects motion simulation.

[0093] Step 4: Simulate crane movement and perform collision detection in the digital twin model; This step receives the updated digital twin model output from step 3 and performs crane motion simulation and collision detection within the model. The specific implementation is as follows: Motion simulation: Based on a physical behavior model, a numerical integration method is used to calculate the crane's trajectory at future time steps. The numerical integration method employs a fourth-order Runge-Kutta algorithm. The time step is determined based on the crane's speed; for example, when the speed exceeds 1 meter per second, the time step is set to 0.1 seconds; when the speed is lower, the time step is set to 0.5 seconds. The simulated trajectory includes boom swing and hook movement.

[0094] Collision detection: A hierarchical bounding box method is used for geometric intersection testing. The hierarchical bounding boxes include axis-aligned bounding boxes (AABB) and orientation bounding boxes (OBB) to simplify the complex geometry of the crane structure and obstacles. Spatial distance is calculated by solving for the minimum Euclidean distance between the crane structure and obstacles using the GJK algorithm. If the minimum Euclidean distance is below a safety threshold, it is marked as a potential collision risk.

[0095] Dynamic adjustment of safety threshold: The safety threshold is dynamically adjusted according to the crane type and working environment. The adjustment process is based on the statistical analysis of historical collision data. Historical collision data includes records of close-range events in previous operations, such as the frequency of events with a distance of less than 2 meters in the past 100 operations. The safety threshold is also corrected by environmental complexity. Environmental complexity is evaluated by a combination of obstacle density and movement speed. Obstacle density is calculated by the number of obstacles per unit area, and movement speed is calculated by the average obstacle speed. Finally, the safety threshold is set as a base value (e.g., 1.5 meters) multiplied by a complexity coefficient (ranging from 0.8 to 1.2).

[0096] Step 5: Generate obstacle avoidance monitoring results and trigger the early warning mechanism; This step receives the collision detection results output from step 4, generates obstacle avoidance monitoring results, and triggers the early warning mechanism. The specific implementation is as follows: Obstacle avoidance status assessment: Based on the risk level and risk location in the collision detection results, the risk level is divided according to the ratio of spatial distance to a safety threshold. A ratio less than 0.5 indicates a dangerous risk level, a ratio between 0.5 and 0.8 indicates a warning, and a ratio greater than 0.8 indicates a safe risk level. The risk location is output through coordinates in the digital twin model.

[0097] Warning mechanism trigger: If the status is warning or danger, visual (such as a flashing red light) and audible (such as a buzzer) signals are issued through the sound and light alarm, or a stop signal is sent to the crane drive unit through the control system. The stop signal is transmitted via the Modbus protocol.

[0098] Recommended obstacle avoidance actions: The warning signal includes the risk location and recommended obstacle avoidance actions. The recommended obstacle avoidance actions are calculated based on the simulation results in the digital twin model. The simulation results generate the optimal obstacle avoidance path through the path planning algorithm. The path planning algorithm adopts the A* algorithm, which considers the crane's motion constraints (such as the minimum turning radius) and the obstacle's motion trend. The heuristic function uses the Manhattan distance.

[0099] Multi-level response and logging: The multi-level response is triggered according to risk level. For warning states, the early warning mechanism only issues a warning signal without interrupting crane operation; for dangerous states, the early warning mechanism immediately sends a stop signal and initiates emergency braking, which controls the crane motor via a relay. Logging includes storing obstacle avoidance monitoring results, warning signals, and operation history. Log data is stored in an SQL database and supports querying and visualization via a web interface. The early warning mechanism is also integrated into a remote monitoring platform, which receives real-time data via a 4G wireless network, allowing operators to remotely intervene and adjust obstacle avoidance parameters, such as modifying safety thresholds.

[0100] This invention encompasses any substitutions, modifications, equivalent methods, and solutions made within the spirit and scope of this invention. To provide the public with a thorough understanding of this invention, specific details are described in detail in the following preferred embodiments; however, those skilled in the art will fully understand the invention even without these details. Furthermore, to avoid unnecessary misunderstanding of the essence of this invention, well-known methods, processes, procedures, components, and circuits are not described in detail.

[0101] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for online monitoring of crane obstacle avoidance function based on computer vision and digital twin, characterized in that, Includes the following steps: Step 1: Collect image data of the crane's working area and crane motion status data. The image data is collected by multiple cameras deployed on the crane structure, including color images and depth images. The acquisition frequency is dynamically adjusted according to the crane's working speed. The crane motion status data includes the crane's current position, speed, and direction of movement, which is acquired through encoders and position sensors. The timestamps of the image data and the crane motion status data are synchronized. Step 2: Process image data to extract obstacle information, including image preprocessing, obstacle detection, and obstacle feature extraction. Preprocessing includes denoising and enhancement. Obstacle detection is achieved through feature matching and contour analysis. Obstacle feature extraction includes calculating the position coordinates, size, and movement trend of obstacles. Step 3: Construct a digital twin model of the crane and update the model parameters. The digital twin model of the crane includes a three-dimensional geometric model and a physical behavior model of the crane. The model parameter update is achieved by mapping obstacle information and crane motion state data into the digital twin model. The update frequency is consistent with the data acquisition frequency. Step 4: Simulate the crane's movement and perform collision detection in the digital twin model. The motion simulation calculates the crane's future trajectory based on the physical behavior model. Collision detection is achieved by calculating the spatial distance between the crane structure and obstacles. The distance calculation method adopts geometric intersection test. If the distance is lower than the safety threshold, it is marked as a potential collision risk. Step 5: Generate obstacle avoidance monitoring results and trigger the early warning mechanism. The obstacle avoidance monitoring results include the crane's obstacle avoidance status assessment. The assessment outputs a safe, warning, or dangerous status based on the risk level and risk location. The early warning mechanism triggers an audible and visual alarm or sends a stop signal based on the assessment results.

2. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 1, characterized in that, The specific process of acquiring image data of the crane's working area and crane motion status data in step 1 includes: Multiple cameras are installed on the crane boom, hook, and cab to cover the main areas of the crane's working range and avoid blind spots. The image data collected by the cameras includes high-resolution color images and depth images. The depth images are acquired through stereo vision or time-of-flight camera technology. The acquisition frequency is dynamically adjusted according to the crane's working speed. The adjustment process is based on real-time calculation of the crane's movement speed. When the crane's movement speed increases, the acquisition frequency increases accordingly, and when the crane's movement speed decreases, the acquisition frequency decreases accordingly. The crane's motion status data is collected by encoders to obtain the rotation angle and displacement of the crane's winch and traveling mechanism, and by position sensors to obtain the crane's global positioning system coordinates or indoor positioning system coordinates. The motion speed in the crane's motion status data is calculated by the rate of change of displacement, and the motion direction is calculated by the direction vector of the coordinate sequence. The timestamp synchronization of image data and crane motion status data is achieved through a high-precision clock module. The clock module is deployed in the data acquisition unit to ensure that the time deviation between the image data frame and the crane motion status data point is less than the preset tolerance. The data is sent to the central processing unit through a network transmission protocol, and data compression and encryption measures are adopted during the transmission process.

3. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 1, characterized in that, The specific process of processing image data to extract obstacle information in step 2 includes: Image preprocessing first uses Gaussian filtering to remove noise and reduce ambient light and dust interference, and then uses histogram equalization to enhance the image and improve its contrast. Image segmentation uses a thresholding method to distinguish between foreground and background. The foreground includes potential obstacle areas, while the background includes crane structures and fixed environments. Obstacle detection is achieved through feature matching, which uses a scale-invariant feature transformation algorithm to extract key points in the image and match them with a predefined obstacle template. Obstacle detection is also achieved through contour analysis, which uses an edge detection algorithm to extract obstacle boundaries and combines morphological operations to fill the internal regions of the boundaries. In obstacle feature extraction, the position coordinates are obtained by transforming the pixel coordinates to the world coordinate system. The transformation process is based on camera calibration parameters and depth image data. The size is calculated by the image scale, which is determined based on the camera focal length and object distance. The motion trend is calculated by the displacement of the obstacle in consecutive frame images, and the displacement is analyzed by optical flow or block matching methods.

4. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 3, characterized in that, Step 2, obstacle feature extraction, also includes obstacle type identification and obstacle motion prediction: Obstacle type recognition is achieved through a convolutional neural network model, which is pre-trained on various obstacle datasets, including categories of people, vehicles, and equipment. The recognition results are used to adjust the obstacle avoidance strategy. Obstacle motion prediction is achieved through time series analysis, which uses an autoregressive integral moving average model to predict future positions based on historical location coordinates. The prediction results are used to update obstacle behavior in the digital twin model. Obstacle feature extraction also involves multi-camera data fusion. Multi-camera data fusion integrates image data from different perspectives through triangulation to improve the accuracy of position coordinates. The fusion process includes data association and error correction.

5. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 1, characterized in that, The specific process of constructing the digital twin model of the crane and updating the model parameters in step 3 includes: The three-dimensional geometric model of the crane is created based on computer-aided design drawings, including the geometry and connection relationships of the crane boom, hook, support and traveling mechanism. The physical behavior model is constructed based on multibody dynamics theory to simulate the kinematic and dynamic characteristics of the crane, including mass, inertia and friction parameters. The model parameter update is driven by real-time data. The obstacle position coordinates and size in the obstacle information are mapped to the digital twin model as obstacle entities. The crane's current position, speed and direction of movement in the crane motion state data are mapped to the digital twin model to update the crane state. The update frequency is consistent with the data acquisition frequency to ensure that the digital twin model is synchronized with the physical world. The model parameter update also includes environmental parameters, which are obtained through background analysis of image data, including the effects of ground flatness and wind speed. Environmental parameters are integrated into the physical behavior model to adjust the simulation accuracy.

6. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 5, characterized in that, Step 3, model parameter updates, also includes real-time calibration and error compensation: Real-time calibration is achieved by comparing the output of the digital twin model with the actual sensor data. If the deviation exceeds a preset threshold, the model parameters are adjusted. The calibration process uses Kalman filtering to reduce the impact of noise. Error compensation targets environmental factors, including temperature changes and mechanical wear. Error compensation is achieved through lookup tables or linear regression models. Lookup tables are built based on experimental data, while linear regression models are trained based on historical error data. Model parameter updates also involve dynamic load processing. The dynamic load is obtained through the lifting weight information in the crane's motion state data. The lifting weight information is calculated through force sensors or motor current. The dynamic load is integrated into the physical behavior model and affects the motion simulation.

7. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 1, characterized in that, Step 4, which involves simulating crane movement and performing collision detection in the digital twin model, includes the following specific steps: The motion simulation is based on a physical behavior model and uses a numerical integration method to calculate the trajectory of the crane in the future time step. The time step is determined according to the crane's speed; the higher the speed, the smaller the time step. Collision detection employs a hierarchical bounding box method for geometric intersection testing. The hierarchical bounding box includes axis-aligned bounding boxes and orientation bounding boxes, which are used to simplify the complex geometry of crane structures and obstacles. Spatial distance calculation is achieved by solving the minimum Euclidean distance between the crane structure and the obstacle. If the minimum Euclidean distance is lower than the safety threshold, it is marked as a potential collision risk. The safety threshold is dynamically adjusted based on the crane type and working environment. The adjustment process is based on statistical analysis of historical collision data, which includes records of close-range events in previous operations. The safety threshold is also corrected for environmental complexity, which is evaluated by a combination of obstacle density and movement speed.

8. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 1, characterized in that, The specific process of generating obstacle avoidance monitoring results and triggering the early warning mechanism in step 5 includes: Obstacle avoidance status assessment is based on the risk level and risk location in the collision detection results. The risk level is divided according to the ratio of spatial distance to safety threshold. The smaller the ratio, the higher the risk level. The risk location is output through the coordinates in the digital twin model. The assessment output is a safe state, a warning state, or a dangerous state. A safe state indicates no collision risk, a warning state indicates a potential collision risk that needs attention, and a dangerous state indicates an immediate collision risk that requires intervention. The early warning mechanism is triggered based on the assessment results. If the status is warning or danger, visual and auditory signals are issued through the sound and light alarm, or a stop signal is sent to the crane drive unit through the control system. The warning signal includes the risk location and recommended obstacle avoidance actions. The recommended obstacle avoidance actions are calculated based on the simulation results in the digital twin model. The simulation results generate the optimal obstacle avoidance path through a path planning algorithm, which takes into account the crane's motion constraints and the obstacle's motion trend.

9. The online monitoring method for crane obstacle avoidance function based on computer vision and digital twin as described in claim 8, characterized in that, Step 5 of the early warning mechanism also includes multi-level response and logging: The multi-level response is triggered according to the risk level. In the warning state, the early warning mechanism only issues a warning signal without interrupting the crane operation. In the dangerous state, the early warning mechanism immediately sends a stop signal and initiates emergency braking. The log records include obstacle avoidance monitoring results, warning signals, and operation history. The log data is used for subsequent analysis and system optimization. The storage format adopts a structured database and supports querying and visualization. The early warning mechanism is also integrated into the remote monitoring platform, which receives real-time data via a wireless network, allowing operators to remotely intervene and adjust obstacle avoidance parameters.

10. An online monitoring system for crane obstacle avoidance function based on computer vision and digital twin, characterized in that, The system includes a memory configured to store instructions, a processor configured to retrieve the instructions from the memory, and, when executing the instructions, to implement the online monitoring method for crane obstacle avoidance based on computer vision and digital twins as described in any one of claims 1 to 9.

Images