Intelligent port crane operation monitoring method and system

Abstract

The invention discloses an intelligent port crane operation monitoring method and system, and relates to the technical field of crane systems.The system comprises a data acquisition unit, an edge computing node, a cloud processor and an early warning monitoring unit, through deep fusion and efficient treatment of multi-source data, a dynamic twinborn data pool architecture is established, and the dynamic twinborn data pool architecture is established; cross-data-type space-time alignment is realized through multi-modal coding, and the feature fusion accuracy is improved; the dynamic twin data pool is intelligently analyzed, and the crane operation path is optimized in real time, so that the track of the crane operation path is dynamically adjusted, and the working efficiency is improved; through a digital twin model and integration of a multi-modal physical engine, the operation physical state deviation of the crane is evaluated, an operation plan is corrected in real time, and the operation precision is improved; video streams are deeply analyzed, virtual-real comparison is carried out, and model prediction video deviation is monitored, so that the analogue simulation precision is improved; by establishing a closed-loop early warning mechanism, full-process digital management and control are realized, and intelligent upgrading of a port is promoted.

Description

Technical Field

[0001] The present invention relates to the technical field of crane systems, and in particular, to a method and system for monitoring the operation of intelligent port cranes. Background Art

[0002] The technical means and system functions for monitoring the operation of intelligent port cranes include: collecting and analyzing the real-time data of cranes through digital twin technology, Internet of Things technology, big data, and artificial intelligence technology to achieve the health status monitoring and operation standardization of equipment; collecting various operation parameters and status information of cranes through sensors to monitor the operation of equipment in real time; and monitoring the operation process of cranes in real time through cameras and high-resolution monitors to ensure operation safety and good equipment status.

[0003] However, the current crane operation monitoring technology has problems such as data island fragmentation, insufficient decision-making real-time performance, and difficulty in comprehensively conducting risk early warning monitoring in multiple dimensions; due to the independent storage of multi-source data such as sensors, video streams, and equipment logs, lacking the support of a unified data middle platform, information fragmentation occurs, high-concurrency data streams cannot be processed in real time, it is difficult to adapt to dynamic operation scenarios and complex mechanical interactions, resulting in insufficient accuracy of digital twin video streams and low credibility of virtual-real comparison, thus leading to slow response to sudden changes, lagging real-time decision-making processing, and lacking comprehensive risk assessment with multi-dimensional scoring, resulting in deficiencies in the operation efficiency and safety of port cranes. In view of the above technical defects, a solution is proposed herein. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems of data island fragmentation, insufficient decision-making real-time performance, and difficulty in comprehensively conducting risk early warning monitoring in multiple dimensions existing in the existing crane operation monitoring technology.

[0005] To achieve the above purpose, the present invention adopts the following technical solutions: A method for monitoring the operation of an intelligent port crane, comprising the following steps: S1, collecting multi-source heterogeneous data of crane operation in real time and constructing a dynamic twin data pool: collecting multi-source heterogeneous data of crane operation in real time through the data middle platform of the port digital control and management platform; S2, intelligently analyzing the dynamic twin data pool and real-time optimizing the crane operation path: analyzing the deviation of port production operation through multi-source heterogeneous data, obtaining the efficiency index score of crane operation, and outputting a dynamic adjustment strategy for the crane; S3. Build a digital twin model and embed a multimodal physics engine: By synchronously simulating the mechanical interaction process of the crane operation in the real environment, the physical parameters of the crane grab are detected in real time, and a physical evaluation report of the crane operation is analyzed and generated, so as to correct the operation plan in real time and push warning instructions to the surrounding systems; S4. Deeply analyze the video stream and conduct virtual-real comparison: Through the monitoring video stream of the port crane operation and the simulated video stream of the digital twin model, extract the key action features of the crane and conduct spatio-temporal alignment comparison to generate a video deviation score for the crane operation; S5. Establish a hierarchical warning mechanism for automatic exception handling: Compare the thresholds through the efficiency index score, physical evaluation report and deviation risk score of the crane operation, output the device warning log, and thus automatically generate a traceability report for the abnormal events of the port crane for closed-loop processing. Furthermore, the multi-source heterogeneous data includes sensor data, device log data and video stream data; Unify the physical coordinates of the crane, container, transport vehicle and port infrastructure in the real environment to the world coordinate system through the spatio-temporal alignment algorithm to generate a high-precision dynamic digital twin model; Clean and denoise the original data, fill in missing values and standardize it through the wavelet transform model, and set a sliding window for dynamic adaptive standardization, so as to unify the time stamp and space coordinate system. The specific process is as follows: Mark the current moment as t; mark the length of the sliding window as W, and the length of the sliding window is the number of data points included in the window; mark the index variable of the moment as i, which is used to traverse all data points of the sliding window; mark the original data value at the i-th moment as ; Thus, obtain the local mean of the sliding window data corresponding to the moment t ; Furthermore, obtain the local standard deviation of the sliding window data corresponding to the moment t ; Through the local mean of the sliding window data corresponding to the moment t and the local standard deviation , standardize the original data value at the moment t, and mark the standardized data value as ; Divide the data types into a real-time stream data layer, a historical data layer and a metadata layer through the distributed storage technology to build a dynamic twin data pool architecture;

[0006] Furthermore, the specific process of S2 is as follows: Extract indicators from device log data, including the task completion time, empty running rate, unit cargo energy consumption, emergency stop times, and load overlimit rate of the crane; mark the number of indicators of the device log data as n0; Mark the reference value of any indicator p of the device log data as Dp0, mark the actual value of indicator p at time t as Dpt, and obtain the deviation amplitude value of indicator p through the difference between the reference value and the actual value of indicator p ; By calculating the deviation amplitude values of n0 indicators And performing maximum-minimum normalization to obtain the normalized value Sp of indicator p; then calculate the information entropy Hp of indicator p to obtain the weight coefficient of indicator p ; Through the deviation amplitude value of indicator p And the weight coefficient Perform weighted fusion to calculate the efficiency index score SC1 of the crane operation.

[0007] Furthermore, the specific process of S3 is as follows: The physical parameters include the grab acceleration of the crane, load stress, container displacement deviation, and vehicle positioning error. Mark the number of indicators of the physical parameters as n1; Mark the reference value of any indicator r of the physical parameters as Dr0, mark the actual value of indicator r at time t as Drt, and obtain the deviation amplitude value of indicator r through the difference between the reference value and the actual value of indicator r , and then obtain the normalized value Sr, information entropy Hr, and weight coefficient of indicator r ; Through the deviation amplitude value of indicator r And the weight coefficient Perform weighted fusion to calculate the physical action score SC2 of the crane operation.

[0008] Furthermore, the specific process of S4 is as follows: Through frame-level parsing of the monitoring video stream, extract the key action features of the crane and perform spatio-temporal alignment and comparison with the digital twin simulation video stream; If the predicted categories of the key action features of the crane by the digital twin model video stream are divided into n2, then the predicted category probability distribution of the digital twin model is ; Encode the true category label of the monitoring video stream of the crane operation as ; Mark the predicted bounding box of the digital twin simulation video stream as B: , where the center coordinates of the predicted bounding box are (b1, b2), the length is b3, and the width is b4; Mark the true bounding box of the monitoring video stream of the crane operation as C: , where the center point coordinates of the ground truth box are (c1, c2), the length is c3, and the width is c4; Calculate the cross-entropy of the virtual-real comparison of the crane action characteristics through spatio-temporal alignment comparison and the complete intersection over union CIoU to obtain the object detection loss Ld; Mark the position coordinate sequence of the target point in consecutive m frames as E, thereby calculating the difference in changes between adjacent frames, and marking the displacements of any pair of adjacent frames in the x direction and y direction as , , thereby obtaining the tracking consistency loss Lt; Calculate the velocity and acceleration of any pair of adjacent frames in the x direction and y direction through the position coordinates of the target point in consecutive m frames and the time interval between adjacent frames, thereby obtaining the kinematic rationality loss Lm; Perform weighted fusion through the object detection loss Ld, the tracking consistency loss Lt, and the kinematic rationality loss Lm to generate the video deviation score SC3 of the crane operation.

[0009] Further, the specific process of S5 is as follows: Establish a hierarchical early warning mechanism, compare the thresholds through the efficiency index score of the crane operation, the physical assessment report, and the deviation risk score, and output the device early warning log; Set the threshold Q1 of the efficiency index score SC1 of the crane operation. When the efficiency index score SC1 is lower than the threshold Q1, re-plan the crane path trajectory, take the crane operation path corresponding to the maximum value of the efficiency index score SC1 as the optimal path strategy, and mark it as the crane dynamic adjustment strategy and output it; Set the threshold Q2 of the physical action score SC2 of the crane operation, evaluate the physical action state of the crane operation through threshold comparison, and generate corresponding early warning instructions to prompt the surrounding system to perform corresponding crane operation management; Set the threshold Q3 of the video deviation score SC3 of the crane operation, evaluate the virtual-real deviation degree of the crane through threshold comparison, and give corresponding early warning prompts to the back-end technical personnel; Furthermore, automatically generate the abnormal event traceability report of the port crane: Integrate the historical operation data of the port data center and the device early warning log into the abnormal event traceability report, and push it to the preventive equipment collaborative management platform for closed-loop processing.

[0010] A smart port crane operation monitoring system includes a data acquisition unit, an edge computing node, a cloud processor, and an early warning monitoring unit. Among them, the data acquisition unit, the edge computing node, the cloud processor, and the early warning monitoring unit are communicatively connected, and this system applies the above-mentioned smart port crane operation monitoring method; The data acquisition unit is used to collect multi-source heterogeneous data of crane operations in real time and build a dynamic twin data pool; establish a communication connection between the data acquisition unit and the data center of the port digital control platform to collect multi-source heterogeneous data of crane operations in real time; The edge computing node is used to preprocess multi-source heterogeneous data, and through the spatio-temporal alignment algorithm, uniformly map the physical coordinates of cranes, containers, transport vehicles and port infrastructure in the real environment to the world coordinate system to generate a high-precision dynamic digital twin model; The cloud processor is used to analyze multi-source heterogeneous data: the cloud processor includes a path adjustment module, an operation control module and a video comparison module; the path adjustment module is used to intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time; the operation control module is used to build a digital twin model and embed a multi-modal physical engine; the video comparison module is used to deeply analyze the video stream and perform virtual-real comparison; The early warning monitoring unit is used to establish a hierarchical early warning mechanism for automatic abnormal processing.

[0011] In summary, due to the adoption of the above technical solutions, the beneficial effects of the present invention are: The present invention establishes a dynamic twin data pool architecture through deep fusion and efficient governance of multi-source data, adopts sliding window adaptive standardization to solve the problems of sensor data noise and dynamic load fluctuation, improves data cleaning efficiency, and realizes spatio-temporal alignment across data types through multi-modal coding to improve the accuracy of feature fusion; The present invention intelligently analyzes the dynamic twin data pool and optimizes the crane operation path in real time, thereby dynamically adjusting the crane operation path trajectory to improve work efficiency; through the digital twin model and integrating a multi-modal physical engine, evaluate the physical state deviation of crane operations and correct the operation plan in real time to improve operation accuracy; through deeply analyzing the video stream and performing virtual-real comparison, monitor the video deviation predicted by the model, thereby improving the simulation accuracy; This solution systematically solves the problems of data islands, decision-making lag, and incomplete risk warning in traditional port crane operations through multi-source data fusion, high-precision digital twins, intelligent optimization algorithms and a closed-loop early warning mechanism, realizes full-process digital control, and promotes the intelligent upgrade of ports. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 Shows a connection diagram of the system modules of the present invention; Figure 2 Shows a flowchart of the method of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0013] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0014] Example 1 like Figure 1 - Figure 2 As shown, a smart port crane operation monitoring system includes a data acquisition unit, an edge computing node, a cloud processor and an early warning monitoring unit, wherein the data acquisition unit, the edge computing node, the cloud processor and the early warning monitoring unit are communicatively connected; The data acquisition unit is used to collect multi-source heterogeneous data of crane operations in real time and build a dynamic twin data pool; establish a communication connection between the data acquisition unit and the data center of the port digital management and control platform to collect multi-source heterogeneous data of crane operations in real time; Edge computing nodes are used to pre-process multi-source heterogeneous data and uniformly map the physical coordinates of cranes, containers, transport vehicles and port infrastructure in the real environment to the world coordinate system through a spatiotemporal alignment algorithm to generate a high-precision dynamic digital twin model. The cloud processor is used to analyze multi-source heterogeneous data: the cloud processor includes a path adjustment module, an operation control module, and a video comparison module; the path adjustment module is used to intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time; the operation control module is used to build a digital twin model and embed a multimodal physical engine; the video comparison module is used to deeply analyze the video stream and compare virtuality with reality; The early warning monitoring unit is used to establish a hierarchical early warning mechanism for automatic abnormal processing.

[0015] The working steps are as follows: S1, real-time collection of multi-source heterogeneous data of crane operations and construction of a dynamic twin data pool: through the data center of the port digital management and control platform, real-time collection of multi-source heterogeneous data of crane operations; The business database of each business system is synchronized to the intermediate database. The data center will uniformly extract data from the intermediate database, and the analyzed data will be returned to each business system. Each business system will transform its system based on the feedback results. The list of business systems involved includes: business and financial integration system, equipment and energy system, integrated pipe network system, production and business integration system, container business system, roll-on / roll-off logistics production system, production cost accounting system, comprehensive safety management, logistics information network, Internet of Things collection, etc., to connect the production, operation, equipment and facilities, safety and environmental protection of the port, and realize data sharing, business collaboration and refined management of port operations.

[0016] S11, multi-source heterogeneous data includes sensor data, device log data, and video stream data; The sensor data includes coordinate parameters, angle parameters and weight parameters: The sensors include position sensors, inclination sensors and load sensors. The position sensors include GPS and laser radar. The position sensors collect coordinate parameters (x, y, z), the inclination sensors collect angle parameters (θ, φ), and the load sensors collect weight parameters F. Equipment log data includes crane operating speed v, motor current I and operation time T; The video stream data includes the crane visual parameters collected by the camera, including the gripper posture and cargo status; S12, through the time-space alignment algorithm, the physical coordinates of cranes, containers, transport vehicles and port infrastructure in the real environment are uniformly mapped to the world coordinate system to generate a high-precision dynamic digital twin model; The original data is cleaned, denoised, missing values filled and standardized through the wavelet transform model, and a sliding window is set for dynamic adaptive standardization to unify the timestamp and spatial coordinate system. The specific process is as follows: Mark the current time as t; mark the sliding window length as W, which is the number of data points contained in the window; mark the index variable of the time as i, which is used to traverse all data points in the sliding window; mark the original data value at the i-th time as ; Thus, the local mean of the sliding window data corresponding to time t is obtained : ; Then obtain the local standard deviation of the sliding window data corresponding to time t : ; The local mean of the sliding window data corresponding to time t and the local standard deviation , standardize the original data value at time t, and mark the standardized data value as : ; in, is the smoothing factor and Greater than 0, used to avoid The denominator of is 0, thus eliminating the impact of sudden outliers to adapt to dynamic load changes in crane operations; S13, divides data types into real-time streaming data layer, historical data layer and metadata layer through distributed storage technology, and builds a dynamic twin data pool architecture; Encode the preprocessed multi-source heterogeneous data through a deep learning model and then map it to a unified vector space. Among them, sensor data is encoded through an LSTM long short-term memory network model, video stream data is encoded through a CNN convolutional neural network model, and device log data is encoded through a Transformer self-attention mechanism neural network model; S14. Then, define a timestamp synchronization function T(ts, tv) to align the sensor data timestamp ts with the video stream data timestamp tv, and then align the sensor, video stream, and log through the historical time nodes of the device log data for multi-source data spatio-temporal alignment; Through dynamic adaptive processing and multi-modal deep fusion, the accuracy and efficiency of port crane data acquisition and twin modeling are significantly improved, so that the mathematical model has both theoretical rigor and engineering practicality.

[0017] S2. Intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time: Analyze the port production operation deviation through multi-source heterogeneous data, obtain the efficiency index score of the crane operation, and output the crane dynamic adjustment strategy. The specific process is as follows: Intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time; Analyze the port production operation deviation through multi-source heterogeneous data and obtain the efficiency index score of the crane operation; S21. Extract the task completion time, empty running rate, unit cargo energy consumption, emergency stop times, and load overrun rate of the crane from the device log; Mark the number of indicators of the device log data as n0; Mark the reference value of any indicator p of the device log data as Dp0, mark the actual value of indicator p at time t as Dpt, and obtain the deviation amplitude value of indicator p through the difference between the reference value and the actual value of indicator p : ; S22. By calculating the deviation amplitude values of n0 indicators And performing maximum-minimum normalization, obtain the normalized value Sp of indicator p: , where and Are the maximum and minimum values of n0 deviation amplitude values respectively; S23. Then, by calculating the information entropy Hp of indicator p, obtain the weight coefficient of indicator p ; Among them, ; , Hq is the information entropy Hq of any indicator q among n0 indicators, and P is the sequence set of n0 indicators; When the information entropy Hp of indicator p is smaller, the discrimination degree of indicator p is higher, and the weight coefficient The larger; S24. Through the deviation magnitude value of index p and the weight coefficient perform weighted fusion to calculate the efficiency index score SC1 of crane operation: .

[0018] S3. Build a digital twin model and embed a multimodal physics engine: thereby synchronously simulating the mechanical interaction process of crane operation in the real environment, real-time detecting the physical parameters of the crane grab, and analyzing and generating a physical evaluation report of crane operation, so as to real-time correct the operation plan and push warning instructions to the surrounding systems. The specific process is as follows: Build a digital twin model and embed a multimodal physics engine, synchronously simulate the mechanical interaction process of crane operation in the real environment, real-time detect the physical parameters of the crane grab through sensors. The physical parameters include the acceleration of the crane grab, load stress, container displacement deviation, and vehicle positioning error, and analyze and calculate the physical action score SC2 of crane operation: ; Among them, the number of indicators marking the physical parameters is n1; the reference value of any indicator r marking the physical parameters is Dr0, mark the actual value of indicator r at time t as Drt, and obtain the deviation magnitude value of indicator r through the difference between the reference value and the actual value of indicator r , and then obtain the standardized value Sr, information entropy Hr, and weight coefficient of indicator r , and thus obtain the physical action score SC2 of crane operation.

[0019] S4. Deeply analyze the video stream and conduct virtual-real comparison: Through the monitoring video stream of port crane operation and the simulated video stream of the digital twin model, extract the key action features of the crane and conduct spatio-temporal alignment comparison to generate the video deviation score of crane operation. The specific process is as follows: Compare and analyze the monitoring video stream of port crane operation and the simulated video stream of the digital twin model; Through frame-level analysis of the monitoring video stream, extract the key action features of the crane and conduct spatio-temporal alignment comparison with the digital twin simulated video stream; S41. Divide the predicted categories of the key action features of the crane by the digital twin model video stream into n2, then the predicted category probability distribution of the digital twin model is : , and ; Encode the true category label of the monitoring video stream of crane operation as : , define the true category position y as 1, otherwise y is 0; Label the prediction bounding box of the digital twin simulation video stream as B: , where the center point coordinates of the prediction bounding box are (b1, b2), the length is b3, and the width is b4; Label the ground truth bounding box of the crane operation monitoring video stream as C: , where the center point coordinates of the ground truth bounding box are (c1, c2), the length is c3, and the width is c4; S42. Calculate the cross entropy of the virtual-real comparison of the crane action features through spatio-temporal alignment comparison and the complete intersection over union CIoU to obtain the object detection loss Ld; S4201. Cross entropy ; When the prediction probability is closer to the ground truth label , the value of the cross entropy is smaller; when the correct class is predicted, and at this time , the cross entropy is 0; when the prediction is incorrect, then is close to 0, is negative and the higher the absolute value, the larger the cross entropy will be; S4202. Complete intersection over union CIoU: ; where IoU is the intersection over union and ; is the distance between the center points of the two bounding boxes and ; is the diagonal length of the smallest bounding rectangle that contains the two bounding boxes; ; is the consistency measurement factor of the aspect ratio, , when is higher, it means less consistent; is 's weight coefficient, ; S4203. Object detection loss Ld: ; S43. Mark the position coordinate sequence of the target point in consecutive m frames as E, and then calculate the difference in the change between adjacent frames. Mark the displacements of any pair of adjacent frames in the x direction and y direction as , ; ; Thus, obtain the tracking consistency loss Lt: ; The smoother the trajectory of the target point is, the smaller the change in position between adjacent frames is, and the smaller the tracking consistency loss Lt is. S44. Calculate the velocity and acceleration in the x-direction and y-direction for any pair of adjacent frames based on the position coordinates of the target point in consecutive m frames and the time interval between adjacent frames. Mark the measured velocity and acceleration in the x-direction as Vx1 and Ax1 respectively; mark the measured velocity and acceleration in the y-direction as Vy1 and Ay1 respectively. Set the reasonable velocity and reasonable acceleration in the x-direction as Vx2 and Ax2 respectively; set the reasonable velocity and reasonable acceleration in the y-direction as Vy2 and Ay2 respectively, and the specific values are preset through experiments and experience. Thus, obtain the kinematic rationality loss Lm: ; S45. Perform weighted fusion on the target detection loss Ld, tracking consistency loss Lt, and kinematic rationality loss Lm to generate the video deviation score SC3 for crane operation: ; Among them, 、 、 are the weight factor coefficients of the target detection loss Ld, tracking consistency loss Lt, and kinematic rationality loss Lm respectively, and the weight factor coefficients are preset through calculation with a large amount of experimental data.

[0020] S5. Establish a hierarchical early warning mechanism for automatic anomaly handling: Compare the thresholds of the efficiency index score, physical evaluation report, and deviation risk score of crane operation, output the device early warning log, and thus automatically generate the traceability report of abnormal events of the port crane for closed-loop processing. The specific process is as follows: S51. Set the threshold Q1 of the efficiency index score SC1 of crane operation. When the efficiency index score SC1 is lower than the threshold Q1, re-plan the crane path trajectory, take the crane operation path corresponding to the maximum value of the efficiency index score SC1 as the optimal path strategy, and mark it as the crane dynamic adjustment strategy and output it. The crane dynamic adjustment strategy includes dynamically adjusting the crane grabbing sequence, moving trajectory, and collaborative operation strategy. S52. Set the threshold Q2 for the physical action score SC2 of the crane operation. Evaluate the physical action status of the crane operation by comparing with the threshold, and generate corresponding warning instructions, so as to prompt the surrounding systems to perform corresponding crane operation management; when the physical action score SC2 is lower than the threshold Q2, re-plan the crane grasping path, take the crane operation strategy corresponding to the maximum value of the efficiency index score SC1 as the optimal operation plan, and integrate and generate a physical evaluation report of the crane operation, so as to correct the operation plan in real time and push warning instructions to the surrounding systems; S53. Set the threshold Q3 for the video deviation score SC3 of the crane operation. Evaluate the virtual-real deviation degree of the crane by comparing with the threshold. When the video deviation score SC3 is higher, it is evaluated that the simulation prediction effect of the digital twin model is not good, and corresponding warning prompts are given to the back-end technicians; Furthermore, automatically generate an abnormal event traceability report for the port crane: Integrate the historical operation data and equipment warning logs of the port data center into an abnormal event traceability report, and push it to the preventive equipment collaborative management platform for closed-loop processing; Realize the full-process digital monitoring and intelligent control of the smart port crane operation, and improve the port operation efficiency and safety.

[0021] In summary, the present invention deeply integrates and efficiently manages multi-source data, establishes a dynamic twin data pool architecture, adopts sliding window adaptive standardization to solve the problems of sensor data noise and dynamic load fluctuation, improves the data cleaning efficiency, and realizes spatio-temporal alignment across data types through multi-modal coding, improving the feature fusion accuracy; The present invention dynamically adjusts the crane operation path trajectory by intelligently analyzing the dynamic twin data pool and real-time optimizing the crane operation path, thereby improving the work efficiency; evaluates the physical state deviation of the crane operation through the digital twin model and integrates a multi-modal physical engine, and corrects the operation plan in real time to improve the operation accuracy; monitors the model prediction video deviation by deeply analyzing the video stream and performing virtual-real comparison, thereby improving the simulation accuracy; This solution systematically solves the problems of data islands, decision-making lag, and incomplete risk warning in traditional port crane operations through multi-source data fusion, high-precision digital twins, intelligent optimization algorithms, and closed-loop warning mechanisms, realizes full-process digital control, and promotes the intelligent upgrade of ports.

[0022] The above embodiments can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, the above embodiments can be implemented in whole or in part in the form of a computer program product. Those of ordinary skill in the art will appreciate that the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are executed in hardware or software depends on the specific application and design constraints of the technical solution.

[0023] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or they may be distributed to multiple network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0024] The above data processing is to remove the dimension and take its numerical calculation. The setting of the size of the interval and threshold is for the convenience of comparison. Regarding the size of the threshold, it depends on the amount of sample data and the number of base numbers set by those skilled in the art for each group of sample data, as long as it does not affect the proportional relationship between the parameters and the quantized values. The preset parameters are set by those skilled in the art according to the actual situation.

[0025] The above is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art within the technical scope disclosed by the present invention, according to the technical solution and inventive concept of the present invention, makes equivalent replacements or changes, and should be covered by the protection scope of the present invention.

Claims

1. A method for monitoring the operation of a smart port crane, characterized in that, Including the following steps: S1. Collect multi-source heterogeneous data of crane operations in real time and build a dynamic twin data pool: Through the data middle platform of the port digital control and management platform, collect multi-source heterogeneous data of crane operations in real time; S2. Intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time: Analyze the deviation of port production operations through multi-source heterogeneous data, obtain the efficiency index score of crane operations, and output the dynamic adjustment strategy of the crane; S3. Build a digital twin model and embed a multi-modal physical engine: By synchronously simulating the mechanical interaction process of crane operations in the real environment, detect the physical parameters of the crane grab in real time, and analyze and generate a physical evaluation report of crane operations, so as to correct the operation plan in real time and push early warning instructions to the surrounding systems; S4. Deeply analyze the video stream and conduct virtual-real comparison: Through the monitoring video stream of port crane operations and the simulated video stream of the digital twin model, extract the key action features of the crane and conduct spatio-temporal alignment comparison to generate a video deviation score of crane operations; S5. Establish a hierarchical early warning mechanism for automatic abnormal handling: Compare the thresholds through the efficiency index score, physical evaluation report and deviation risk score of crane operations, output the equipment early warning log, and thus automatically generate a traceability report of abnormal events of port cranes for closed-loop processing.

2. A monitoring method for intelligent port crane operations according to claim 1, characterized in that: The multi-source heterogeneous data includes sensor data, equipment log data and video stream data; Unify the physical coordinates of cranes, containers, transport vehicles and port infrastructure in the real environment to the world coordinate system through a spatio-temporal alignment algorithm to generate a high-precision dynamic digital twin model; Clean and denoise the original data, fill in missing values and standardize it through a wavelet transform model, and set a sliding window for dynamic adaptive standardization, so as to unify the time stamp and space coordinate system. The specific process is as follows: Mark the current moment as t; mark the length of the sliding window as W, where the length of the sliding window is the number of data points contained in the window; mark the index variable of the moment as i, which is used to traverse all the data points in the sliding window; mark the original data value at the i-th moment as ; thus obtaining the local mean of the sliding window data corresponding to the moment t ; Furthermore, obtain the local standard deviation of the sliding window data corresponding to time t ; The local mean of the sliding window data corresponding to time t and the local standard deviation , standardize the original data value at time t, and mark the standardized data value as ; Divide the data types into a real-time stream data layer, a historical data layer and a metadata layer through a distributed storage technology to build a dynamic twin data pool architecture; Align the multi-source data of sensors, video streams and logs in space and time by defining a time stamp synchronization function T(ts, tv) and the historical time nodes of equipment log data.

3. The intelligent port crane operation monitoring method according to claim 2, characterized in that: The specific process of S2 is as follows: Extract indicators through equipment log data, including the task completion time, empty running rate, unit cargo energy consumption, emergency stop times and load overrun rate of the crane; Mark the number of indicators of equipment log data as n0; The reference value of any index p of the marked device log data is Dp0, the actual value of index p at time t is marked as Dpt, and the deviation amplitude value of index p is obtained through the difference between the reference value and the actual value of index p ; By measuring the deviation amplitude values of n0 indicators And performing maximum-minimum normalization to obtain the normalized value Sp of index p; Then, by calculating the information entropy \(H_p\) of the indicator \(p\), the weight coefficient of the indicator \(p\) is obtained ; The deviation magnitude value of index p and the weight coefficient are weighted and fused to calculate the efficiency index score SC1 of crane operation.

4. A method for monitoring the operation of a smart port crane according to claim 3, characterized in that: The specific process of S3 is as follows: The physical parameters include the acceleration of the crane grab, load stress, container displacement deviation and vehicle positioning error. Mark the number of indicators of physical parameters as n1; Mark the reference value of any index r of the physical parameter as Dr0, mark the actual value of the index r at time t as Drt, and obtain the deviation amplitude value of the index r through the difference between the reference value and the actual value of the index r , and then obtain the standardized value Sr, information entropy Hr and weight coefficient of the index r ; By the deviation magnitude value of the index r and the weight coefficient perform weighted fusion to calculate the physical action score SC2 of the crane operation.

5. A method for monitoring the operation of a smart port crane according to claim 4, characterized in that: The specific process of S4 is as follows: Through frame-level parsing of the monitoring video stream, extract the key action features of the crane and conduct spatio-temporal alignment comparison with the digital twin simulation video stream; If the number of prediction categories of the digital twin model video stream for the key action features of the crane is divided into n2, then the probability distribution of the prediction categories of the digital twin model is ; Encode the true category labels of the monitoring video stream of the crane operation as ; Mark the prediction box of the digital twin simulation video stream as B: , where the center point coordinates of the prediction box are (b1, b2), the length is b3, and the width is b4; Mark the ground truth bounding boxes of the crane operation monitoring video stream as C: , where the center point coordinates of the ground truth bounding box are (c1, c2), the length is c3, and the width is c4; Calculate the cross entropy of the virtual-real contrast of the crane action characteristics through spatio-temporal alignment contrast and the complete intersection over union CIoU to obtain the object detection loss Ld; Mark the position coordinate sequence of the target point in consecutive m frames as E, and then calculate the change difference between adjacent frames. Mark the displacements of any pair of adjacent frames in the x-direction and y-direction as , respectively, so as to obtain the tracking consistency loss Lt; Calculate the speed and acceleration of any pair of adjacent frames in the x direction and y direction through the position coordinates of the target point in consecutive m frames and the time interval between adjacent frames, so as to obtain the kinematic rationality loss Lm; The video deviation score SC3 of the crane operation is generated by weighted fusion of the object detection loss Ld, the tracking consistency loss Lt, and the kinematic rationality loss Lm.

6. The operation monitoring method of an intelligent port crane according to claim 5, wherein: The specific process of S5 is as follows: A hierarchical early warning mechanism is established. By comparing the threshold values of the efficiency index score of the crane operation, the physical assessment report, and the deviation risk score, the equipment early warning log is output. Set the threshold value Q1 of the efficiency index score SC1 of the crane operation. When the efficiency index score SC1 is lower than the threshold value Q1, the crane path trajectory is replanned. The crane operation path corresponding to the maximum value of the efficiency index score SC1 is used as the optimal path strategy, which is marked as the crane dynamic adjustment strategy and output. Set the threshold value Q2 of the physical action score SC2 of the crane operation. By comparing the threshold values, the physical action state of the crane operation is evaluated, and corresponding early warning instructions are generated to prompt the surrounding systems to perform corresponding crane operation management. Set the threshold value Q3 of the video deviation score SC3 of the crane operation. By comparing the threshold values, the virtual-real deviation degree of the crane is evaluated, so as to give corresponding early warning prompts to the back-end technicians. Furthermore, an abnormal event traceability report of the port crane is automatically generated: The historical operation data of the port data middle platform and the equipment early warning log are integrated into an abnormal event traceability report and pushed to the preventive equipment collaborative management platform for closed-loop processing.

7. An operation monitoring system for a smart port crane, characterized in that: It includes a data acquisition unit, an edge computing node, a cloud processor, and an early warning monitoring unit. Among them, the data acquisition unit, the edge computing node, the cloud processor, and the early warning monitoring unit are communicatively connected. This system applies the intelligent port crane operation monitoring method described in any one of claims 1-6 above. The data acquisition unit is used to collect multi-source heterogeneous data of the crane operation in real time and construct a dynamic twin data pool; a communication connection is established between the data acquisition unit and the data middle platform of the port digital control platform to collect multi-source heterogeneous data of the crane operation in real time. The edge computing node is used to preprocess multi-source heterogeneous data. By using the spatio-temporal alignment algorithm, the physical coordinates of the crane, container, transport vehicle, and port infrastructure in the real environment are uniformly mapped to the world coordinate system to generate a high-precision dynamic digital twin model. The cloud processor is used to analyze multi-source heterogeneous data: The cloud processor includes a path adjustment module, an operation control module, and a video comparison module; the path adjustment module is used to intelligently analyze the dynamic twin data pool and optimize the crane operation path in real time; the operation control module is used to construct a digital twin model and embed a multi-modal physical engine; the video comparison module is used to deeply analyze the video stream and perform virtual-real comparison. The early warning monitoring unit is used to establish a hierarchical early warning mechanism for automatic abnormal processing.

Images