Intelligent Diagnosis and Early Warning Method and System for Container Loading and Unloading Equipment Based on Multi-Source Data Fusion

Abstract

This application relates to the field of artificial intelligence applications, specifically to a method and system for intelligent diagnosis and early warning of container loading and unloading equipment based on multi-source data fusion. The method includes: real-time acquisition of multi-source data from the equipment; preprocessing and preliminary fusion of the multi-source data; global temporal alignment and deep fusion of the preliminary fused feature data; extraction of multi-dimensional features using a deep learning network incorporating an attention mechanism; filtering and optimization using an improved sparrow search algorithm to generate a situation map; dynamic optimization of equipment scheduling strategies based on the situation map with the objectives of minimizing completion time and maximizing utilization; simultaneous input of feature data into a fault prediction model for real-time diagnostic evaluation; triggering graded early warning and safety protection based on the evaluation results; and feeding back the diagnostic conclusions to the scheduling system to achieve coordinated optimization of work plans and safety management. This realizes closed-loop management of real-time equipment status perception, dynamic optimization of scheduling strategies, and graded early warning of safety risks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence application technology, specifically to a method and system for intelligent diagnosis and early warning of container loading and unloading equipment based on multi-source data fusion. Background Technology

[0002] As a key hub in sea-rail intermodal transport, the operational efficiency of container terminals directly impacts the overall efficiency of the logistics network. At the terminal site, large port machinery such as quay cranes, rail-mounted gantry cranes, and automated guided vehicles work together to complete the loading, unloading, and transshipment of containers. However, current terminal equipment management and safety control generally suffer from the following problems: First, existing technologies primarily rely on periodic maintenance and manual inspections, making it difficult to monitor equipment health status in real time and provide early warnings before malfunctions occur. Sudden equipment failures not only disrupt operations but can also lead to safety accidents. Second, while various sensors and monitoring equipment, including vibration sensors, lidar, cameras, and positioning tags, are deployed at port operation sites, this data belongs to different systems and lacks an effective fusion mechanism, hindering a unified understanding of the overall situation at the operation site. Third, existing dispatching systems primarily focus on operational efficiency, failing to adequately consider equipment health status and personnel safety risks. When equipment malfunctions or personnel enter hazardous areas, the dispatching system cannot respond promptly, making it difficult to adjust work plans to mitigate risks. Finally, traditional safety protection relies on simple methods such as audible and visual alarms, lacking tiered early warning and coordinated control capabilities. It cannot implement differentiated protective measures based on risk levels, nor can it effectively feed diagnostic conclusions back to the dispatching system for collaborative optimization.

[0003] Therefore, there is an urgent need for an intelligent diagnostic and early warning method that can integrate multi-source sensing data, diagnose equipment status in real time, dynamically optimize scheduling strategies, and achieve hierarchical early warning linkage, in order to improve the operational safety and efficiency of container terminals. Summary of the Invention

[0004] To address the aforementioned issues, this invention provides a method and system for intelligent diagnosis and early warning of container loading and unloading equipment based on multi-source data fusion.

[0005] In a first aspect, the present invention provides a method for intelligent diagnosis and early warning of container loading and unloading equipment based on multi-source data fusion, comprising the following steps: S1. Real-time acquisition of operating status data of loading and unloading equipment, operational environment perception data based on radar and vision, personnel location data obtained by ultra-wideband positioning technology, and dispatch system business data. S2. Edge computing is used to preprocess and initially fuse the collected multi-source data; S3. The preliminary fusion feature data is globally aligned and deeply fused through the cloud platform, and a deep learning network model with an attention mechanism is used to extract multi-dimensional features reflecting equipment health status, operation efficiency and safety risks. An improved sparrow search algorithm is used to filter and optimize the extracted features to generate a situation map that represents the comprehensive situation of the operation site in real time. S4. Based on the situation map, dynamically optimize the equipment scheduling strategy with the goal of minimizing the completion time and maximizing the utilization rate; at the same time, input the feature data in the situation map into the pre-trained fault prediction model to perform real-time diagnosis and assessment of equipment health status and personnel safety risks. S5. Trigger graded early warnings and safety protection based on the diagnostic assessment results, and feed back the diagnostic conclusions to the scheduling system in real time to achieve coordinated optimization of work plans and safety management.

[0006] The system achieves deep integration and global situational awareness of multi-dimensional data on equipment, environment, personnel, and scheduling in container loading and unloading operations, breaking down data silos and fragmented technology applications in each link. At the same time, it deeply couples equipment health diagnosis, personnel safety risk assessment, and operation scheduling optimization, improving equipment utilization and shortening operation completion time while ensuring operational safety, thus achieving dual optimization of port loading and unloading operation safety management and operational efficiency.

[0007] As a further limitation of the technical solution of the present invention, in S2, the step of preprocessing and initially fusing the collected multi-source data using edge computing includes: S21. Edge computing nodes perform real-time cleaning of incoming multi-source data streams, removing abnormal noise points and erroneous data; for visual image data, a histogram equalization algorithm is used for illumination correction and contrast enhancement; for lidar point cloud data, a statistical filtering algorithm is used for downsampling and outlier removal; for sensor time-series data, a sliding window filtering algorithm is used for smoothing. S22. Based on the high-precision timestamps carried by each sensor data, time alignment is achieved through hardware synchronization signals to ensure that data from different sources correspond to the same physical event under the same time reference; at the same time, a spatial coordinate transformation relationship is established to uniformly transform the lidar point cloud, visual image, and UWB positioning data into the preset global coordinate system. S23. Through a deep learning model deployed at edge nodes, real-time feature extraction is performed on spatiotemporally aligned multi-source data. The features include vibration and load features reflecting the health status of equipment, target category and location features reflecting the working environment, and trajectory and posture features reflecting personnel activities. S24. Perform cross-modal association matching on the extracted visual features, point cloud features, radar features and UWB positioning features. Based on the spatial position consistency and motion trajectory similarity of the target, construct the feature mapping relationship of the same physical target under different perception modalities to form a preliminary fused feature vector.

[0008] By performing real-time processing at edge nodes close to the data source, the amount of data transmitted to the cloud platform is significantly reduced, saving network bandwidth resources; the application of algorithms such as sliding window filtering and statistical filtering effectively eliminates noise and outliers, improving data quality; spatiotemporal alignment and coordinate system transformation ensure that the same physical event perceived by different sensors can be accurately correlated.

[0009] As a further limitation of the technical solution of the present invention, step S24 includes: S241. Based on the target features of each modality extracted in S23, spatial association is performed using UWB positioning tags as anchor points; the Euclidean distance between visual targets, lidar targets, millimeter-wave radar targets and UWB tags is calculated, and those with distances less than the first threshold are selected as candidate association pairs; for areas without UWB coverage, the spatial distance between each pair of targets in each modality is calculated, and the Hungarian algorithm is used to solve for the optimal match and generate candidate association pairs. S242. Statistically analyze the spatial position deviation of candidate association pairs in consecutive frames, calculate the root mean square error and variance, and remove association pairs whose position fluctuations are greater than the second threshold. S243. The dynamic time warping algorithm is used to calculate the trajectory shape similarity of the two targets in the candidate association pair in the past N frames. At the same time, the velocity vector angle and acceleration difference are calculated. When the trajectory similarity is higher than the third threshold and the motion features meet the preset range, the association is confirmed to be valid. S244. A learning-based offset estimation network is used to perform sub-pixel alignment between millimeter-wave radar points and visual targets; an iterative nearest-point algorithm is used to perform fine registration between the lidar point cloud and the 3D bounding box of the visual target. S245. After verification and alignment, the modal attribute information is concatenated into vectors to generate a preliminary fusion feature vector containing target identifier, three-dimensional position, motion state, geometric dimensions and appearance features.

[0010] By eliminating correlation pairs with large positional fluctuations, erroneous matching caused by false detection in a single frame is effectively avoided; the dynamic time warping algorithm can accurately measure the similarity of target trajectories of different modalities, solving the trajectory alignment problem caused by different sampling frequencies.

[0011] As a further limitation of the technical solution of the present invention, step S3 includes: S31. The cloud platform receives the preliminary fusion feature data uploaded by the edge nodes, and uses an interpolation method based on high-precision timestamps to resample the asynchronously sampled multi-source data to a unified time reference, forming a time-aligned multimodal data sequence, and reconstructing it into a three-dimensional tensor form. S32. Construct a deep learning network model that incorporates a hybrid attention mechanism, and reflect three types of multidimensional features—equipment health status, operational efficiency, and safety risks—based on three-dimensional tensor outputs. S33. An improved sparrow search algorithm incorporating the Levy flight mechanism is used to filter and optimize the extracted multidimensional features. The fitness function is used to evaluate the classification accuracy and feature dimension of the feature subset, and the optimal features and weight coefficients are output. S34. Organize the selected optimal features according to three dimensions, map them to a unified feature space using a multilayer perceptron, and construct a graph neural network to generate a situation map representing the overall situation of the work site. S35. Repeat steps S31 to S34 by sliding the preset time window step size to achieve real-time dynamic updating of the situation map.

[0012] The hybrid attention mechanism can adaptively enhance the response of key features and suppress redundant information, thereby improving the targeting and effectiveness of feature extraction. The improved sparrow search algorithm expands the search space through the Levy flight mechanism, which can quickly select the optimal feature subset in the high-dimensional feature space and avoid feature redundancy.

[0013] As a further limitation of the technical solution of the present invention, in S33, an improved sparrow search algorithm incorporating the Levy flight mechanism is used for feature selection optimization, specifically including: S331. Initialize the sparrow population using binary encoding, with each individual representing a feature subset selection scheme; S332. Construct a fitness function that balances classification accuracy and feature dimension;

[0014] In the formula, For the first The classification accuracy of a subset of features selected for each individual sparrow on the validation set; For the first The number of features selected by each individual; The total number of features; and For the weighting coefficients, satisfying ; S333. Take the sparrows with the highest fitness M in the population as the discoverers, and introduce the Levy flight perturbation step size into the discoverer position update formula to expand the search space and avoid getting trapped in local optima. S334. Joiners update their positions following the discoverers, while vigilants randomly update their positions to guide population migration. S335. Use the Sigmoid function to convert consecutive position values ​​into binary values; S336. Calculate the fitness value of each sparrow individual after the update, and adopt the elite retention strategy to directly retain the optimal set number of individuals in the current population to the next generation, replacing the same number of the worst individuals. S337. Through iterative optimization, output the optimal feature subset and its weight coefficients.

[0015] Levy flight perturbation step size enables the algorithm to escape local optima, expanding the exploration range while maintaining fine search capabilities; the elite preservation strategy ensures that a subset of excellent features is inherited, accelerating algorithm convergence.

[0016] As a further limitation of the technical solution of the present invention, step S4 includes: S41. Construct an equipment-task allocation model based on the situation map, with the dual objectives of minimizing completion time and maximizing utilization, and take equipment health status and personnel location as constraints. S42. A multi-objective intelligent optimization algorithm is used to solve the scheduling model. The real-time health status of the devices in the situation map is used as the reference information for the initial population of the algorithm. The optimal device-task allocation scheme and operation sequence are output through iterative optimization. S43. Input the equipment health status feature sequence in the situation map into a pre-trained fault prediction model based on attention mechanism and long short-term memory network, and output the remaining life of the equipment and the probability of failure. S44. Input personnel location, equipment status, and work area environmental information into a pre-trained graph neural network safety risk assessment model, and output the personnel safety risk level. S45. Integrate the allocation plan, fault prediction results and safety assessment results to generate a comprehensive decision package, issue it for execution and provide feedback to update the situation map.

[0017] As a further limitation of the technical solution of the present invention, step S5 includes: S51. Based on the diagnostic assessment results, the early warning levels are divided into Level 1, Level 2, and Level 3; and Level 1 is mapped to high risk, Level 2 to medium risk, and Level 3 to low risk. S52. Trigger differentiated early warning responses based on the early warning level. Level 1 early warning triggers emergency shutdown of equipment and audible and visual alarms exceeding the set decibel level. Level 2 early warning triggers deceleration of equipment and audible and visual alarms. Level 3 early warning triggers vibration alerts on smart bracelets. S53. Trigger corresponding protective measures according to the warning type, including emergency braking of equipment, personnel evacuation guidance, and warning of dangerous areas; S54. Real-time feedback of equipment failure prediction, personnel risk level, early warning level and protection implementation status to the equipment dispatching system; S55. The scheduling system dynamically adjusts task allocation based on feedback information, suspends task allocation for high-risk equipment, reduces the priority of medium-risk equipment, and re-plans routes for risk areas. S56. After the risk is eliminated, the warning can be manually or automatically lifted, and normal operation can be restored.

[0018] Differentiated early warning responses are triggered based on risk levels, avoiding the impact of excessive protection on operational efficiency and ensuring timely intervention in high-risk scenarios. Strategies such as suspending task allocation for high-risk equipment, reducing the priority of medium-risk equipment, and replanning routes for risk areas achieve a dynamic balance between safety control and operation scheduling. The operation status is automatically restored after the early warning is lifted, minimizing the impact of safety incidents on the continuity of operations.

[0019] Secondly, the technical solution of the present invention also provides an intelligent diagnosis and early warning system for container loading and unloading equipment based on multi-source data fusion, comprising: The data acquisition layer is used to collect real-time data on the operating status of loading and unloading equipment, the perception data of the working environment, personnel location data, and scheduling business data. The edge computing layer, connected to the data acquisition layer, is used to preprocess multi-source data, perform spatiotemporal alignment, feature extraction, and cross-modal association matching, generate preliminary fused feature vectors, and trigger local emergency response in case of emergency. The cloud platform layer, connected to the edge computing layer, is used to perform deep fusion analysis on the preliminary fusion feature data, generate a situation map that represents the comprehensive situation of the work site in real time, and diagnose and assess the health status of equipment and personnel safety risks. The scheduling decision layer, connected to the cloud platform layer, is used to dynamically optimize equipment scheduling strategies based on the situation map and generate a comprehensive decision package containing scheduling instructions, equipment early warning information, and personnel safety tips. The early warning and protection execution layer, connected to the cloud platform layer and the scheduling decision layer, is used to trigger graded early warnings and security protection based on diagnostic assessment results, and to execute scheduling instructions. The feedback closed-loop unit is connected to the cloud platform layer, scheduling decision layer and early warning and protection execution layer respectively, and is used to update the execution feedback data to the situation map in real time to form a continuous optimization closed loop.

[0020] As a further limitation of the technical solution of the present invention, the edge computing layer includes: The data preprocessing unit is used to perform real-time cleaning, filtering and noise reduction, image enhancement and outlier removal on the collected multi-source data. The spatiotemporal alignment unit is used to achieve time synchronization of multi-source data based on high-precision timestamps and to establish spatial coordinate transformation relationships to unify the data of each modality into the global coordinate system. The edge feature extraction unit deploys a lightweight deep learning model for real-time extraction of device status features, environmental target features, and personnel activity features. The cross-modal association matching unit is used to construct the feature mapping relationship of the same target under different perception modalities based on spatial location consistency and motion trajectory similarity, and generate a preliminary fused feature vector; The local emergency response unit is used to directly trigger local warnings or equipment emergency stop commands when an emergency danger is detected.

[0021] As a further limitation of the technical solution of the present invention, the cloud platform layer includes: The data receiving and temporal reconstruction unit is used to receive the preliminary fusion feature data uploaded by the edge nodes, resample the asynchronous data to a unified time base through interpolation methods, and reconstruct it into a three-dimensional tensor form. The deep feature extraction unit deploys a deep learning network model that incorporates a hybrid attention mechanism to extract three types of multidimensional feature vectors that reflect equipment health status, operational efficiency, and safety risks. The feature selection and optimization unit uses an improved sparrow search algorithm that incorporates the Levy flight mechanism to select and optimize the extracted multidimensional features, and outputs the optimal feature subset and weight coefficients. The situation map construction unit is used to construct a graph of the relationship between equipment, environment and personnel through a graph neural network after screening the best features, generate a situation map that represents the comprehensive situation of the work site in real time, and update it dynamically according to a preset time window; The fault prediction and diagnosis unit deploys a pre-trained fault prediction model based on attention mechanism and long short-term memory network to perform real-time diagnosis of equipment health status and output remaining life and failure probability. The safety risk assessment unit deploys a pre-trained graph neural network safety risk assessment model to perform real-time assessments of personnel safety risks and output risk levels.

[0022] As can be seen from the above technical solutions, this application has the following advantages: it integrates and processes four types of heterogeneous data—equipment operation status, environmental perception, personnel positioning, and business data—within the same framework, breaking through the limitations of data silos in traditional monitoring methods; it achieves real-time representation of the comprehensive situation at the work site by generating situation maps; and it feeds back the diagnostic and evaluation results to the scheduling system to form collaborative optimization, solving the problem of the disconnect between safety control and work plans in existing technologies, and improving the safety and efficiency of dock operations.

[0023] A deep learning network incorporating a hybrid attention mechanism is used to accurately extract three core features: equipment, efficiency, and safety, enhancing the responsiveness of key features. An improved sparrow search algorithm is employed to optimize feature selection, eliminating redundant features and improving feature representation efficiency. A graph neural network is used to construct the relationships between equipment, environment, and personnel, generating a dynamically updated situation map. This solution achieves deep fusion and efficient representation of multi-source feature data. The situation map can reflect the overall status of the work site in real time and comprehensively, providing an intuitive and accurate analytical foundation for subsequent scheduling optimization, fault diagnosis, and safety assessment. Simultaneously, a sliding window mechanism enables dynamic updates to the situation map, ensuring real-time perception of changes in the work site's status. Attached Figure Description

[0024] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a flowchart illustrating the method provided in an embodiment of the present invention.

[0026] Figure 2 A block diagram of a system provided in an embodiment of the present invention. Detailed Implementation

[0027] To make the purpose, features, and advantages of this application more apparent and understandable, specific embodiments and accompanying drawings will be used to clearly and completely describe the technical solution protected by this application. Obviously, the embodiments described below are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Unless otherwise defined, all technical and scientific terms used in this application have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used in this application and in the specification of this invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention.

[0029] like Figure 1 As shown, this embodiment of the invention provides an intelligent diagnosis and early warning method for container loading and unloading equipment based on multi-source data fusion, including the following steps: S1. Real-time acquisition of operating status data of loading and unloading equipment, operational environment perception data based on radar and vision, personnel location data obtained by ultra-wideband positioning technology, and dispatch system business data. S2. Edge computing is used to preprocess and initially fuse the collected multi-source data; S3. The preliminary fusion feature data is globally aligned and deeply fused through the cloud platform, and a deep learning network model with an attention mechanism is used to extract multi-dimensional features reflecting equipment health status, operation efficiency and safety risks. An improved sparrow search algorithm is used to filter and optimize the extracted features to generate a situation map that represents the comprehensive situation of the operation site in real time. S4. Based on the situation map, dynamically optimize the equipment scheduling strategy with the goal of minimizing the completion time and maximizing the utilization rate; at the same time, input the feature data in the situation map into the pre-trained fault prediction model to perform real-time diagnosis and assessment of equipment health status and personnel safety risks. S5. Trigger graded early warnings and safety protection based on the diagnostic assessment results, and feed back the diagnostic conclusions to the scheduling system in real time to achieve coordinated optimization of work plans and safety management.

[0030] In this embodiment of the invention, the specific steps of S1 include: S11. Vibration sensors, temperature sensors, and load sensors are deployed on key components such as motor bearings, gearboxes, and hoisting mechanisms of quay cranes, rail gantry cranes, and automated guided vehicles to collect vibration acceleration, temperature changes, and load data of the equipment in real time at a sampling frequency of not less than 10kHz. At the same time, the operating speed, acceleration, hoisting height, and current operating status parameters of the equipment are read in real time through the built-in encoder or PLC control system. S12. Deploy lidar and millimeter-wave radar in key operating areas of loading and unloading equipment. LiDAR is used to acquire three-dimensional point cloud data of the operating environment and accurately measure the distance and shape of containers and obstacles. Millimeter-wave radar is used to stably detect the distance and speed information of targets in low visibility environments such as rain, fog, and dust. At the same time, deploy high-definition vision cameras to collect video stream data of the operating area in real time. Through multi-view deployment, ensure coverage of the quay crane lifting radius, AGV travel path, and blind spots in the yard operation. S13. Integrate ultra-wideband positioning tags on workers' safety helmets or work badges, deploy ultra-wideband positioning base stations at key locations in port machinery equipment and work areas, and use time difference of arrival or two-way ranging positioning methods to obtain the precise location coordinates of personnel in real time at an update frequency of not less than 10Hz, with positioning accuracy controlled within ±0.3 meters; at the same time, deploy vehicle-mounted positioning tags on port machinery equipment to realize real-time calculation of the relative distance between equipment and personnel. S14. Establish a secure connection with the container terminal operating system or equipment scheduling system through the RESTful API interface, obtain access tokens using the OAuth 2.0 authentication mechanism, and subscribe to or periodically retrieve operation scheduling instructions, task allocation information, container operation sequence, and equipment task list data in real time or on a regular basis; the data interface content includes currently executing operation tasks, task queues to be assigned, and historical operation records of the equipment; S15. Add a high-precision timestamp based on a network time protocol or pulse synchronization signal to all collected data.

[0031] In this embodiment of the invention, an industrial-grade edge computing gateway is deployed near the control cabinet of large port machinery equipment such as quay cranes and rail-mounted gantry cranes, and an embedded edge computing unit is deployed within the AGV on-board control system. The edge computing node is configured with a GPU or NPU acceleration chip to support real-time data processing and lightweight inference operations. Correspondingly, in S2, the step of preprocessing and initially fusing the collected multi-source data using edge computing includes: S21. Edge computing nodes perform real-time cleaning of incoming multi-source data streams, removing abnormal noise points and erroneous data; for visual image data, a histogram equalization algorithm is used for illumination correction and contrast enhancement; for lidar point cloud data, a statistical filtering algorithm is used for downsampling and outlier removal; for sensor time-series data, a sliding window filtering algorithm is used for smoothing. S22. Based on the high-precision timestamps carried by each sensor data, time alignment is achieved through hardware synchronization signals to ensure that data from different sources correspond to the same physical event under the same time reference; at the same time, a spatial coordinate transformation relationship is established to uniformly transform the lidar point cloud, visual image, and UWB positioning data into the preset global coordinate system. S23. Through a deep learning model deployed at edge nodes, real-time feature extraction is performed on spatiotemporally aligned multi-source data. The features include vibration and load features reflecting the health status of equipment, target category and location features reflecting the working environment, and trajectory and posture features reflecting personnel activities. S24. Perform cross-modal association matching on the extracted visual features, point cloud features, radar features and UWB positioning features. Based on the spatial position consistency and motion trajectory similarity of the target, construct the feature mapping relationship of the same physical target under different perception modalities to form a preliminary fused feature vector.

[0032] It should be further explained that steps S24 include: S241. Based on the target features of each modality extracted in step S23, and using UWB positioning labels as high-confidence anchor points, generate cross-modal candidate association pairs. For targets wearing UWB tags, the center of the visual detection frame, the clustering center of the LiDAR, and the detection point of the millimeter-wave radar are spatially associated with the coordinates of the UWB tags. Specifically, the center of the visual detection frame is converted into three-dimensional spatial coordinates through a camera projection model, and the Euclidean distance is calculated between it and the coordinates of the UWB tags; the coordinates of the center point of the LiDAR target are directly compared with the coordinates of the UWB tags to calculate the distance; the millimeter-wave radar target is converted to global coordinates through the radar extrinsic matrix and then the distance is calculated between it and the coordinates of the UWB tags; target-tag pairs with a distance less than a first threshold are selected as candidate association pairs. For targets such as containers, AGVs, and quay cranes without UWB tag coverage, the nearest neighbor matching between multimodal targets is adopted; the spatial distance between each pair of visual targets, lidar targets, and millimeter-wave radar targets is calculated, a distance matrix is ​​constructed, and the Hungarian algorithm is used to solve for the optimal matching to generate candidate association pairs between cross-modal targets; S242. Perform preliminary screening on the candidate association pairs generated in S241 based on the spatial location consistency accumulated over multiple frames. For each candidate association pair, calculate the mean and variance of its spatial position deviation in M ​​consecutive frames; Calculate the root mean square error of the positional deviation. When the root mean square error is less than the second threshold and the variance of the positional deviation is less than the third threshold, it indicates that the association pair has stable spatial positional consistency at different time points, and the candidate association pair is retained. Candidate association pairs with significant spatial location fluctuations were eliminated. S243. For candidate association pairs that have passed the spatial location consistency screening, similarity verification is performed using the historical motion trajectory of the target. The motion trajectory of each target in the past N frames is modeled by Kalman filtering to obtain a smoothed trajectory sequence, including the three-dimensional position, motion velocity and acceleration of each frame; For two targets in a candidate association pair, their respective trajectory sequences in the past N frames are extracted, and the shape similarity between the two trajectories is calculated using the dynamic time warping algorithm. Simultaneously calculate the angle between the velocity vectors and the difference in acceleration between the two trajectories at corresponding time points, as an index of consistency of motion characteristics; When the trajectory shape similarity is higher than the fourth threshold, and the velocity vector angle is less than the fifth threshold and the acceleration difference is less than the sixth threshold, the candidate association pair is confirmed to be valid; otherwise, it is discarded. S244. For the association pairs that pass the motion trajectory similarity verification, a learning-based offset estimation network and an iterative nearest point algorithm are used for spatial alignment correction. A lightweight neural network is constructed, which uses visual ROI features and millimeter-wave radar point features as inputs to predict the two-dimensional or three-dimensional offset between the radar measurement point and the center of the visual target, so as to achieve sub-pixel-level precise alignment between the radar point and the visual target. For LiDAR point clouds and visual targets, the iterative nearest point algorithm is used to perform fine registration of the 3D bounding boxes of the point cloud and the visual target, iteratively optimize the 3D position, orientation and size of the target, and minimize the average distance between the point cloud and the model surface. Update the corrected spatial position and orientation information to the target's state parameters; S245. The aligned and corrected modal attribute information is fused to construct a multimodal feature representation of the same physical target; Attribute-level stitching: The appearance features of the visual modality, the geometric features of the lidar modality, the motion features of the millimeter-wave radar modality, and the identity features of the UWB modality are stitched together as vectors to form basic fusion features; Feature-level weighted fusion: Based on the confidence of each modal sensor in the scene, the stitched features are weighted and fused; among them, UWB positioning data is given higher weight in the near-field high-confidence area, LiDAR data is given higher weight in the long-distance and nighttime, and visual data is given higher weight under good lighting conditions. Output fused feature vector: Generate a unified dimensional feature vector containing the target's unique identifier, three-dimensional spatial location, instantaneous motion state (velocity, acceleration), motion trajectory prediction, geometric dimensions, appearance features, and identity information, as the output of step S24.

[0033] In some embodiments, step S3 includes: S31. The cloud platform receives the preliminary fusion feature data uploaded by the edge nodes. Based on the high-precision timestamps of each data stream, it uses linear interpolation or spline interpolation methods to resample the asynchronously sampled multi-source data to a unified time base, forming a time-aligned multimodal data sequence. At the same time, it reconstructs the data sequence within the continuous time window into a three-dimensional tensor form, which is used as the input for subsequent deep learning models. S32. Construct a deep learning network model that incorporates a hybrid attention mechanism to extract deep features from multi-source data after temporal alignment. A convolutional neural network is used to extract spatial features from each modality of data and generate a spatial feature map. A bidirectional long short-term memory network is used to extract the temporal dimension features of the data, capturing the dynamic changes in equipment status and operation process; A convolutional block attention module is introduced to calculate attention weights in both the channel and spatial dimensions, thereby enhancing the response to key fault features and safety risk features and suppressing noise and redundant information. The output includes a multi-dimensional feature vector reflecting the health status of the equipment, a multi-dimensional feature vector reflecting the operational efficiency, and a multi-dimensional feature vector reflecting the safety risks. S33. An improved sparrow search algorithm incorporating the Levy flight mechanism is used to filter and optimize the multidimensional features extracted in step S32. For the three types of multidimensional feature vectors reflecting equipment health status, operation efficiency, and safety risks output in step S32, differentiated fitness evaluation functions are constructed respectively. Among them, the fitness function of equipment health status features is mainly based on fault diagnosis accuracy and secondarily on feature dimension; the fitness function of operation efficiency features is mainly based on operation time prediction accuracy and secondarily on feature dimension; and the fitness function of safety risk features is mainly based on risk identification recall rate and secondarily on feature dimension. The sparrow population was divided into three subpopulations based on three types of characteristics. Each subpopulation independently selected its characteristics, and the subpopulations shared the optimal subset of characteristics through an information exchange mechanism. The Levy flight perturbation step size is introduced into the discoverer location update formula, and the Levy flight step size factor is adaptively adjusted according to the subpopulation type and iteration stage: For the equipment health status feature subpopulation, a smaller step size factor is used for fine search in the early stage of iteration, and the step size factor is gradually increased in the later stage of iteration to escape the local optimum. For the subpopulation with the characteristic of work efficiency, a fixed medium step size factor is used to maintain search stability; For the subpopulation of safety risk features, a larger step size factor is used in the early stage of iteration to quickly cover the search space, and the step size factor is gradually reduced in the later stage of iteration to finely locate key risk features. At each preset interval, the optimal feature subsets of the three subpopulations are cross-fused to generate candidate feature combinations. The deep learning network model in step S32 is used to verify the comprehensive performance of the candidate combinations, and the improved feature combinations are fed back to each subpopulation to participate in subsequent iterations. The process is iteratively optimized until convergence, and three optimal feature subsets and their corresponding weight coefficients are output for equipment health status diagnosis, operation efficiency assessment, and safety risk warning, respectively.

[0034] In the adaptive Levy flight discoverer location update, the step size factor of the Levy flight is adaptively adjusted according to the subpopulation type and iteration stage. The specific calculation formula is as follows: (1) Set the step size factor baseline value according to the subpopulation type. : For the subpopulation of equipment health status characteristics, take ; For the subpopulation with the characteristic of work efficiency, take ; For subpopulations with safety risk characteristics, take ; (2) Introduce iteration number decay factor and step size factor Dynamically changes with the number of iterations:

[0035] In the formula, This represents the current iteration number; This represents the maximum number of iterations. The attenuation coefficient has a value range of [2,5]. The minimum step size factor is set to 0.01. (3) Introduce a historical search direction correction factor and dynamically adjust the step size based on the number of consecutive unimproved algebras:

[0036] In the formula, For continuous unimproved algebras; To determine the maximum number of consecutive unimproved tolerance algebras, we take a value between 5 and 10. This is the direction correction factor, ranging from 0.2 to 0.5; (4) Calculation of Levy flight disturbance step size:

[0037] In the formula, , , , It is determined by the Levy distribution formula.

[0038] S34. The selected optimal feature subset is organized according to three dimensions: equipment health status, operation efficiency, and safety risk. A multilayer perceptron is used to map the features of different dimensions to a unified feature space. A graph neural network is used to construct a relationship graph between equipment, environment, and personnel. Equipment nodes, environment nodes, personnel nodes, and their edge weights represent the real-time situation of the work site. The node features and edge features output by the graph neural network are visualized and encoded to generate a situation map that represents the comprehensive situation of the work site in real time. S35. Repeat steps S31 to S34 according to the preset time window sliding step size to realize real-time dynamic updates of the situation map, providing continuous situational awareness input for subsequent scheduling optimization and diagnostic early warning.

[0039] It should be further noted that in S33, an improved sparrow search algorithm incorporating the Levy flight mechanism is used for feature selection optimization, specifically including: S331. Initialize the sparrow population using binary encoding, with each individual representing a feature subset selection scheme; Each feature in the multidimensional feature vector output by S32 is regarded as a candidate dimension. The sparrow population is initialized using binary encoding. Each individual sparrow corresponds to a feature subset selection scheme. The length of the individual position vector is the total number of features. Each dimension takes a value of 1 to indicate that the feature is selected, and a value of 0 indicates that the feature is not selected. The population size is set to 20-50, and the maximum number of iterations is set to 100-300. S332. Construct a fitness function that balances classification accuracy and feature dimension;

[0040] In the formula, For the first The classification accuracy of a subset of features selected for each individual sparrow on the validation set; For the first The number of features selected by each individual; The total number of features; and For the weighting coefficients, satisfying ; S333. Select the sparrows with the highest fitness ranking in the population as discoverers, and introduce the Levy flight perturbation step size into the discoverer position update formula to expand the search space and avoid getting trapped in local optima. The sparrows in the top 10%-20% of the population in terms of fitness are selected as discoverers, and their positions are updated according to the improved formula that incorporates the Levy flight mechanism:

[0041] In the formula, the Levy flight disturbance term , , , , It is determined by the Levy distribution formula. Sparrows are allowed to wander randomly in the feature space to explore new combinations of features. This is a warning value. This is the safety threshold.

[0042] S334. Joiners update their positions following the discoverers, while vigilants randomly update their positions to guide population migration. S335. Use the Sigmoid function to convert consecutive position values ​​into binary values; Boundary checks are performed on the updated sparrow positions to pull individuals that have exceeded the search space boundary back within it; the Sigmoid function is used to convert continuous position values ​​into binary values.

[0043] S336. Calculate the fitness value of each sparrow individual after the update, and adopt the elite retention strategy to directly retain the best 10% of individuals in the current population to the next generation, replacing the same number of the worst individuals. S337. Through iterative optimization, output the optimal feature subset and its weight coefficients.

[0044] In some embodiments, step S4 includes: S41. Based on the situation map, obtain equipment status information, task queue information, personnel location information and environmental constraint information in the current operation scenario, and construct an equipment-task allocation model; take minimizing the operation completion time and maximizing equipment utilization as the dual objective function, and use the equipment health status extracted from the situation map as the equipment availability constraint and the personnel location as the safety avoidance constraint to form a constrained multi-objective optimization model;

[0045] In the formula, For the first The completion time of each loading and unloading task; For the first Utilization rate of loading and unloading equipment; These are the weighting coefficients; The total number of loading and unloading tasks. This represents the total number of loading and unloading equipment.

[0046] S42. A multi-objective intelligent optimization algorithm is used to solve the constraint scheduling model constructed in step S41. The real-time equipment health status in the situation map is used as the reference information for the initial population of the algorithm. Equipment with high health is prioritized to be assigned to critical tasks. Pareto optimal frontier is output through iterative optimization, and equipment-task allocation scheme and operation sequence that meet the current operation requirements are selected from it. S43. During the execution of the scheduling scheme output in step S42, the following two diagnostic evaluations are performed simultaneously: Equipment failure prediction: Input the real-time updated equipment health status feature sequence from the situation map into a pre-trained failure prediction model based on attention mechanism and long short-term memory network, and output the remaining life prediction value and failure probability distribution of key equipment components. The training steps for a fault prediction model based on attention mechanisms and long short-term memory networks specifically include: a1. Collect operational data of key components of loading and unloading equipment from the port operation history database, including vibration signals, temperature curves, load spectra, current waveforms, running time and corresponding fault records; perform noise reduction, normalization and missing value imputation on the collected raw data, and remove abnormal samples. a2. Based on fault records, an exponential decay or linear decay function is used to label the health status at each running moment. The health status value ranges from [0,1], where 1 represents complete health and 0 represents a fault state. A sliding time window is used to divide the continuous time series data into a fixed-length sample sequence. Each sample contains the feature data of the first T time steps and the corresponding health status label or remaining life label. a3. Divide the sample set into training set, validation set and test set according to time order or random sampling method to ensure that data from the same device in continuous time period is not split into different sets to avoid data leakage; a4. Construct a fault prediction model for a long short-term memory network that incorporates an attention mechanism. The network structure includes: Input layer: Receives multi-dimensional time-series feature data, with dimensions (batch_size, time_steps, features); Long Short-Term Memory (LSTM) network layer: contains 1-3 stacked LSTM units, with 64-256 hidden units in each layer, used to extract the dynamic change patterns of time series data; Attention mechanism layer: Calculates the attention weights at each time step in the time dimension, performs a weighted summation of the hidden states output by the LSTM, and enhances the feature response at key time steps; Fully connected layer: contains 2-3 fully connected network layers, reduces dimensionality layer by layer, and introduces the Dropout mechanism to prevent overfitting; Output layer: The output layer is set according to the prediction task type. The remaining lifetime prediction task uses a linear activation function to output continuous values, while the failure probability prediction task uses a Sigmoid activation function to output probability values ​​in the range [0,1]. a5. Select the appropriate loss function based on the type of prediction task: For the task of predicting remaining lifetime, mean squared error or mean absolute error is used as the loss function. For the fault probability prediction task, binary cross-entropy is used as the loss function; For multi-task joint prediction, a weighted sum composite loss function is used. a6. Use Adam or RMSprop optimizer to train the model end-to-end, set the initial learning rate to 0.001-0.01, the batch size to 32-128, and the number of training epochs to 50-200; introduce an early stopping mechanism during training, stop training when the validation set loss does not decrease for 10-20 consecutive epochs, and save the model parameters with the best performance on the validation set. a7. The trained model is evaluated using a test set. For the remaining lifetime prediction task, the root mean square error and the coefficient of determination R² are used as evaluation metrics. For the failure probability prediction task, the accuracy, recall and AUC are used as evaluation metrics. The model that passes the evaluation is deployed to the cloud platform.

[0047] Personnel safety risk assessment: Input personnel location trajectories, equipment operating status and work area environmental information from the situation map into a pre-trained graph neural network safety risk assessment model to construct a personnel-equipment-environment interaction graph and output personnel safety risk level and potential conflict areas; The training steps of the graph neural network security risk assessment model specifically include: b1. Collect multi-source heterogeneous data from the port operation history database, including personnel location trajectory sequences, equipment operation status data, operation area environmental perception data, and historical safety accident records; perform spatiotemporal alignment and coordinate unification preprocessing on the collected data; b2. Abstract the work site within each time window into graph-structured data. ,in: Node set This includes personnel nodes, equipment nodes, and environment nodes, with each node associated with a corresponding multimodal feature vector; edge set It represents the spatial and interactive relationships between nodes. When the Euclidean distance between two nodes is less than a preset threshold, an edge connection is established. The characteristics of the edge include relative distance, relative speed, relative direction, etc. b3. Based on historical safety incident records, label the nodes and edges in each graph sample with risk tags: Node risk label: Based on the personnel or equipment involved in the accident record, mark their risk level (high, medium, low or no risk). Edge risk labeling: Based on the interaction relationships between pairs of entities in the incident record, label whether there is a risky interaction on the edge; Map-level risk labels: Based on whether a safety incident occurs within the entire time window, the overall risk level of the map sample is marked; b4. Divide the graph sample set into training set, validation set and test set according to time order to ensure that graph samples in the same time period are not split into different sets; b5. Construct a graph neural network model for security risk assessment, the network structure of which includes: Graph input layer: Receives pre-constructed graph structure data, including node feature matrix, adjacency matrix, and edge feature matrix; Graph Convolutional Layer: A graph attention network is used to aggregate and update node features. The importance weights of neighboring nodes are calculated through a multi-head attention mechanism to capture the spatial interaction relationship between people, equipment and environment. The node feature update formula is:

[0048] In the formula, Indicates the first Layer nodes eigenvectors, For nodes The set of neighboring nodes, Attention coefficient For learnable weight matrix, This is the activation function.

[0049] Temporal graph convolutional layer: Introducing temporal convolutional networks or gated recurrent units to capture the dynamic evolution of node states over time on graph sequences with multiple consecutive time windows; Node-level risk prediction head: The hidden state of each node is mapped through a fully connected layer, and the probability distribution of the node's risk level is output. Edge-level risk prediction head: The hidden states of two connected nodes are concatenated and input into the fully connected layer, and the binary probability of whether there is a risk interaction between the edges is output. Graph-level risk prediction head: After global pooling of the features of all nodes in the graph, the input is fed into the fully connected layer, and the output is the overall risk level of the graph sample; b6. Optimize the model using a multi-task joint loss function:

[0050] In the formula, Cross-entropy loss for node risk classification, Cross-entropy loss for edge risk classification, For graph-level risk classification, the cross-entropy loss is used. These are the weighting coefficients for each loss term; b7. Use the Adam optimizer to train the model end-to-end, set the initial learning rate to 0.001-0.005, the batch size to 16-64, and the number of training epochs to 100-300; introduce an early stopping mechanism and a learning rate decay strategy during training, stop training when the validation set loss does not decrease for 10-20 consecutive epochs, and save the model parameters with the best performance on the validation set. b8. The trained model is evaluated using a test set. For node risk prediction, accuracy and macro average F1 score are used as evaluation metrics. For edge risk prediction, precision, recall and AUC are used as evaluation metrics. For graph-level risk prediction, accuracy and confusion matrix are used as evaluation metrics. The evaluated model is then deployed to the cloud platform.

[0051] S44: Integrate the scheduling scheme output in step S42, the fault prediction results output in step S43, and the safety risk assessment results to generate a comprehensive decision package containing scheduling instructions, equipment early warning information, and personnel safety tips; if the fault prediction results indicate that a certain piece of equipment is at high risk, mark the equipment as "pending maintenance" in the scheduling instructions and suspend new task assignments; if the safety risk assessment finds that personnel have entered a dangerous area, trigger an immediate warning and adjust the work plans of surrounding equipment. S45. The integrated decision package generated in step S44 is sent to the equipment control system and personnel early warning terminal. At the same time, the feedback data during the execution process (such as the actual response of the equipment and changes in the position of personnel) is updated to the situation map in real time, triggering the next iteration of steps S41 to S44, forming a continuous optimization closed loop of perception-diagnosis-scheduling-execution-feedback.

[0052] In some embodiments, step S5 includes: S51. Based on the equipment failure prediction results and personnel safety risk assessment results, the early warning level is divided into three levels: Level 1 Warning (Red): Equipment has less than 24 hours of remaining lifespan or a failure probability of more than 90%, or personnel enter a high-risk area and are less than 1 meter away from the hazard source; Level 2 Warning (Orange): Equipment remaining lifespan is less than 72 hours or failure probability is higher than 70%, or personnel enter the danger zone and are 1-3 meters away from the danger source; Level 3 warning (yellow): The remaining lifespan of the equipment is less than 168 hours or the probability of failure is higher than 50%, or personnel are approaching the dangerous area and are 3-5 meters away from the source of danger. The warning level is mapped to the corresponding risk level, which serves as the basis for subsequent scheduling adjustments: A Level 1 warning corresponds to high risk: it indicates an imminent security threat or an impending equipment malfunction, requiring immediate intervention; Level 2 warning corresponds to medium risk: indicating significant safety hazards or a marked decline in equipment performance, requiring close monitoring; Level 3 warning corresponds to low risk: indicating that there is a potential safety risk or a minor equipment malfunction, requiring continuous monitoring; S52. Based on the warning level classification results of S51, trigger the corresponding level of warning response: Level 1 warning: Triggers emergency shutdown of equipment, simultaneously sends a high-decibel audible and visual alarm to the work site, sends a vibration alarm to the smart bracelets worn by on-site personnel, pushes a pop-up alarm to the central control room, and automatically calls safety management personnel; Level 2 warning: Triggers equipment to slow down, sends audible and visual alarms to on-site personnel, sends vibration alarms to on-site personnel's smart wristbands, and pushes warning information to the central control room; Level 3 early warning: Send vibration alerts to the smart wristbands of on-site personnel, push alert information to the central control room, and record abnormal statuses for subsequent analysis; S53. Based on the warning level and type, trigger the corresponding safety protection measures: Equipment-side protection: For equipment failures detected in Level 1 warnings, the equipment is triggered to apply emergency braking and cut off the power source; for equipment anomalies detected in Level 2 warnings, the equipment is triggered to automatically decelerate to a safe speed. Personnel protection: In the event of personnel intrusion into a dangerous area, the smart bracelet guides the personnel to evacuate, while triggering the avoidance mechanism of surrounding devices; Environmental protection: For hazardous areas, warning lights are automatically activated, and no-entry zones are marked on the electronic fence; S54. The diagnostic assessment results (including equipment failure prediction, personnel safety risk level, trigger warning level, risk level, and implementation status of protective measures) shall be fed back to the equipment dispatching system in real time. The feedback information includes: Equipment Dimensions: Equipment ID, Fault Type, Remaining Lifespan, Fault Probability, Warning Level, Risk Level, Recommended Maintenance Time; Personnel dimension: Personnel ID, risk level, current location, type of dangerous behavior; Regional dimensions: Region number, risk level, prohibited entry status, warning status; S55. After receiving the feedback information, the scheduling system dynamically adjusts the subsequent scheduling strategy according to the risk level: High-risk equipment (Level 1 warning): Immediately remove from the available equipment pool; assigned tasks should be urgently interrupted and reassigned to other healthy equipment; schedule priority maintenance. Medium-risk equipment (Level 2 warning): Reduce its task allocation priority, shorten continuous operation time, avoid assigning critical tasks and long-cycle tasks, and schedule maintenance windows in subsequent scheduling; Low-risk equipment (Level 3 warning): Marked as "needs attention", maintain normal task allocation but strengthen monitoring, and avoid scheduling overload operations in subsequent scheduling; High-risk areas: Automatically avoid high-risk areas during AGV path planning, replan the travel route, and prohibit personnel and equipment from entering; Medium-risk areas: The number of people entering is limited, and detours should be prioritized in route planning. If entry is absolutely necessary, protective equipment must be worn. Low-risk areas: Maintain normal operations but strengthen monitoring and record information of personnel entering; S56. After the faulty equipment has been repaired and passed the test, or after personnel have left the danger zone and safety has been confirmed, the safety management personnel shall manually or automatically lift the warning in the central control room. After the warning is lifted, the corresponding equipment is added back to the available equipment pool, the corresponding area returns to normal operation, and the dispatching system resumes normal task allocation for the equipment and area. The entire process of warning triggering, protection execution, and dispatching adjustment is recorded and stored in the historical database for subsequent analysis and model optimization. At the same time, the execution effect is fed back to the situation map update mechanism in step S5 to form a continuous optimization closed loop.

[0053] like Figure 2 As shown, this embodiment of the invention also provides an intelligent diagnosis and early warning system for container loading and unloading equipment based on multi-source data fusion, including: The data acquisition layer is used to collect real-time data on the operating status of loading and unloading equipment, the perception data of the working environment, personnel location data, and scheduling business data. The edge computing layer, connected to the data acquisition layer, is used to preprocess multi-source data, perform spatiotemporal alignment, feature extraction, and cross-modal association matching, generate preliminary fused feature vectors, and trigger local emergency response in case of emergency. The cloud platform layer, connected to the edge computing layer, is used to perform deep fusion analysis on the preliminary fusion feature data, generate a situation map that represents the comprehensive situation of the work site in real time, and diagnose and assess the health status of equipment and personnel safety risks. The scheduling decision layer, connected to the cloud platform layer, is used to dynamically optimize equipment scheduling strategies based on the situation map and generate a comprehensive decision package containing scheduling instructions, equipment early warning information, and personnel safety tips. The early warning and protection execution layer, connected to the cloud platform layer and the scheduling decision layer, is used to trigger graded early warnings and security protection based on diagnostic assessment results, and to execute scheduling instructions. The feedback closed-loop unit is connected to the cloud platform layer, scheduling decision layer and early warning and protection execution layer respectively, and is used to update the execution feedback data to the situation map in real time to form a continuous optimization closed loop.

[0054] In some embodiments, the edge computing layer includes: The data preprocessing unit is used to perform real-time cleaning, filtering and noise reduction, image enhancement and outlier removal on the collected multi-source data. The spatiotemporal alignment unit is used to achieve time synchronization of multi-source data based on high-precision timestamps and to establish spatial coordinate transformation relationships to unify the data of each modality into the global coordinate system. The edge feature extraction unit deploys a lightweight deep learning model for real-time extraction of device status features, environmental target features, and personnel activity features. The cross-modal association matching unit is used to construct the feature mapping relationship of the same target under different perception modalities based on spatial location consistency and motion trajectory similarity, and generate a preliminary fused feature vector; The local emergency response unit is used to directly trigger local warnings or equipment emergency stop commands when an emergency danger is detected.

[0055] In some embodiments, the cloud platform layer includes: The data receiving and temporal reconstruction unit is used to receive the preliminary fusion feature data uploaded by the edge nodes, resample the asynchronous data to a unified time base through interpolation methods, and reconstruct it into a three-dimensional tensor form. The deep feature extraction unit deploys a deep learning network model that incorporates a hybrid attention mechanism to extract three types of multidimensional feature vectors that reflect equipment health status, operational efficiency, and safety risks. The feature selection and optimization unit uses an improved sparrow search algorithm that incorporates the Levy flight mechanism to select and optimize the extracted multidimensional features, and outputs the optimal feature subset and weight coefficients. The situation map construction unit is used to construct a graph of the relationship between equipment, environment and personnel through a graph neural network after screening the best features, generate a situation map that represents the comprehensive situation of the work site in real time, and update it dynamically according to a preset time window; The fault prediction and diagnosis unit deploys a pre-trained fault prediction model based on attention mechanism and long short-term memory network to perform real-time diagnosis of equipment health status and output remaining life and failure probability. The safety risk assessment unit deploys a pre-trained graph neural network safety risk assessment model to perform real-time assessments of personnel safety risks and output risk levels.

[0056] Those skilled in the art will clearly understand that the techniques in the embodiments of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium such as a USB flash drive, mobile hard drive, read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk, or other media capable of storing program code. It includes several instructions to cause a computer terminal (which may be a personal computer, server, or a second terminal, network terminal, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention.

[0057] In the embodiments provided by this invention, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0059] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0060] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A container handling equipment intelligent diagnosis and early warning method based on multi-source data fusion, characterized in that, Includes the following steps: S1. Real-time acquisition of operating status data of loading and unloading equipment, operational environment perception data based on radar and vision, personnel location data obtained by ultra-wideband positioning technology, and dispatch system business data. S2. Edge computing is used to preprocess and initially fuse the collected multi-source data; S3. Perform global temporal alignment and deep fusion on the preliminary fused feature data through the cloud platform, and use a deep learning network model with an attention mechanism to extract multi-dimensional features that reflect the health status of equipment, operating efficiency and safety risks. An improved sparrow search algorithm is used to filter and optimize the extracted features, generating a situation map that represents the overall situation of the work site in real time. S4. Based on the situation map, dynamically optimize the equipment scheduling strategy with the goal of minimizing the completion time and maximizing the utilization rate; at the same time, input the feature data in the situation map into the pre-trained fault prediction model to perform real-time diagnosis and assessment of equipment health status and personnel safety risks. S5. Trigger graded early warnings and safety protection based on the diagnostic assessment results, and feed back the diagnostic conclusions to the scheduling system in real time to achieve coordinated optimization of work plans and safety management.

2. The container handling equipment intelligent diagnosis and early warning method based on multi-source data fusion according to claim 1, characterized in that, In S2, the steps for preprocessing and preliminary fusion of the collected multi-source data using edge computing include: S21. Edge computing nodes perform real-time cleaning of incoming multi-source data streams, removing abnormal noise points and erroneous data; for visual image data, a histogram equalization algorithm is used for illumination correction and contrast enhancement; for lidar point cloud data, a statistical filtering algorithm is used for downsampling and outlier removal; for sensor time-series data, a sliding window filtering algorithm is used for smoothing. S22. Based on the high-precision timestamps carried by each sensor data, time alignment is achieved through hardware synchronization signals to ensure that data from different sources correspond to the same physical event under the same time reference; at the same time, a spatial coordinate transformation relationship is established to uniformly transform the lidar point cloud, visual image, and UWB positioning data into the preset global coordinate system. S23. Through a deep learning model deployed at edge nodes, real-time feature extraction is performed on spatiotemporally aligned multi-source data. The features include vibration and load features reflecting the health status of equipment, target category and location features reflecting the working environment, and trajectory and posture features reflecting personnel activities. S24. Perform cross-modal association matching on the extracted visual features, point cloud features, radar features and UWB positioning features. Based on the spatial position consistency and motion trajectory similarity of the target, construct the feature mapping relationship of the same physical target under different perception modalities to form a preliminary fused feature vector.

3. The container handling equipment intelligent diagnosis and early warning method based on multi-source data fusion according to claim 2, characterized in that, The steps in S24 include: S241. Based on the target features of each modality extracted in S23, spatial association is performed using UWB positioning tags as anchor points; the Euclidean distance between visual targets, lidar targets, millimeter-wave radar targets and UWB tags is calculated, and those with distances less than the first threshold are selected as candidate association pairs; for areas without UWB coverage, the spatial distance between each pair of targets in each modality is calculated, and the Hungarian algorithm is used to solve for the optimal match and generate candidate association pairs. S242. Statistically analyze the spatial position deviation of candidate association pairs in consecutive frames, calculate the root mean square error and variance, and remove association pairs whose position fluctuations are greater than the second threshold. S243. The dynamic time warping algorithm is used to calculate the trajectory shape similarity of the two targets in the candidate association pair in the past N frames. At the same time, the velocity vector angle and acceleration difference are calculated. When the trajectory similarity is higher than the third threshold and the motion features meet the preset range, the association is confirmed to be valid. S244. A learning-based offset estimation network is used to perform sub-pixel alignment between millimeter-wave radar points and visual targets; an iterative nearest-point algorithm is used to perform fine registration between the lidar point cloud and the 3D bounding box of the visual target. S245. After verification and alignment, the modal attribute information is concatenated into vectors to generate a preliminary fusion feature vector containing target identifier, three-dimensional position, motion state, geometric dimensions and appearance features.

4. The container handling equipment intelligent diagnosis and early warning method based on multi-source data fusion according to claim 1, characterized in that, The steps in S3 include: S31. The cloud platform receives the preliminary fusion feature data uploaded by the edge nodes, and uses an interpolation method based on high-precision timestamps to resample the asynchronously sampled multi-source data to a unified time reference, forming a time-aligned multimodal data sequence, and reconstructing it into a three-dimensional tensor form. S32. Construct a deep learning network model that incorporates a hybrid attention mechanism, and reflect three types of multidimensional features—equipment health status, operational efficiency, and safety risks—based on three-dimensional tensor outputs. S33. An improved sparrow search algorithm incorporating the Levy flight mechanism is used to filter and optimize the extracted multidimensional features. The fitness function is used to evaluate the classification accuracy and feature dimension of the feature subset, and the optimal features and weight coefficients are output. S34. Organize the selected optimal features according to three dimensions, map them to a unified feature space using a multilayer perceptron, and construct a graph neural network to generate a situation map representing the overall situation of the work site. S35. Repeat steps S31 to S34 by sliding the preset time window step size to achieve real-time dynamic updating of the situation map.

5. The intelligent diagnosis and early warning method for container loading and unloading equipment based on multi-source data fusion according to claim 4, characterized in that, In S33, an improved sparrow search algorithm incorporating the Levy flight mechanism is used for feature selection optimization, specifically including: S331. Initialize the sparrow population using binary encoding, with each individual representing a feature subset selection scheme; S332. Construct a fitness function that balances classification accuracy and feature dimension; In the formula, For the first The classification accuracy of a subset of features selected for each individual sparrow on the validation set; For the first The number of features selected by each individual; The total number of features; and Let be the weighting coefficient, satisfying ; S333. Take the sparrows with the highest fitness M in the population as the discoverers, and introduce the Levy flight perturbation step size into the discoverer position update formula to expand the search space and avoid getting trapped in local optima. S334. Joiners update their positions following the discoverers, while vigilants randomly update their positions to guide population migration. S335. Use the Sigmoid function to convert consecutive position values ​​into binary values; S336. Calculate the fitness value of each sparrow individual after the update, and adopt the elite retention strategy to directly retain the optimal set number of individuals in the current population to the next generation, replacing the same number of the worst individuals. S337. Through iterative optimization, output the optimal feature subset and its weight coefficients.

6. The intelligent diagnosis and early warning method for container loading and unloading equipment based on multi-source data fusion according to claim 1, characterized in that, The steps in S4 include: S41. Construct an equipment-task allocation model based on the situation map, with the dual objectives of minimizing completion time and maximizing utilization, and take equipment health status and personnel location as constraints. S42. A multi-objective intelligent optimization algorithm is used to solve the scheduling model. The real-time health status of the devices in the situation map is used as the reference information for the initial population of the algorithm. The optimal device-task allocation scheme and operation sequence are output through iterative optimization. S43. Input the equipment health status feature sequence in the situation map into a pre-trained fault prediction model based on attention mechanism and long short-term memory network, and output the remaining life of the equipment and the probability of failure. S44. Input personnel location, equipment status, and work area environmental information into a pre-trained graph neural network safety risk assessment model, and output the personnel safety risk level. S45. Integrate the allocation plan, fault prediction results and safety assessment results to generate a comprehensive decision package, issue it for execution and provide feedback to update the situation map.

7. The intelligent diagnosis and early warning method for container loading and unloading equipment based on multi-source data fusion according to claim 1, characterized in that, The steps in S5 include: S51. Based on the diagnostic assessment results, the early warning levels are divided into Level 1, Level 2, and Level 3; and Level 1 is mapped to high risk, Level 2 to medium risk, and Level 3 to low risk. S52. Trigger differentiated early warning responses based on the early warning level. Level 1 early warning triggers emergency shutdown of equipment and audible and visual alarms exceeding the set decibel level. Level 2 early warning triggers deceleration of equipment and audible and visual alarms. Level 3 early warning triggers vibration alerts on smart bracelets. S53. Trigger corresponding protective measures according to the warning type, including emergency braking of equipment, personnel evacuation guidance, and warning of dangerous areas; S54. Real-time feedback of equipment failure prediction, personnel risk level, early warning level and protection implementation status to the equipment dispatching system; S55. The scheduling system dynamically adjusts task allocation based on feedback information, suspends task allocation for high-risk equipment, reduces the priority of medium-risk equipment, and re-plans routes for risk areas. S56. After the risk is eliminated, the warning can be manually or automatically lifted, and normal operation can be restored.

8. A smart diagnostic and early warning system for container loading and unloading equipment based on multi-source data fusion, characterized in that, include: The data acquisition layer is used to collect real-time data on the operating status of loading and unloading equipment, the perception data of the working environment, personnel location data, and scheduling business data. The edge computing layer, connected to the data acquisition layer, is used to preprocess multi-source data, perform spatiotemporal alignment, feature extraction, and cross-modal association matching, generate preliminary fused feature vectors, and trigger local emergency response in case of emergency. The cloud platform layer, connected to the edge computing layer, is used to perform deep fusion analysis on the preliminary fusion feature data, generate a situation map that represents the comprehensive situation of the work site in real time, and diagnose and assess the health status of equipment and personnel safety risks. The scheduling decision layer, connected to the cloud platform layer, is used to dynamically optimize equipment scheduling strategies based on the situation map and generate a comprehensive decision package containing scheduling instructions, equipment early warning information, and personnel safety tips. The early warning and protection execution layer, connected to the cloud platform layer and the scheduling decision layer, is used to trigger graded early warnings and security protection based on diagnostic assessment results, and to execute scheduling instructions. The feedback closed-loop unit is connected to the cloud platform layer, scheduling decision layer and early warning and protection execution layer respectively, and is used to update the execution feedback data to the situation map in real time to form a continuous optimization closed loop.

9. The intelligent diagnosis and early warning system for container loading and unloading equipment based on multi-source data fusion according to claim 8, characterized in that, The edge computing layer includes: The data preprocessing unit is used to perform real-time cleaning, filtering and noise reduction, image enhancement and outlier removal on the collected multi-source data. The spatiotemporal alignment unit is used to achieve time synchronization of multi-source data based on high-precision timestamps and to establish spatial coordinate transformation relationships to unify the data of each modality into the global coordinate system. The edge feature extraction unit deploys a lightweight deep learning model for real-time extraction of device status features, environmental target features, and personnel activity features. The cross-modal association matching unit is used to construct the feature mapping relationship of the same target under different perception modalities based on spatial location consistency and motion trajectory similarity, and generate a preliminary fused feature vector; The local emergency response unit is used to directly trigger local warnings or equipment emergency stop commands when an emergency danger is detected.

10. The intelligent diagnosis and early warning system for container loading and unloading equipment based on multi-source data fusion according to claim 8, characterized in that, The cloud platform layer includes: The data receiving and temporal reconstruction unit is used to receive the preliminary fusion feature data uploaded by the edge nodes, resample the asynchronous data to a unified time base through interpolation methods, and reconstruct it into a three-dimensional tensor form. The deep feature extraction unit deploys a deep learning network model that incorporates a hybrid attention mechanism to extract three types of multidimensional feature vectors that reflect equipment health status, operational efficiency, and safety risks. The feature selection and optimization unit uses an improved sparrow search algorithm that incorporates the Levy flight mechanism to select and optimize the extracted multidimensional features, and outputs the optimal feature subset and weight coefficients. The situation map construction unit is used to construct a graph of the relationship between equipment, environment and personnel through a graph neural network after screening the best features, generate a situation map that represents the comprehensive situation of the work site in real time, and update it dynamically according to a preset time window; The fault prediction and diagnosis unit deploys a pre-trained fault prediction model based on attention mechanism and long short-term memory network to perform real-time diagnosis of equipment health status and output remaining life and failure probability. The safety risk assessment unit deploys a pre-trained graph neural network safety risk assessment model to perform real-time assessments of personnel safety risks and output risk levels.

Images