Quay crane intelligent tallying system and method

Abstract

The invention discloses a quay crane intelligent tallying system and method. The system comprises a front-end acquisition system, a tallying client, a tallying application end and an external tallying system interface. The front-end acquisition system is used for acquiring real-time multi-modal data of the container in the tallying operation process; the tallying application end is used for receiving the operation plan information transmitted by the external tallying system interface; the tallying client is used for supporting manual additional recording of real-time multi-modal data of the container; the tallying application end is also used for carrying out data preprocessing on the real-time multi-modal data; according to a multi-layer comparison strategy, comparing the real-time multi-modal data of the container subjected to data enhancement processing with the transmitted operation plan information; and when the multi-layer comparison does not pass the condition, determining operation deviation information, triggering an early warning level matched with the corresponding layer, completing graded early warning, and outputting an operation deviation correction suggestion to the tallying client. The automatic operation process of the quay crane intelligent tallying system can be realized, and the tallying speed and accuracy are improved.

Description

Technical Field

[0001] This application relates to the field of port tallying data processing and management technology, specifically to an intelligent quay crane tallying system and method. Background Technology

[0002] Intelligent tallying systems play a crucial role in modern terminal operations. They help staff accelerate and improve the accuracy of tallying, enhance the tallying environment, ensure safe production, and adjust the allocation of machinery and manpower in a timely manner, providing scientific data analysis for terminal management. With the continuous development of terminal operations and the increasing demands for efficiency and accuracy, intelligent tallying has become an essential tool in terminal operations.

[0003] In the past, conventional methods were typically used to improve the efficiency and accuracy of cargo handling at terminals. For example, some terminals relied on manual labor, with staff recording and verifying container numbers and types on-site. While this method ensured a certain level of accuracy, it was inefficient and labor-intensive. Other terminals used traditional image acquisition equipment, such as fixed-angle cameras, for simple image capture and identification of containers. However, this approach only focused on image acquisition and lacked the ability to collect real-time data on environmental conditions such as lighting, temperature, and humidity, as well as operational speed and progress. Furthermore, using vision devices for simple image capture and identification only focused on obtaining surface image information and lacked the ability to deeply analyze and process the information.

[0004] Existing conventional tallying methods have significant shortcomings. Manual tallying is inefficient and cannot meet the demands of modern terminals for large-scale, high-efficiency operations, and is prone to human error. Traditional image acquisition equipment has limited functionality and cannot comprehensively and accurately collect relevant information about containers. Furthermore, due to a lack of effective data processing and comparison capabilities, it cannot promptly identify deviations during the tallying process, and therefore cannot generate timely corrective suggestions. This hinders the automation of loading and unloading processes, significantly impacting the accuracy and efficiency of tallying operations. Summary of the Invention

[0005] In order to automate the operation process of the intelligent cargo handling system for quay cranes and improve the speed and accuracy of cargo handling, this application provides an intelligent cargo handling system and method for quay cranes.

[0006] In a first aspect, this application provides an intelligent cargo handling system for quay cranes, comprising: a front-end data acquisition system, a cargo handling client, a cargo handling application terminal, and an external cargo handling system interface; The front-end acquisition system includes at least one industrial control computer and a multimodal acquisition device. The multimodal acquisition device is communicatively connected to the industrial control computer and is used to acquire real-time multimodal data of containers during the tallying operation according to the control instructions of the industrial control computer and send it to the tallying application terminal. The tallying application terminal communicates with the tallying client and is used to receive the work vessel manifest, pre-allocation information and work plan transmitted from the external tallying system interface, determine and display the container operation queue information of the current bay in real time on the tallying client; The cargo handling client is used to support manual input of real-time multimodal data for containers. The cargo tallying application is also used to perform data augmentation on real-time multimodal container data that has been collected or manually supplemented. Following a multi-layered comparison strategy, it compares the augmented real-time multimodal container data with the transmitted vessel manifest, pre-allocation information, and container operation queue information for the current bay in the operation plan. The multi-layered comparison strategy includes rapid comparison of basic information, deep feature comparison, and operation logic verification. If all multi-layered comparisons pass, a record of the current operation is generated, and the cargo tallying progress is synchronously updated to the external tallying system. If any multi-layered comparison fails, the operation deviation information is determined, and a corresponding level of warning is triggered. A graded warning is then issued, and operation correction suggestions are output to the cargo tallying client.

[0007] By adopting the above scheme, a front-end data acquisition system is set up to collect real-time multimodal data of containers during tallying operations. The tallying application receives operation plan information and displays the real-time operation queue. The tallying client supports manual data entry. The tallying application is set up to perform data enhancement and multi-level comparison. After all multi-level comparisons pass, the operation progress is updated. If there are any failures, the operation deviation information is determined, a graded warning is issued, and a correction suggestion is output. This realizes the automated operation process of the quay crane intelligent tallying system and improves the speed and accuracy of tallying.

[0008] Preferred options also include: The front-end acquisition system is also used to collect real-time data on the working environment, working speed, and working progress during the cargo handling process using a multimodal acquisition device, and upload it to the cargo handling application terminal. The cargo handling application is also used to dynamically adapt the acquisition strategy based on the operating environment data, operating speed data, and operating progress data, and feed it back to the industrial control computer in the front-end acquisition system. This enables the industrial control computer to generate and execute control commands according to the adapted acquisition strategy, thereby completing the real-time multimodal data acquisition of containers during the cargo handling operation. The acquisition strategy includes parameter combinations that select different parameters and set parameter values ​​from the self-acquisition frequency, acquisition angle, acquisition window duration, camera parameters, and number of acquisition modes. It also includes parameter combinations that match the combination of operating environment data, operating speed data, and operating progress data. The sorting application is also used to adapt different data augmentation algorithms based on the work environment data, work speed data, and work progress data. The different data augmentation algorithms include: different image augmentation algorithms adapted to different work environments, data augmentation models with different structural complexities adapted to different work speeds, and different region adaptive augmentation algorithms adapted to different work progresses.

[0009] By adopting the above scheme, the front-end data acquisition system collects and uploads data on the working environment, speed, and progress. This allows the cargo handling application to dynamically adapt its acquisition strategy based on this data and feed it back to the industrial control computer. The industrial control computer then collects data according to the strategy, enabling flexible adjustment of acquisition parameters based on different working conditions, thereby improving the accuracy and effectiveness of data collection. At the same time, the cargo handling application can adapt different data augmentation algorithms based on this data, effectively enhancing the data for different working environments, speeds, and progress, improving data quality, and ultimately improving the accuracy and efficiency of cargo handling operations.

[0010] Preferred options also include: The cargo tallying application is also used to determine the current operation scenario type based on the vessel manifest, pre-allocation information, and operation plan; to learn and acquire key operation information and its deviation threshold for comparison at each layer under different operation scenarios using machine learning algorithms; and to adaptively adjust the key operation information and the deviation threshold of the key operation information involved in the comparison at each layer according to different operation scenarios. The sorting application terminal is also used to determine whether the deviation of the key operation information compared at each level is greater than the deviation threshold of the key operation information. When it is greater than the deviation threshold, the operation deviation information is determined and the corresponding level matching warning level is triggered. When it is not greater than the deviation threshold, the comparison at the next level continues until all levels are compared. When it is not greater than the deviation threshold, the difference between the deviation of the key operation information compared at the corresponding level and the deviation threshold of the key operation information is calculated and recorded as the first difference. Then, the second difference between the absolute value of the first difference and the preset difference is calculated, and the number of key operation information entering the next level is adaptively adjusted according to the second difference. The smaller the second difference, the more key operation information is entered at the next level.

[0011] By adopting the above scheme, the cargo handling application is set to determine the current operation scenario type. Machine learning algorithms are used to learn and acquire key operation information and deviation thresholds under different operation scenarios, and the key operation information and deviation thresholds of each layer are adaptively adjusted to more accurately identify operation deviations under different operation scenarios. The number of key operation information entering the next level is adjusted according to the difference between the calculated difference and the preset difference, thereby improving the accuracy and pertinence of cargo handling operation deviation detection.

[0012] Preferred options also include: The cargo handling application is also used to extract features from container multimodal data, container operation queue information at the current bay, actual operation data obtained from the front-end acquisition system, and historical operation deviation information, and to construct a feature set. The constructed feature set is used to train a cargo handling operation deviation prediction model. This model employs a fused temporal prediction model and a spatial correlation model, with the temporal prediction model serving as the temporal branch and the spatial correlation model as the location branch, respectively. An attention mechanism is introduced to weightedly calculate the fused feature vector and output the predicted cargo handling operation deviation for the next container. The currently collected container multimodal data, vessel manifest, pre-allocation information, operation plan, and actual operation data are input into the cargo handling operation deviation prediction model to predict the cargo handling operation deviation for the next container. Based on the predicted cargo handling operation deviation for the next container, the corresponding level of early warning is triggered in advance, completing the tiered early warning in advance.

[0013] By adopting the above scheme, the tallying application is set up to construct a feature set based on the collected data and historical information and train a prediction model to predict the tallying operation deviation of the next container, thereby triggering early warning and completing the hierarchical warning, effectively reducing the losses caused by tallying operation deviation.

[0014] Preferably, it also includes: a backend data acquisition system; The back-end data acquisition system is used to collect operational data from back-end operators of the sorting and loading operations; The cargo handling application is also used to record operational deviation information and update the operational deviation frequency; when it is determined that there is an operational deviation and the current operational deviation frequency is greater than the preset deviation frequency, it evaluates the current external factor influence index; the external factor influence index is obtained by constructing a quantitative evaluation rule for the degree of influence of external factors on operational deviation based on the operational environment data, operational speed data, operational progress data, and operational equipment data collected by the front-end acquisition system and the operational data obtained by the back-end acquisition system, quantifying the evaluation and weighting the calculation; comparing the evaluated current external factor influence index with the preset influence index threshold, selecting a verification strategy based on the comparison result and executing the verification; the verification strategy includes: a fast verification process with only basic information comparison verification, an enhanced verification process with deep feature comparison verification added on the basis of fast verification, and a strict verification process with all comparison verifications.

[0015] By adopting the above scheme, a backend data collection system is set up to collect the operation data of the backend operators of the cargo handling operation, and a cargo handling application is set up to record operation deviation information and update the deviation frequency. The impact of current external factors on cargo handling operation deviation is quantitatively evaluated, and an appropriate verification strategy is selected based on the evaluation results, thereby improving the pertinence and accuracy of cargo handling operation verification.

[0016] Preferred options also include: The tallying application is also used to construct a set of associated constraints based on the received vessel manifest, pre-allocation information, and operation plan. Each constraint includes the designed container set, the type of associated constraint, the condition of the associated constraint, and the weight. After each container has completed its operation, it searches for all constraints involving the corresponding container in the set of associated constraints. For each constraint found, if all the containers involved have completed their operations, it checks whether the corresponding constraint is satisfied and records the result. Whenever a preset number of containers are completed, it calculates the weighted satisfaction rate of the currently detected constraints and determines whether it is less than the satisfaction rate threshold. If it is less than the satisfaction rate threshold, it is determined that there is an operation deviation in the tallying operation of the current preset number of containers, generates an early warning prompt, and lists the unsatisfied constraints to the tallying client.

[0017] By adopting the above scheme, the tallying application can be configured to construct a set of associated constraints, which can effectively constrain the relationships between container operations; the tallying operations can be constrained and the satisfaction rate calculated, and deviations in the tallying process can be detected in a timely manner and early warning prompts can be generated.

[0018] Preferred options also include: The front-end acquisition system is also used to acquire multiple sets of real-time multimodal data of containers during the cargo handling process according to the control instructions of the industrial control computer, and select one set as backup multimodal data; The cargo handling application is also used to initiate fault tolerance processing when the warning level triggered during the cargo handling operation of a container is greater than the preset warning level, and to re-verify whether there is a false alarm. The fault tolerance processing includes data augmentation and multi-level comparison of the backup multi-modal data to obtain the multi-level comparison results and verify the consistency between the current multi-level comparison results and the original multi-level comparison results. If they are consistent, the operation deviation information is confirmed. If they are inconsistent, it is determined that there is an operation deviation false alarm, and the industrial control computer is controlled to generate control commands to re-collect container multi-modal data.

[0019] By adopting the above scheme, the front-end acquisition system collects multiple sets of real-time multimodal data of containers during the tallying operation, and selects one set as a backup multimodal data. The tallying application is set to start fault tolerance processing when the triggered warning level is greater than the preset warning level. The backup multimodal data is used for comparison and verification to identify false alarms in the tallying operation, avoid unnecessary intervention caused by false alarms, and improve the accuracy and efficiency of the tallying operation.

[0020] Secondly, this application provides an intelligent cargo handling method for quay cranes, comprising: Collect real-time multimodal data of containers during tallying operations; receive operational manifests, pre-allocation information, and operational plans transmitted from the external tallying system interface; determine and display the container operation queue information of the current bay in real time on the tallying client; receive manually entered real-time multimodal data of containers. Data augmentation is performed on real-time multimodal data of containers that have been collected or manually supplemented. According to the multi-layer comparison strategy, the real-time multimodal data of containers after data augmentation processing is compared with the container operation queue information of the current bay in the transmitted operation vessel manifest, pre-allocation information, and operation plan. If all multi-level comparisons pass, a record of the current operation is generated and the progress of the cargo handling operation is updated synchronously to the external cargo handling system; if there are any failures in the multi-level comparisons, the operation deviation information is determined and the corresponding level of warning is triggered, the graded warning is completed and the operation correction suggestions are output to the cargo handling client.

[0021] By adopting the above scheme, real-time multimodal data of containers during tallying operations are collected, the container operation queue information of the current bay is displayed, manual data entry is supported, the data is enhanced, and comparison is performed according to a multi-level comparison strategy. If the multi-level comparison passes, the operation progress is updated; if it fails, deviation information is determined, the warning level is triggered, and correction suggestions are output. This realizes the automation and refined management of intelligent tallying of quay cranes, and improves the efficiency and accuracy of tallying.

[0022] Thirdly, this application provides a computer-readable storage medium including a stored computer program, wherein the computer program, when running, controls the device where the computer-readable storage medium is located to perform the method described above.

[0023] Fourthly, this application provides a computer device, the computer device including a memory, a processor and a program stored in the memory and executable thereon, the program being executed by the processor to implement the steps of the method described above.

[0024] In summary, this application has the following beneficial effects: 1. By setting up a front-end data acquisition system to collect multimodal data of containers in real time during the tallying process, comprehensive and accurate information is provided for tallying operations; a tallying client is set up to facilitate manual data entry by users, improving data integrity; a tallying application is set up to enhance the processing of multimodal data and adopt a multi-layer comparison strategy to promptly identify operational deviations, effectively improving the accuracy and reliability of tallying operations; and a hierarchical early warning and correction system for tallying operations is implemented to ensure the smooth progress of tallying operations and timely updates on tallying progress. 2. By setting up a front-end data acquisition system to collect real-time data on the operating environment, speed, and progress, and by setting up the tallying application to dynamically adapt the data acquisition strategy and data augmentation algorithm accordingly, the system can comprehensively and accurately collect container-related information. The tallying application can also adaptively adjust the key operational information and deviation thresholds for each layer of comparison based on different tallying operation scenarios, improving the accuracy and relevance of tallying deviation detection. Furthermore, the system can quantitatively assess the impact of current external factors and select appropriate verification strategies based on the assessment results, improving the relevance and efficiency of data verification in tallying operations. Finally, the system can perform constraint detection and satisfaction rate calculation for tallying operations, promptly identifying operational deviations and generating early warning prompts. 3. Configure the tallying application to predict tallying deviations for the next container in advance, trigger early warning levels to complete tiered early warnings, thereby improving the efficiency and accuracy of tallying operations; set conditions to initiate fault tolerance processing, and use backup multimodal data for comparison and verification, which can effectively identify false alarms of tallying deviations and avoid unnecessary intervention caused by false alarms. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the intelligent cargo handling system for quay cranes described in a specific embodiment; Figure 2 This is a schematic diagram of data transmission in the intelligent cargo handling system of the quay crane described in a specific embodiment; Figure 3 This is a schematic diagram of the intelligent cargo handling system for quay cranes described in a specific embodiment; Figure 4 This is a flowchart of the intelligent cargo handling method for quay cranes described in a specific embodiment; Figure 5 This is a flowchart of the loading / unloading operations in the intelligent cargo tallying method for quay cranes described in a specific embodiment. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0027] like Figure 1 As shown in the illustration, this application discloses an intelligent cargo handling system for quay cranes, specifically including: a front-end data acquisition system 1, a cargo handling application terminal 2, a cargo handling client 3, and an external cargo handling system interface 4, etc. These modules are interconnected to achieve data collection, access, processing, and display. Figure 2 As shown.

[0028] The front-end acquisition system 1 includes at least one industrial control computer and a multimodal acquisition device. The industrial control computer is typically an industrial-grade computer, communicating with the PLC controller of the gantry crane and the multimodal acquisition device in the ECS (Electronic Control System). The multimodal acquisition device includes network cameras, multimodal sensors (environmental sensors, laser rangefinders, etc.), RFID readers, supplementary lighting equipment, etc. Multiple sets of multimodal acquisition devices can be set up, all deployed on the quay crane; for example, four cameras can be installed on each side of the gantry crane's crossbeam and saddle beam, and one camera can be installed on the top of the gantry crane to capture images of the containers in the four working lanes of the gantry crane and the working lane of the quay crane's rear beam. High-frequency RFID readers are installed on the quay crane spreaders, and laser rangefinders and environmental sensors are deployed on the quay crane's crossbeams.

[0029] The front-end acquisition system 1 is used to collect real-time multimodal data of containers during the cargo handling process according to the control instructions (acquisition frequency, acquisition angle, etc.) of the industrial control computer. This includes: image data of the container body taken from different angles, RFID tag information of the container body read (including container number, container type, weight, etc.), collection of container size data, and operation environment data. The industrial control computer receives real-time operation data uploaded by the bridge crane PLC controller, including: equipment status data, control operation data, safety monitoring data, location positioning data, and fault diagnosis data.

[0030] The tallying application terminal 2 can be composed of multiple servers, communicating with both the tallying client 3 and the external tallying system interface 4. It receives core data transmitted from the external tallying system interface 4, such as the vessel manifest (including container number, type, weight, owner, port of destination, etc.), pre-allocation information (the correspondence between containers and bay / cabin spaces), and operation plans (including current bay space, operation sequence, loading / unloading type, quay crane number, etc.) from the terminal system. Figure 3 As shown. Specifically, manifests and pre-allocation information can be retrieved in real time via an interface; and operation plan update messages can be subscribed to via an MQTT message queue to ensure real-time synchronization of dynamic information such as operation bay positions and operation sequences. The tallying application will process this information and display the container operation queue information for the current (pending operation) bay in real time on the tallying client 2, including: container number, container type, weight, container owner, destination port, hold position, and operation sequence information corresponding to the current bay position. The tallying client 3 can be a computer terminal installed in the control room or a portable mobile terminal, allowing operators to view the operation queue information at any time to assist users in tallying operations. Correspondingly, the tallying application can also display the real-time multimodal data of containers collected in real time on the tallying client 2 according to user selection, so that users can verify and supplement the data.

[0031] The tallying client 3 is used to support users in manually entering real-time multimodal data for containers. Considering that the data collected by the multimodal acquisition device may be incomplete or inaccurate, users can manually enter the data in the tallying client. For example, if the container number recognition software makes a mistake, the user can manually enter the correct container number in the tallying client.

[0032] The cargo handling application 2 is also used to preprocess the real-time multimodal data of containers collected and / or manually supplemented, such as data augmentation. Data augmentation can employ various methods, such as using image enhancement algorithms to denoise and sharpen the collected container images to improve image clarity; using pre-trained data augmentation models (such as visual recognition algorithms, YOLOv8 object detection algorithms based on deep learning, combined with multi-view image data, to address issues such as blurring, occlusion, and tilting of container number characters, adding a character enhancement module (image deblurring, contrast enhancement, tilt correction), and simultaneously constructing a massive container number sample library (containing samples of different fonts, wear levels, and harsh environments), etc.) to optimize the collected container data, reduce incomplete data, and make it more consistent with the actual situation.

[0033] The tallying application terminal 2 is also used to compare the real-time multimodal data of the container after data augmentation processing with the transmitted work vessel manifest, pre-allocation information, and container operation queue information of the current bay in the operation plan according to the multi-level comparison strategy; if all multi-level comparisons pass, a record of the current container operation is generated and the tallying operation progress is updated synchronously to the external tallying system; if there are cases of failure in multi-level comparisons, the operation deviation information is determined and the corresponding level of warning is triggered, the graded warning is completed and the operation correction suggestion is output to the tallying client.

[0034] To promptly detect deviations in current container handling operations, such as incorrect container handling sequence, incorrect container placement order, misalignment of cargo hold / bay locations, and errors in yard pre-allocation, a multi-layered comparison strategy is implemented using the tallying application terminal 2. This strategy ranges from simple basic information comparison to logical verification comparison, thereby improving monitoring efficiency and accuracy. Specifically, the multi-layered comparison strategy includes rapid basic information comparison, in-depth feature comparison, and operational logic verification.

[0035] First, a basic information rapid comparison layer is used to identify basic container information in real-time multimodal container data, such as container number and container type. It uses string matching and simple rule matching to compare the identified basic information with the container queue information of the current bay in the transmitted ship manifest, pre-allocation information, and operation plan. This includes comparing container numbers and container types to quickly verify the basic container information and identify obvious errors, such as containers not being in the corresponding queue or incorrect container type. A basic information deviation threshold can be set for the basic information comparison; if the comparison error exceeds the corresponding basic information error threshold, the comparison fails. For the basic information rapid comparison layer, basic information weights and a passing ratio are set. When the ratio reaches a preset deviation threshold, the basic information rapid comparison layer is considered to have passed.

[0036] Secondly, after the basic information quick comparison layer passes, deep feature comparison continues. To verify more detailed information and status of the containers and ensure compliance with operational requirements, a deep feature comparison layer is set up to extract deep features from real-time multimodal data of the containers, such as: container number (including check digits), container type (including ISO code), weight, dangerous goods category, refrigerated container temperature, seal status, container condition (damage, cleanliness, etc.), and container surface features. Rule matching, database query, and image recognition results are used to compare the identified deep features with the container queue information of the current bay in the work vessel manifest, pre-allocation information, and work plan, such as dangerous goods category comparison, weight comparison, and refrigerated container temperature comparison, to obtain the comparison results for each deep feature. For each type of deep feature, a corresponding feature deviation threshold can be preset. If the comparison error exceeds the corresponding deep feature error threshold, it indicates that the corresponding basic information comparison has failed. Similarly, for the deep feature comparison layer, deep feature weights and deep feature pass rates can be set. When the rate reaches the preset deviation threshold, the deep feature comparison layer is considered to have passed.

[0037] Finally, after the deep feature comparison layer passes, the logical verification layer continues with verification and comparison. Verification is performed from the perspectives of operational logic and planning to ensure the overall correctness of the operation. The logical verification layer determines the current container loading / unloading sequence and location from real-time multimodal data and compares it with the container queue information at the current bay in the ship manifest, pre-allocation information, and operational plan. It determines whether the container complies with logical constraints such as loading / unloading sequence, load balancing, dangerous goods segregation, weight distribution, and yard planning. If any logical constraints are not met, the operation is considered unsuccessful; otherwise, it is considered successful. Specific judgments can be based on rules and neural network models for verification.

[0038] Furthermore, after a multi-layered comparison strategy, if all comparisons pass, it indicates that there are no deviations in the current container handling process. A record of the current operation's success is then generated, and the handling progress is synchronously updated to the external handling system. If any multi-layered comparison fails, the specific operational deviation is determined based on the comparison results, such as an incorrect container number. This deviation information is recorded, and corresponding warning levels are triggered based on the level of the deviation information. For example, a level one warning corresponds to the basic information rapid comparison layer, a level two warning to the deep feature comparison layer, and a level three warning to the logic verification layer. After issuing the warning, a pre-trained neural network model can be used to input the determined handling deviation, output suggestions for corrective action, and send them to the handling client, such as prompting operators to intervene manually.

[0039] The above solution utilizes a front-end data acquisition system to comprehensively collect real-time multimodal data of containers. The tallying application then processes and compares this data, achieving an automated tallying workflow. Furthermore, a multi-layered comparison strategy allows for multi-level checks of the accuracy of tallying operations, enabling timely detection and correction of deviations. Simultaneously, the tallying client supports manual data entry, improving data accuracy and completeness.

[0040] A specific embodiment, differing from the above embodiments, dynamically adjusts the data acquisition strategy and data augmentation algorithm according to different operational conditions, enabling the system to better adapt to complex and changing operational environments and improve the accuracy and efficiency of cargo handling operations; the system also includes: The front-end acquisition system 1 is also used to collect real-time data on the working environment (such as temperature, humidity, light, etc.), working speed (normal, high speed, low speed) and working progress (start, middle, end) during the sorting operation using a multimodal acquisition device, and upload it to the sorting application terminal.

[0041] The cargo handling application terminal 2 is also used to dynamically adapt the data acquisition strategy based on the operating environment data, operating speed data, and operating progress data, and feed it back to the industrial control computer in the front-end data acquisition system. This allows the industrial control computer to generate and execute control commands according to the adapted acquisition strategy, completing the real-time multimodal data acquisition of containers during the cargo handling operation. From the perspective of data acquisition equipment optimization, the acquisition strategy includes parameter combinations that select different parameters and set parameter values ​​from multiple parameters such as self-acquisition frequency, acquisition angle, acquisition window duration, camera parameters, and number of acquisition modes; and sets parameter combinations that match the combination of operating environment data, operating speed data, and operating progress data.

[0042] In practical data applications, considering sufficient daytime lighting, a light shield can be used; at night in low light conditions, supplementary lighting can be used; at the beginning of the operation, the container is positioned low, so the acquisition angle needs to be adjusted; in the middle stage, the camera angle needs to be adjusted according to changes in container height; at the end stage, the speed may be increased due to the urgency to complete the operation, so the acquisition frequency needs to be increased; using deep learning, based on historical acquisition strategies verified by experts and their adapted operating environment data, operating speed data, and operating progress data, a model for acquiring acquisition strategies is trained and generated. The model takes the current operating environment data, operating speed data, and operating progress data as input and outputs a parameter combination that matches the combination of operating environment data, operating speed data, and operating progress data.

[0043] The cargo handling application 2 is also used to adapt different data augmentation algorithms based on operational environment data, operational speed data, and operational progress data, so that the adapted data augmentation algorithms can be used for subsequent data augmentation. These different data augmentation algorithms include: First, different image augmentation algorithms adapted to different operational environments; for example, training multiple image augmentation models for different operational environments, based on a real-time operational environment switching model, including: a cloudy day model, a nighttime model, and a rain / fog model; Second, data augmentation models with different structural complexities adapted to different operational speeds; for example, in the fast operational phase, a data augmentation model with low structural complexity (the structural complexity level can be determined based on network depth thresholds and channel number thresholds) is used to quickly process image clarity and contrast; in the medium-to-low speed operational phase, a data augmentation model with medium structural complexity is used to apply noise reduction, sharpening, and contrast adjustment to ensure image detail integrity; Third, different region-adaptive augmentation algorithms adapted to different operational progresses, performing local augmentation on the area where the container number is located; for example, when operations begin, local image augmentation is performed on the area corresponding to the container bay involved in the initial operational phase of the operational plan.

[0044] In addition, if there is a conflict between the matching data augmentation algorithms, the difference between the current operating environment, operating speed and operating progress and the corresponding parameters of the container queue information of the current bay is determined. The larger the difference, the higher the priority, and the data augmentation algorithm with the higher priority parameter is adopted first.

[0045] In a specific embodiment, considering that different cargo handling scenarios have different focuses regarding cargo handling deviations, in order to better adapt to different scenarios, accurately determine operational deviation information, and trigger corresponding early warning levels, thereby improving the accuracy and efficiency of cargo handling operations; the system further includes: The cargo handling application terminal 2 is also used to determine the current operation scenario type based on the vessel manifest, pre-allocation information and operation plan; the operation scenario type includes loading operation (loading containers from the dock onto the ship), unloading operation (unloading containers from the ship onto the dock) and transshipment operation (transferring containers from one ship to another).

[0046] The tallying application terminal 2 is also used to utilize machine learning algorithms to learn and acquire key operational information and its deviation thresholds for comparison at each layer under different operational scenarios. Specifically, it selects historical data marked with key operational information and its deviation thresholds for the corresponding operational scenarios, under the condition that the early warning accuracy rate is higher than the preset accuracy rate, to complete model learning and training. For loading operations, strict adherence to the stowage plan is required. Common tallying deviations include: incorrect container number (wrong container loaded), incorrect container type (size or type mismatch), incorrect location (bay position, column, layer error), incorrect weight (overweight containers not placed in the designated location), incorrect location of dangerous goods containers, and abnormal status of refrigerated containers, etc., to initially determine the marked key operational information. For unloading operations, it is required to follow the ship's plans and unloading plan. Common tallying deviations include: incorrect container number (overloading, underloading, incorrect unloading), incorrect container type (not matching the manifest), damage records, and abnormal seals, etc., to initially determine the marked key operational information. For transshipment operations, it is necessary to prepare to track the flow of transshipment containers. Key cargo handling deviations include: incorrect container number, incorrect port of destination, incorrect transshipment route, etc., and to initially determine the key operational information to be marked.

[0047] The cargo handling application terminal 2 is also used to adaptively adjust the key operational information and the deviation threshold of the key operational information involved in each layer of comparison according to different operational scenarios. For example, in the loading operation, the depth feature comparison layer focuses on comparing weight, dangerous goods category, and refrigerated container temperature, while in the unloading operation, the same comparison layer focuses on comparing the container condition (damage, whether the lead seal is intact); in the transshipment operation, the same comparison layer focuses on comparing the destination port and transshipment port.

[0048] The sorting application terminal 2 is also used to determine the deviation degree (weighted comprehensive deviation degree) of the key operation deviation information of each level comparison and the deviation degree threshold of the key operation deviation information (preset deviation degree threshold of the corresponding level). When it is greater than the deviation degree threshold, it indicates that the deviation is large, and the operation deviation information can be directly determined and the corresponding level matching warning level can be triggered to achieve timely triggering; when it is not greater than the deviation degree threshold, it indicates that the deviation of the current level is not large, and the comparison of the next level continues until all levels are compared and the full level comparison result is output.

[0049] To further optimize the comparison results, the cargo handling application terminal 2 is also used to calculate the difference between the deviation degree of the key operation information of the corresponding level comparison and the deviation degree threshold of the key operation information when it is not greater than the deviation degree threshold, and record it as the first difference value; that is, to determine whether the deviation degree of the current level is close to the identified deviation degree threshold. The closer it is, the more likely there is a deviation, and the next level needs to be compared more carefully. Then, a second difference value is set between the absolute value of the first difference value and the preset difference value, and the number of key operation information entering the next level is adaptively adjusted according to the second difference value. The smaller the second difference value, the more key operation information is set for the next level. Specifically, when using machine learning algorithms to learn and obtain the key operation deviation information of each level comparison under different operation scenarios, different levels of key operation deviation information can be learned and obtained simultaneously.

[0050] In one specific embodiment, to further and promptly identify potential deviations in cargo handling operations, historical and real-time data are used to predict possible deviations and issue early warnings, thereby reducing errors and improving the accuracy and efficiency of cargo handling operations. The system also includes: The cargo handling application terminal 2 is also used to extract features from multimodal container data (such as container number, container type, size, weight, cargo owner, destination, dangerous goods class, etc. collected by the front-end system), container operation queue information of the current bay, actual operation data (actual operation data uploaded by the PLC controller obtained by the front-end acquisition system, including actual operation time, actual location, equipment status, etc.) and historical operation deviation information (historical cargo handling error records, such as wrong container, misplacement, wrong order, etc.). Specifically, it includes: temporal features: operation sequence, time interval, operation speed, etc.; positional features: distribution of bay, row, column, and layer, and positional relationship of adjacent containers; auxiliary features: container attribute features including container type, size, weight, dangerous goods class, etc., and contextual features including weather, visibility, equipment status, etc.

[0051] The cargo handling application terminal 2 is also used to construct a feature set based on the extracted features; train a cargo handling operation deviation prediction model using the constructed feature set; input the currently collected container multimodal data, operation plan, and actual operation data into the cargo handling operation deviation prediction model to predict the cargo handling operation deviation of the next container; and trigger the corresponding level matching early warning level in advance based on the predicted cargo handling operation deviation of the next container to complete the graded early warning in advance.

[0052] To more accurately predict cargo handling deviations, a fusion of a temporal prediction model and a spatial correlation model is used as the model's structural framework for the deviation prediction model. An attention mechanism is simultaneously introduced to strengthen the weights of key correlation features, achieving fusion judgment. Specifically, firstly, a temporal prediction model (LSTM) is constructed as the temporal branch. The model inputs are temporal feature vectors and auxiliary features, and the model outputs are: the predicted theoretical operation time for the next container and a temporal compliance score (the score is set based on the difference between the theoretical operation time and the planned time; e.g., 0-1 points, the higher the score, the more consistent with the temporal rules, and the smaller the time difference). Secondly, a spatial correlation model (GNN) is constructed as the location branch. The model inputs are location features and auxiliary features, and the model outputs: the predicted theoretical location for the next container and a location compliance score (the score is set based on the three-dimensional distance between the theoretical coordinates and the planned coordinates; e.g., 0-1 points, the higher the score, the more consistent with the location rules, and the smaller the distance). Finally, an attention mechanism is introduced, and a fusion model is set up. Feature weights are set for the temporal and positional branches (e.g., 0.6 for positional compliance and 0.4 for temporal compliance in loading operations; 0.6 for temporal compliance and 0.4 for positional compliance in unloading operations). The model inputs are temporal feature vectors, spatial feature vectors, auxiliary features, theoretical operation time predictions, temporal compliance scores, theoretical position predictions, and positional compliance scores. The weighted fusion outputs a fusion feature vector. The model outputs the positional deviation and temporal deviation for the next container tallying operation, and also outputs a comprehensive compliance score.

[0053] In a specific embodiment, to further improve the accuracy and reliability of cargo handling operations, an adaptive verification is performed on the comparison results of whether there are deviations in the current container handling process; the system also includes: a back-end data acquisition system 5; The back-end data acquisition system 5 is used to collect the operation data of the back-end operators in the sorting and handling operation. Specifically, it can use a vision device to collect the operation actions and facial expressions of the back-end operators, and determine the operation data of the back-end operators by receiving the operation instructions uploaded by the PLC control.

[0054] The cargo handling application terminal 2 is also used to record operational deviation information and update operational deviation frequency; when it is determined that there is an operational deviation and the current operational deviation frequency is greater than the preset deviation frequency, the current external factor influence index is evaluated; the evaluated current external factor influence index is compared with the preset influence index threshold, and a verification strategy is selected and verification is executed based on the comparison result; specifically, if the current external factor influence index is less than the first preset influence index threshold, a fast verification process is matched; if the current external factor influence index is greater than the first preset influence index threshold and less than the second preset influence index threshold, an enhanced verification process is matched; if the current external factor influence index is greater than the second preset influence index threshold, a strict verification process is matched; if the second preset influence index threshold is greater than the first preset influence index threshold, a fast verification process is matched.

[0055] The external factor impact index is obtained by constructing quantitative assessment rules for the degree of influence of external factors on operational deviations based on operational environment data, operational speed data, operational progress data, and operational equipment data collected by the front-end acquisition system and operational data obtained by the back-end acquisition system. The results are then quantified, assessed, and weighted. In this embodiment, the quantitative assessment rules are constructed by considering equipment factors, human factors, environmental factors, and other unforeseen operational factors; the formula is: Impact Index I = The quantitative evaluation rules for equipment factors use the analysis of whether equipment malfunctions and their severity as quantitative indicators, setting quantitative rules and scoring for each indicator. The quantitative evaluation rules for human factors use the analysis of whether operator fatigue or non-standard operating procedures occur as quantitative indicators, setting quantitative rules and scoring for each indicator. The quantitative evaluation rules for environmental factors use the analysis of whether adverse weather conditions such as rain, fog, snow, and strong winds occur as quantitative indicators, setting quantitative rules and scoring for each indicator. The quantitative evaluation rules for other unforeseen operational factors use the analysis of whether sudden data events occur in operational speed data, operational progress data, and other operational data as quantitative indicators, setting quantitative rules and scoring for each indicator. Furthermore, the weights of these factors can be obtained from historical data analysis or preset manually.

[0056] The verification strategy includes: a rapid verification process for basic information comparison only (re-performing information comparison verification at the basic information comparison layer); an enhanced verification process that adds deep feature comparison verification on top of rapid verification (re-performing information comparison verification at both the basic information comparison layer and the feature comparison layer); and a strict verification process for all comparison verifications (re-performing information comparison verification at all comparison layers). If the verification results are consistent, it indicates that the original comparison results were correct, the verification is successful, and the confirmed operational deviation information is directly output, triggering the corresponding level of warning; otherwise, a prompt indicating a confirmed error in the cargo handling operation is generated and output.

[0057] In a specific embodiment, considering the interrelationships between containers during the tallying process, these relationships can be used to verify whether there are deviations in the current tallying process. Furthermore, by detecting and statistically analyzing operational constraints, it can determine whether tallying deviations exist, issue timely warnings, and list unmet constraints to ensure that the tallying operation complies with the associated constraint conditions, thereby improving the accuracy and standardization of the tallying operation. The system also includes: the tallying application terminal 2, which is further used to construct a set of associated constraints based on the received vessel manifest, pre-allocation information, and operational plan. Each constraint This includes the designed container set, the type of association constraint, the association constraint conditions, and the weights. Specifically, it takes two containers... For a container collection The types of relationships between two containers include spatial, temporal, and logical; the constraints on these relationships include: container bay location constraints and theoretical operation time constraints; weights. Characterize the current constraints The weight can be determined based on whether either of the two containers is marked as important. If an important container is marked, the weight setting value is selected from the first preset weight range; otherwise, the weight setting value is selected from the second preset weight range (which is less than the first preset weight range).

[0058] The cargo handling application terminal 2 is also used for each container After completing the task, the set of self-associative constraints Search for all containers related to the above. Constraints; such as those related to containers. All constraints , , Corresponding container set , , For each constraint found , , If all the containers involved have been processed, that is , All tasks are completed, and the corresponding constraints are checked to see if they are satisfied. The relationship constraints in the two containers are all satisfied, that is, the current two containers If the beta distance satisfies the beta constraint and the theoretical operation time satisfies the theoretical operation time constraint, record the result of whether the constraints are satisfied; for example: satisfy, satisfy, Not satisfied. To prevent overly complex calculations, a satisfaction rate calculation is performed periodically. Whenever a preset number of containers (e.g., 5-7 containers) are completed, the weighted satisfaction rate of the currently detected constraints is calculated and checked against a threshold. If the threshold is less than the threshold, it indicates a problem with the relationship between the containers, preventing mutual verification of operational deviations in container tallying operations (e.g., loading, unloading). This signifies an operational deviation in the tallying process for completing the current preset number of containers, generating an alert and listing the unmet constraints to the tallying client to determine if deviations exist in the container tallying process related to the specific constraints.

[0059] In a specific embodiment, to ensure the safety and efficiency of the correction process, it is essential to ensure the accuracy of the identified operational deviation information. Therefore, upon receiving a graded warning, appropriate fault-tolerant processing is implemented based on the warning level to avoid false triggering of complex correction processes due to high warning levels. The system also includes: The front-end acquisition system 1 is also used to acquire multiple sets of real-time multimodal data of containers during the tallying process according to the control instructions of the industrial control computer, and select one set as backup multimodal data. Specifically, the system acquires real-time multimodal data of containers during the tallying process according to multiple sets of multimodal acquisition devices, and selects one set as backup real-time multimodal data for that container.

[0060] The cargo handling application terminal 2 is also used to address situations where a container triggers a warning level higher than the preset warning level (third warning level) during cargo handling operations. This means that the main problem is not a simple correction by directly replacing the container, but rather involves more complex corrections such as load balance checks or dangerous goods placement. In such cases, to avoid misoperation, fault tolerance processing is initiated to re-verify whether there is a false alarm.

[0061] The fault-tolerant processing includes data augmentation and multi-level comparison of the backup multimodal data to obtain multi-level comparison results and verify the consistency between the current multi-level comparison results and the original multi-level comparison results. If they are consistent, the operation deviation information is confirmed. If they are inconsistent, it is determined that there is a false alarm of operation deviation, and the industrial control computer is controlled to generate control commands to re-acquire container multimodal data.

[0062] like Figure 4 As shown in the figure, this application discloses an intelligent cargo handling method for quay cranes, the specific steps of which include: S1. Collect real-time multimodal data of containers during the tallying process.

[0063] S2. Receive the work vessel manifest, pre-allocation information and work plan transmitted from the external tallying system interface, determine and display the container operation queue information of the current bay in real time on the tallying client.

[0064] S3. Receive manual supplementation of real-time multimodal data of containers from users.

[0065] S4. Perform data augmentation on real-time multimodal data of containers that have been collected or manually supplemented.

[0066] S5. Following a multi-layered comparison strategy, compare the real-time multimodal data of containers after data augmentation with the transmitted operational vessel manifest, pre-allocation information, and container operation queue information of the current bay in the operation plan.

[0067] S6. If all multi-level comparisons pass, a record of the current operation is generated and the progress of the cargo handling operation is updated synchronously to the external cargo handling system; if there are cases where the multi-level comparisons fail, the operation deviation information is determined and the corresponding level of warning is triggered, the graded warning is completed and the operation correction suggestions are output to the cargo handling client.

[0068] The following example, using a specific application in a ship loading and tallying operation scenario, demonstrates the automated tallying operation monitoring and correction process. First, during the loading operation, the truck carrying the container stops at the lane below the quay crane. The quay crane trolley moves above the lane and lowers the spreader. The spreader grabs and lifts the container and moves it. Simultaneously, the quay crane's PLC system uploads the container's operational status to the front-end data acquisition device. Furthermore, during the quay crane's movement of grabbing and lifting the container, the front-end data acquisition device collects image data of the container's movement, achieving real-time multimodal data acquisition of the container during the tallying operation. To address the actual and effective container loading process, such as... Figure 5 As shown, by determining whether the container has risen to a certain height in the container operation status synchronously uploaded by the receiving gantry crane PLC system, the multimodal data acquisition device is triggered when the container has risen to a certain height, collecting real-time multimodal data of the container during the tallying operation. Simultaneously, it receives the vessel manifest, pre-allocation information, and operation plan transmitted from the external tallying system interface, determines and displays the container operation queue information of the current bay in real time on the tallying client, and receives manual supplementary data on the real-time multimodal data of the container from the user.

[0069] Then, the real-time multimodal data of containers that have been collected or manually supplemented are preprocessed. This involves filtering valid images through the application server, initially identifying container numbers, comparing multiple images for damage verification, obtaining and saving the damage verification results, and performing image enhancement and compensation for incomplete images.

[0070] Finally, following a multi-layered comparison strategy, the real-time multimodal data of the containers after data augmentation is compared with the transmitted operational manifest, pre-allocation information, and container operation queue information of the current bay in the operation plan. If any failure occurs in the multi-layered comparison, the operation deviation information is determined and the corresponding level of warning is triggered. The graded warning is completed and the operation correction suggestions are output to the tally client. For example, if the operation deviation information is that the container number is not in the operation plan or the container's position in the ship's hold is inaccurate, it is fed back to the tally client for manual correction.

[0071] Similarly, the unloading and tallying operation scenario is similar to the loading scenario. The gantry crane trolley moves above the deck to grab the container, moves to the inside of the gantry crane and lowers the spreader. At this time, the container operation status is uploaded through the PLC system. When the gantry crane spreader grabs the container and lowers it to a certain height, it triggers the multi-modal data acquisition device to operate. Subsequently, data processing and multi-level comparison are performed to output the operation deviation in the current container operation process.

[0072] This application also discloses a computer-readable storage medium.

[0073] Specifically, the computer-readable storage medium stores a computer program that can be loaded by a processor and executed as described above for the intelligent cargo handling method of the quay crane. The computer-readable storage medium includes, for example, various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0074] This application also discloses a computer device.

[0075] Specifically, the computer device includes a memory and a processor, and the memory stores a computer program that can be loaded by the processor and executed to perform the above-mentioned intelligent cargo handling method for quay cranes.

[0076] The above are all preferred embodiments of this application and are not intended to limit the scope of protection of this application. Any feature disclosed in this specification (including the abstract and drawings) may be replaced by other equivalent or similar features unless specifically stated otherwise. That is, unless specifically stated otherwise, each feature is only one example of a series of equivalent or similar features.

Claims

1. A smart cargo handling system for quay cranes, characterized in that, include: Front-end data acquisition system, inventory management client, inventory management application terminal, and external inventory management system interface; The front-end acquisition system includes at least one industrial control computer and a multimodal acquisition device. The multimodal acquisition device is communicatively connected to the industrial control computer and is used to acquire real-time multimodal data of containers during the tallying operation according to the control instructions of the industrial control computer and send it to the tallying application terminal. The tallying application terminal communicates with the tallying client and is used to receive the work vessel manifest, pre-allocation information and work plan transmitted from the external tallying system interface, determine and display the container operation queue information of the current bay in real time on the tallying client; The cargo handling client is used to support manual input of real-time multimodal data for containers; The cargo tallying application is also used to perform data augmentation on real-time multimodal container data that has been collected or manually supplemented. Following a multi-layered comparison strategy, it compares the augmented real-time multimodal container data with the transmitted vessel manifest, pre-allocation information, and container operation queue information for the current bay in the operation plan. The multi-layered comparison strategy includes rapid comparison of basic information, deep feature comparison, and operation logic verification. If all multi-layered comparisons pass, a record of the current operation is generated, and the cargo tallying progress is synchronously updated to the external tallying system. If any multi-layered comparison fails, the operation deviation information is determined, and a corresponding level of warning is triggered. A graded warning is then issued, and operation correction suggestions are output to the cargo tallying client.

2. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, Also includes: The front-end acquisition system is also used to collect real-time data on the working environment, working speed, and working progress during the cargo handling process using a multimodal acquisition device, and upload it to the cargo handling application terminal. The cargo handling application is also used to dynamically adapt the acquisition strategy based on the operating environment data, operating speed data, and operating progress data, and feed it back to the industrial control computer in the front-end acquisition system. This enables the industrial control computer to generate and execute control commands according to the adapted acquisition strategy, thereby completing the real-time multimodal data acquisition of containers during the cargo handling operation. The acquisition strategy includes parameter combinations that select different parameters and set parameter values ​​from the self-acquisition frequency, acquisition angle, acquisition window duration, camera parameters, and number of acquisition modes. It also includes parameter combinations that match the combination of operating environment data, operating speed data, and operating progress data. The sorting application is also used to adapt different data augmentation algorithms based on the work environment data, work speed data, and work progress data. The different data augmentation algorithms include: different image augmentation algorithms adapted to different work environments, data augmentation models with different structural complexities adapted to different work speeds, and different region adaptive augmentation algorithms adapted to different work progresses.

3. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, Also includes: The cargo handling application is also used to determine the current operation scenario type based on the vessel manifest, pre-allocation information, and operation plan. Using machine learning algorithms, we learn to acquire key task information and its deviation threshold for comparison at each layer under different task scenarios; and adaptively adjust the key task information and its deviation threshold for comparison at each layer according to different task scenarios. The cargo handling application is also used to determine whether the deviation of the key operation information compared at each level is greater than the deviation threshold of the key operation information. When it is greater than the deviation threshold, the operation deviation information is determined and the corresponding level of warning is triggered. If the deviation is not greater than the deviation threshold, continue to the next level of comparison until all levels of comparison are completed. If the deviation is not greater than the deviation threshold, calculate the difference between the deviation of the key operation information of the corresponding level comparison and the deviation threshold of the key operation information, and record it as the first difference. Then calculate the second difference between the absolute value of the first difference and the preset difference, and adaptively adjust the number of key operation information entering the next level according to the second difference. The smaller the second difference, the more key operation information is in the next level.

4. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, Also includes: The cargo handling application is also used to extract features from container multimodal data, container operation queue information at the current bay, actual operation data obtained from the front-end acquisition system, and historical operation deviation information, and to construct a feature set. The constructed feature set is used to train a cargo handling operation deviation prediction model. This model employs a fused temporal prediction model and a spatial correlation model, with the temporal prediction model serving as the temporal branch and the spatial correlation model as the location branch, respectively. An attention mechanism is introduced to weightedly calculate the fused feature vector and output the predicted cargo handling operation deviation for the next container. The currently collected container multimodal data, vessel manifest, pre-allocation information, operation plan, and actual operation data are input into the cargo handling operation deviation prediction model to predict the cargo handling operation deviation for the next container. Based on the predicted cargo handling operation deviation for the next container, the corresponding level of early warning is triggered in advance, completing the tiered early warning in advance.

5. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, It also includes: the backend data acquisition system; The back-end data acquisition system is used to collect operational data from back-end operators of the sorting and loading operations; The sorting application is also used to record operational deviation information and update the operational deviation frequency; when it is determined that there is an operational deviation and the current operational deviation frequency is greater than the preset deviation frequency, the current external factor influence index is evaluated; the external factor influence index is obtained by constructing a quantitative evaluation rule for the degree of influence of external factors on operational deviation based on operational environment data, operational speed data, operational progress data, and real-time operational data collected by the front-end acquisition system and operational data obtained by the back-end acquisition system, and by quantifying, evaluating and weighting the results; the evaluated current external factor influence index is compared with the preset influence index threshold, and a verification strategy is selected and executed based on the comparison result; the verification strategy includes: a fast verification process with only basic information comparison verification, an enhanced verification process with deep feature comparison verification added on the basis of fast verification, and a strict verification process with all comparison verifications.

6. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, Also includes: The tallying application is also used to construct a set of associated constraints based on the received vessel manifest, pre-allocation information, and operation plan. Each constraint includes the designed container set, the type of associated constraint, the condition of the associated constraint, and the weight. After each container has completed its operation, it searches for all constraints involving the corresponding container in the set of associated constraints. For each constraint found, if all the containers involved have completed their operations, it checks whether the corresponding constraint is satisfied and records the result. Whenever a preset number of containers are completed, it calculates the weighted satisfaction rate of the currently detected constraints and determines whether it is less than the satisfaction rate threshold. If it is less than the satisfaction rate threshold, it is determined that there is an operation deviation in the tallying operation of the current preset number of containers, generates an early warning prompt, and lists the unsatisfied constraints to the tallying client.

7. The intelligent cargo handling system for quay cranes according to claim 1, characterized in that, Also includes: The front-end acquisition system is also used to acquire multiple sets of real-time multimodal data of containers during the cargo handling process according to the control instructions of the industrial control computer, and select one set as backup multimodal data; The cargo handling application is also used to initiate fault tolerance processing when the warning level triggered during the cargo handling operation of a container is greater than the preset warning level, and to re-verify whether there is a false alarm. The fault tolerance processing includes data augmentation and multi-level comparison of the backup multi-modal data to obtain the multi-level comparison results and verify the consistency between the current multi-level comparison results and the original multi-level comparison results. If they are consistent, the operation deviation information is confirmed. If they are inconsistent, it is determined that there is an operation deviation false alarm, and the industrial control computer is controlled to generate control commands to re-collect container multi-modal data.

8. A method for intelligent cargo handling of quay cranes using the system described in any one of claims 1 to 7, characterized in that, include: Collect real-time multimodal data of containers during the tallying process; Receive the work vessel manifest, pre-allocation information and work plan transmitted from the external tallying system interface, determine and display the container operation queue information of the current bay in real time on the tallying client; Receive and manually supplement real-time multimodal data of containers; Data augmentation is performed on real-time multimodal data of containers that have been collected or manually supplemented. According to the multi-layer comparison strategy, the real-time multimodal data of containers after data augmentation processing is compared with the container operation queue information of the current bay in the transmitted operation vessel manifest, pre-allocation information, and operation plan. If all multi-level comparisons pass, a record of the current operation is generated and the progress of the cargo handling operation is updated synchronously to the external cargo handling system; if there are any failures in the multi-level comparisons, the operation deviation information is determined and the corresponding level of warning is triggered, the graded warning is completed and the operation correction suggestions are output to the cargo handling client.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method as described in claim 8.

10. A computer device, characterized in that, The computer device includes a memory, a processor, and a program stored in and executable on the memory, the program being executed by the processor to perform the steps of the method as described in claim 8.

Images