Management system for port tallying operation

Abstract

The invention discloses a port tallying operation management system, and relates to the technical field of port tallying operation management. Comprising a data acquisition module, a feature analysis module, a cargo analysis module, a safety analysis module and an early warning analysis module. The data acquisition module is used for acquiring video and image data shot in a port unloading site in real time and executing data preprocessing to form an analysis data set; according to the technical key points, a visual acquisition technology and an image recognition technology are combined, automatic and precise processing of cargo counting in the cargo ship unloading process is achieved, cargos of different specifications and forms can be precisely recognized, counting statistics is automatically completed, manual scanning and recording are not needed, and the working efficiency is improved. Counting deviation caused by artificial fatigue and visual errors is effectively avoided, and counting accuracy is improved. And the manual labor intensity of tallying personnel is remarkably reduced, the labor cost input is reduced, tallying data is ensured to be consistent with the actual loading and unloading condition, and the method has a good use prospect.

Description

Technical Field

[0001] This invention relates to the field of port tallying operation management technology, specifically to a port tallying operation management system. Background Technology

[0002] Port tallying operations, as a core foundational task in the loading, unloading, and transportation of general cargo and containers at ports, refer to the complete process by which tallying personnel, leveraging their professional skills, intelligent equipment, and operational rules, comprehensively count, verify, classify, label, and monitor the status of arriving and departing cargo. Their core responsibilities include verifying the name, category, quantity, specifications, and packaging condition of the cargo; determining the extent of damage; recording the number of hoisting operations and cargo location information; and simultaneously verifying the cargo lists provided by the shipping company and cargo owners to ensure that the actual condition of the cargo matches the information on the documents. This operation covers the entire process from ship unloading, cargo landing, and cargo stacking, requiring a balance between accuracy and timeliness. It must prevent issues such as misloading, mis-unloading, quantity discrepancies, damage, and omissions, while also providing accurate data support for port scheduling, cargo traceability, cost settlement, and safety management. It serves as a crucial link between the shipping company, cargo owners, and the port, directly impacting port operational efficiency, service quality, and the smoothness of trade.

[0003] With the rapid growth of port cargo throughput and the increasing complexity of cargo types, traditional manual cargo handling methods are no longer sufficient to meet operational requirements. There is an urgent need for a port cargo handling operation management system to provide support. Manual cargo handling suffers from problems such as poor efficiency, high counting error rate, inaccurate damage assessment, and difficulty in data traceability. Furthermore, it is difficult to achieve real-time monitoring and risk warning of the entire operation process.

[0004] The existing patent application publication number CN112633797A, entitled "An Intelligent Management System and Method for Port Cargo Operations," describes an invention patent that includes an e-commerce subsystem, a self-service terminal subsystem, a cargo handling subsystem, a weighing subsystem, a mobile commerce subsystem, a BI subsystem, and a data exchange and sharing interface module. These subsystems are interconnected via network data. The system is built according to the port's production organization process, managing cargo dispatch at the bulk cargo terminal. It organically connects port operation plans, cargo owner pickup arrangements, weighing at the weighbridge, and production processes across various departments through an information system, achieving interconnected and closed-loop management. It employs a dual-database model, ensuring operational security and continuous operation.

[0005] The solution described in the aforementioned patent has certain drawbacks. The core of this solution focuses on the comprehensive management and control of the entire process of cargo collection and distribution at the port, emphasizing general aspects such as weighing and measurement, vehicle scheduling, and data statistics, which fall under the category of post-processing. It does not cover the most common and core aspects of unloading and tallying cargo at the port, and is not specific enough for unloading and tallying cargo at the port. It is also unable to directly obtain cargo damage data, which is not conducive to the subsequent damage control and traceability work.

[0006] In summary, existing intelligent management systems for port cargo operations do not meet market demands. Therefore, we propose a new management system for port cargo handling operations. Summary of the Invention

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A port cargo handling management system includes a data acquisition module, a feature analysis module, a cargo analysis module, a safety analysis module, and an early warning analysis module.

[0009] Data acquisition module: Collects real-time video and image data from the port unloading site, performs data preprocessing, and constructs an analysis dataset;

[0010] Feature analysis module: Classifies the data in the analysis dataset, and then performs feature analysis and extraction to obtain cargo feature set and safety feature set;

[0011] Cargo Analysis Module: Based on preset cargo handling rules and convolutional neural networks, it processes data in the cargo feature set, performs accounting and calibration, and obtains a cargo list;

[0012] Security Analysis Module: Based on recurrent neural networks, it processes security feature sets and historical risk case datasets to obtain security risk scores, and assesses risk levels based on these scores.

[0013] Early warning analysis module: Obtains risk level and cargo list, executes corresponding early warning and emergency response strategies based on risk level, compares cargo list with original list, performs cargo statistical analysis, and if cargo statistical analysis does not meet preset indicators, re-executes accounting and calibration until termination conditions are met.

[0014] Preferably, the video and image data captured at the port unloading site includes: images, videos, and numerical data collected by all loading and unloading equipment and recording equipment during the unloading operation; overall appearance images of the cargo captured by a wide-angle camera, side detail images of the cargo captured by a high-speed capture camera, and full-process videos during the tallying operation; three-dimensional dimension data of the cargo captured by a 3D structured light camera during the tallying and unloading operations, close-up images of the cargo binding parts captured by a macro high-definition camera, and simultaneous recording of dynamic short videos of the binding parts; and monitoring videos recorded by an infrared thermal imaging camera that dynamically monitors the safety area in real time.

[0015] Preferably, the preprocessing for performing data classification includes preprocessing images and videos as well as preprocessing numerical data;

[0016] Image and video preprocessing: The image and video filtering and recognition algorithm is run to process video and image data, and invalid video and image data that do not meet the preset filtering rules are removed; According to the preset image and video processing rules, the filtered video and image data are processed to extract several sets of visual features. The visual features are processed through the preset state classification scheme, the data type of each time node is analyzed and labeled, key nodes are set, and N frames of valid images and videos of cargo lifting, operation process, landing moment and key nodes are retained. The data types include mutation, critical and normal.

[0017] The preprocessing of numerical data includes: comparing the numerical values ​​with preset filtering thresholds, removing numerical data that do not meet the filtering thresholds, then extracting the numerical data corresponding to the lifting, operation, landing moment and key nodes of the cargo, performing association and binding operations between the numerical values ​​and N frames of valid images and videos, summarizing the associated and bound data, and constructing an analysis dataset.

[0018] Preferably, the data in the analysis dataset is classified and preprocessed to remove redundant identifiers, fill in missing hoisting numbers and collection timestamps, and then use label coding technology to supplement data source identifiers. The data is then classified and organized according to cargo data, posture data, binding data, personnel data, and auxiliary data. Feature association mapping technology is used to group cargo data and auxiliary data into a cargo dataset, and posture data, binding data, and personnel data into a safety dataset. Hoisting number association technology is used to bind all data in the cargo dataset and safety dataset with their corresponding hoisting numbers, thus constructing an identification system.

[0019] Preferably, the feature analysis and extraction of the cargo dataset includes: using texture matching algorithms, contour feature extraction algorithms, contour clustering algorithms, and multi-frame consensus algorithms to extract data from the cargo dataset, obtaining cargo packaging type, cargo identification data, cargo size data, cargo coordinates, and cargo spacing to form a cargo feature vector; separately labeling data with abrupt changes in data type; extracting data with key data type to obtain quantitative feature data; using edge detection algorithms and area proportion algorithms to extract data from the labeled data to obtain data on packaging damage, percentage of damaged area, and damage type; extracting data from the labeled data using coordinate matching algorithms to obtain the cargo location number after the cargo is landed and the relative position information between the cargo and the cargo location boundary; summarizing the data obtained from the feature analysis and extraction of the cargo dataset, binding the hoisting number and data type, to form a cargo feature set.

[0020] Preferably, the feature analysis and extraction of the safety dataset includes: using a 3D point cloud simplification algorithm and a coordinate calculation algorithm to extract data of key data types and data of abrupt changes selected according to a preset ratio, to obtain the cargo tilt angle, center of gravity offset, and swing amplitude; using a combination of target counting algorithm and gap measurement algorithm to extract data of abrupt changes, to obtain the number of bindings, the maximum gap value of tightness, and the damage mark of binding parts; using a coordinate calibration algorithm and a distance calculation algorithm to extract data of abrupt changes, to obtain the safety area boundary coordinates, personnel-related coordinates, and the shortest distance between personnel and the running trajectory of the lifting equipment; summarizing the data obtained from the feature analysis and extraction of the safety dataset, binding the lifting number with the data type, and thus obtaining the safety feature set.

[0021] Preferably, the data processing based on preset cargo handling rules and convolutional neural networks in the cargo feature set includes: performing normalization processing on the feature data in the cargo feature set to obtain a normalized cargo handling core feature matrix, and removing data with normal data types from the cargo dataset; using a convolutional neural network to compare the cargo handling core feature matrix, the preset cargo handling rules, and the cargo dataset after removing normal data to obtain cargo handling results, anomaly reports, parameter comparison tables, and data lists, conducting verification work, and generating a cargo list after verification and calibration.

[0022] Preferably, the process of processing the safety feature set and historical risk case dataset based on recurrent neural networks includes: performing standardization processing on the feature data in the safety feature set to obtain a standardized safety core feature matrix, and removing data in the safety dataset that are of normal data type; using feature label matching technology to associate the cargo feature vector and quantity feature data in the cargo feature set; and using a recurrent neural network to compare the safety core feature matrix with the historical risk case dataset, performing weighted scoring, and obtaining a safety risk score.

[0023] Preferably, the risk level assessment based on the security risk score includes: mapping the security risk score to a preliminary risk level according to the corresponding mapping rules, verifying the authenticity of the abnormal features by comparing the corresponding images and videos with the mutation data, and after the verification is passed, analyzing the number of abnormal features. If there is a single abnormal feature, the preliminary risk level is determined to be a risk level.

[0024] If multiple abnormal features are present, the risk level upgrade mechanism will be triggered, the initial risk level will be upgraded, and the upgraded initial risk level will be determined as the risk level.

[0025] If the verification fails, the security risk score will be recalculated.

[0026] The preferred early warning and emergency response strategy is as follows: When the risk is determined to be no, no action is required; when the risk is determined to be low, the yellow indicator light on the terminal device flashes and is accompanied by a buzzer; when the risk is determined to be medium, the orange indicator light on the terminal device remains on, a continuous buzzer is emitted, and a reminder window pops up with corresponding images and videos of the mutation data as supporting evidence, and the cause of the anomaly is shown; when the risk is determined to be high, the red indicator light on the terminal device flashes loudly, a high-decibel alarm is issued, an emergency stop or deceleration signal is sent to the lifting device, and detailed information on the mutation data, complete image and video evidence, the urgency of the risk, and the cause of the anomaly are pushed.

[0027] This invention provides a port tallying operation management system, which has the following beneficial effects:

[0028] This invention combines visual acquisition technology with image recognition technology to solve the problems of high deviation rate, data entry delay, and high labor intensity caused by relying mainly on manual counting in existing technologies. It realizes automated and accurate cargo counting during the unloading process of cargo ships. By collecting image information in the unloading scene and using deep learning algorithms to extract features and identify cargo, this invention can accurately identify cargo of different specifications and shapes, automatically complete the counting statistics, and automatically associate lifting information with cargo specification information. It eliminates the need for manual scanning and data entry, effectively avoiding counting deviations caused by human fatigue and visual errors, and improving counting accuracy. It significantly reduces the labor intensity of tallying personnel, reduces labor costs, and ensures that tallying data matches the actual loading and unloading situation, providing accurate data support for subsequent cargo verification and payment settlement.

[0029] This invention utilizes machine vision inspection technology to solve problems such as untimely and inaccurate damage identification, difficulty in tracing damage links, and difficulty in defining responsibility in the current cargo unloading and tallying process. It achieves real-time identification, graded judgment, and full-process traceability of damaged goods. By collecting images and video data in real time during the loading and unloading process through cameras, and using machine vision inspection algorithms, it identifies damage conditions such as breakage, collision, dampness, and deformation on the surface of the goods, generates damage reports in real time, and pushes them to relevant personnel so that remedial measures can be taken in a timely manner. Compared with the manual photo recording mode used in the prior art, it shortens the response time of damage identification, improves the accuracy of damage judgment, effectively reduces the omission of damaged goods, provides a basis for claims and rectification of damaged goods, and reduces the incidence of disputes between ports, cargo owners, and shipping companies.

[0030] This invention combines 3D structured light visual inspection technology with recurrent neural network technology to solve problems in existing technologies such as the lack of a dedicated safety management and control mechanism for shipside unloading and tallying, delayed early warnings, inaccurate risk level assessments, and inability to prevent potential hazards in advance. It achieves real-time monitoring, precise early warning, and scientific management of safety risks throughout the entire shipside unloading and tallying process. It collects real-time three-dimensional data such as cargo posture, binding status, and personnel location, and uses an infrared thermal imaging camera to capture potential risks. Recurrent neural network technology extracts features and identifies anomalies, and combined with preset standards and historical cases, calculates risk scores and determines risk levels. It accurately identifies various potential hazards and automatically triggers corresponding early warning mechanisms. Compared with existing solutions, it can shorten the identification response time, improve the accuracy of judgment, provide early warnings of potential hazards, avoid safety accidents, and record the entire risk handling process. This enhances the refinement of safety management, reduces port operation risks, and ensures the safety of personnel, cargo, and equipment, meeting the safety management and control needs of port cargo shipside operations.

[0031] This invention synergistically utilizes visual acquisition, deep learning, 3D structured light detection, and recurrent neural network technologies to overcome the shortcomings of existing patents in port cargo ship unloading and tallying, such as high reliance on manual labor, poor operational coordination, difficulty in damage tracing, and delayed safety warnings. It achieves intelligent, refined, efficient, and safe management of the entire port cargo ship unloading and tallying process. This invention achieves automated and accurate counting through the combination of visual acquisition and deep learning, solving the problem of manual counting errors; it combines machine vision and blockchain to achieve real-time damage identification and full-process traceability, clarifying responsibilities; and it combines 3D structured light and recurrent neural network technologies to achieve real-time monitoring and accurate early warning of safety risks, preventing accidents. This invention automates the entire process of cargo loading, unloading, and counting, reducing manual intervention, lowering labor intensity and costs, improving the accuracy of tallying data, operational coordination, the effectiveness of damage control, and the ability to prevent safety risks. It enhances various accuracy rates and operational efficiency, avoids safety accidents and operational disputes, standardizes operational processes, compensates for the shortcomings of existing patents, promotes the digital transformation of tallying, provides decision-making support for port management, enhances the core competitiveness of port cargo ship unloading and tallying, and adapts to the needs of port development. Attached Figure Description

[0032] Figure 1 This is a flowchart of a port tallying operation management system according to the present invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Example:

[0035] Please see Figure 1 This embodiment provides a port tallying operation management system, including:

[0036] Data acquisition module: Collects real-time video and image data from the port unloading site, performs data preprocessing, and constructs an analysis dataset;

[0037] The port's cargo handling management system is connected to the existing port management system, allowing direct access to staff information entered at the shipside operation terminal and various basic information about the current cargo ship. This basic information includes the ship's name, sailing number, hatch numbers, main cargo types, detailed lists of mixed cargo types, weight of individual cargo pieces, cargo packaging type, and the number and type of equipment used for cargo retrieval. This facilitates the use of surrounding data collection devices to obtain corresponding data and allows for the determination of the safety zone boundaries based on the equipment used for cargo retrieval, providing convenience for subsequent safety analysis.

[0038] For example, the equipment used to lift and retrieve cargo includes quay container cranes, gantry cranes, etc.

[0039] The port's cargo handling management system will automatically synchronize with the cargo visual feature library already stored in the terminal production management system; if the library does not exist, it will obtain historical data to build it; the cargo visual feature library includes the packaging texture features of bagged goods, the size range standard of boxed goods, the critical value of tilt angle corresponding to different cargo types, the standard of binding and fixing quantity, the radius range of the safe operating area, and is loaded with existing basic image and video recognition models.

[0040] The data acquisition module connects to the corresponding acquisition devices around the equipment used to lift goods, and automatically acquires the data. Before acquiring data, the data acquisition module sends a calibration command to the acquisition device. The acquisition device adjusts the focal length, exposure, and shooting angle according to the lifting requirements to ensure that the acquisition angle and clarity meet the operational requirements, thereby ensuring the recognition accuracy of images and videos.

[0041] The data acquisition module collects real-time video and image data from the port unloading site. During the acquisition process, basic screening is carried out. This screening is based on pre-set minimum resolution requirements and minimum target coverage. Data that does not meet the requirements will be removed, thereby reducing the amount of data to be processed in the subsequent stages.

[0042] The data acquisition module's filtering also includes removing duplicate image frames for image data; removing invalid numerical data for numerical data, with valid data defined as lifting weights exceeding 10 kg and operating speeds exceeding 0.2 m / s, and other data being invalid; and removing videos shorter than 3 seconds, blurry videos, or videos without a target, as the absence of a target indicates no cargo was lifted. This method can significantly reduce the amount of data collected, thereby reducing the amount of data required for subsequent calculations.

[0043] The video and image data captured at the port unloading site includes: images, videos, and numerical data collected by all loading and unloading equipment and recording equipment during the unloading operation; in order to reduce the amount of invalid data, the acquisition equipment will only be activated when the loading and unloading equipment detects that the lifting weight is greater than 0, mainly to achieve comprehensive acquisition of multiple types of data such as images, videos, and numerical data, and improve the accuracy of subsequent analysis.

[0044] The lifting sensors on the loading and unloading equipment collect auxiliary numerical data related to the operation of the equipment and the lifting of goods, such as lifting weight, running speed, and lifting height. They are also linked to the corresponding time period's image and video data to achieve linkage between numerical and audio-visual data, which facilitates subsequent verification.

[0045] During the cargo handling process, wide-angle cameras capture overall images of the cargo's appearance, high-speed capture cameras capture detailed side images of the cargo, and the entire process video is recorded. The main focus is on capturing key information such as cargo packaging, labeling, and quantity, providing both image and video support for subsequent cargo identification, damage assessment, and data type determination.

[0046] During cargo handling and unloading operations, the 3D structured light camera collects three-dimensional dimensional data of the cargo, the macro high-definition camera captures close-up images of the cargo binding parts, and simultaneously records short dynamic videos of the binding parts, and the infrared thermal imaging camera monitors the safety area in real time and records monitoring videos. The main data collected is safety-related data.

[0047] The data acquisition equipment and sensors utilize the port's internal network to transmit data in real time to the data acquisition module. This module employs an edge computing terminal to enable synchronous computation across different ports. During data transmission, advanced encryption standards (256-bit encryption) are used to ensure data transmission security and prevent data leakage or tampering. Simultaneously, transmitted images and videos undergo preliminary compression to improve transmission efficiency and avoid data congestion.

[0048] Preprocessing for data classification includes preprocessing images and videos, as well as preprocessing numerical data;

[0049] Image and video preprocessing: The image and video filtering and recognition algorithm is run to process the video and image data, and invalid video and image data that do not meet the preset filtering rules are removed; that is, the minimum clarity requirements and the minimum target coverage mentioned above are removed, and videos with a duration of less than 3 seconds, blurry images or no target are deleted; no target means no goods are being picked up. This is based on machine vision technology using the image and video filtering and recognition algorithm, so it will not be described in detail.

[0050] Based on the preset image and video processing rules, the filtered video and image data are processed for recognition and several sets of visual features are extracted. This step utilizes the existing image and video recognition model and extracts visual features such as the packaging status of goods, the tightness of binding, the posture angle, and the situation of personnel in the safe area.

[0051] Several sets of visual features are processed by a preset state classification scheme, the data types of each time node are analyzed and labeled, key nodes are set, and N frames of valid images and videos of cargo lifting, operation process, landing moment and key nodes are retained. The data types include mutation, critical and normal.

[0052] Sudden changes occur when numerical data exceeds preset thresholds, such as cargo tilt angle > 3°, insufficient binding quantity, lifting weight fluctuation > ±2%; image and video recognition results show damaged cargo packaging, loose binding, abnormal posture, personnel entering the safe area, or abnormal equipment status; or the fluctuation difference between a single set of data and adjacent data is > 5%.

[0053] The key data is the data collected during the lifting, operation, and landing of the goods, excluding any sudden changes. The normal data is the data excluding any sudden changes and key data. Key nodes are defined using an equal division method, that is, the time point reached at each time interval is designated as a key node.

[0054] The preprocessing of numerical data includes: comparing the numerical values ​​with preset filtering thresholds, removing numerical data that do not meet the filtering thresholds, namely, empty data with a lifting weight of less than 10 kg, data collected when the equipment is in abnormal operation, and repeatedly reported stable values; then extracting the numerical data corresponding to the lifting, operation, landing moment, and key nodes of the cargo, performing association and binding operations between the numerical values ​​and N frames of valid images and videos, summarizing the associated and bound data, and constructing an analysis dataset.

[0055] The analysis dataset includes numerical data at mutation time points, short videos and image data before and after mutation time points, numerical data at key nodes, short videos and image data before and after key nodes, and some normal data.

[0056] Short videos are typically set to 5 seconds to reduce the amount of data processing required.

[0057] The normal data in the analysis dataset consists of time points randomly selected from the time of mutation and the period before and after key nodes. Then, the numerical data, image data and short videos before and after the time point are obtained to form the normal data.

[0058] The association and binding operation between numerical values ​​and N frames of valid images and videos is mainly aimed at the association and binding of mutation time points and key nodes.

[0059] For the collected and analyzed data, edge computing technology is used to perform caching operations after classification and coding to ensure that the data classification conforms to the specifications, the caching is secure, and the transmission is efficient.

[0060] When caching, mutation data has a higher priority than critical data, and critical data has a higher priority than normal data. During data transmission, high-speed transmission protocols are used to transmit mutation data and critical data. At this time, mutation data and critical data are transmitted synchronously, mainly for the convenience of comparison. Normal data can be transmitted using conventional transmission protocols.

[0061] This invention combines visual acquisition technology with image recognition technology to solve the problems of high deviation rate, data entry delay, and high labor intensity caused by relying mainly on manual counting in existing technologies. It realizes automated and accurate cargo counting during the unloading process of cargo ships. By collecting image information in the unloading scene and using deep learning algorithms to extract features and identify cargo, this invention can accurately identify cargo of different specifications and shapes, automatically complete the counting statistics, and automatically associate lifting information with cargo specification information. It eliminates the need for manual scanning and data entry, effectively avoiding counting deviations caused by human fatigue and visual errors, and improving counting accuracy. It significantly reduces the labor intensity of tallying personnel, reduces labor costs, and ensures that tallying data matches the actual loading and unloading situation, providing accurate data support for subsequent cargo verification and payment settlement.

[0062] Feature analysis module: Classifies the data in the analysis dataset, and then performs feature analysis and extraction to obtain cargo feature set and safety feature set;

[0063] The data in the analysis dataset is classified, which involves preprocessing the data, removing redundant labels, and filling in missing hoisting numbers and collection timestamps to ensure data integrity and accuracy for subsequent analysis. Then, label coding technology is used to supplement data source identifiers for easy traceability. Next, the data is classified and organized according to cargo data, posture data, binding data, personnel data, and auxiliary data to achieve initial data organization. Feature association mapping technology is used for classification and summarization. During classification and summarization, a feature association algorithm is used to group cargo data and auxiliary data into a cargo dataset, and posture data, binding data, and personnel data into a safety dataset. Hoisting number association technology is used to bind all data in the cargo dataset and safety dataset with their corresponding hoisting numbers, constructing an identification system.

[0064] The identification system includes the hoisting number, timestamp, data category, feature set type, and data type. The data categories are video, image, and numerical data.

[0065] The feature analysis and extraction process for the cargo dataset includes: using texture matching algorithms, contour feature extraction algorithms, contour clustering algorithms, and multi-frame consensus algorithms to extract data from the cargo dataset, obtaining cargo packaging type, cargo identification data, cargo size data, cargo coordinates, and cargo spacing to form a cargo feature vector; separately labeling data with abrupt changes in data type, and performing data calibration by combining key data from corresponding adjacent time points; extracting key data of data type to obtain quantitative feature data; using edge detection algorithms and area proportion algorithms to extract data from the labeled data, obtaining data on packaging damage, the proportion of damaged area, and damage type using edge detection algorithms, which can be compared and verified by extracting packaging features from key data; and extracting data from the labeled data using coordinate matching algorithms to obtain the cargo location number after the cargo is landed and the relative position information between the cargo and the cargo location boundary. The data obtained from the feature analysis and extraction of the cargo dataset are then summarized, bound to the hoisting number and data type to form a cargo feature set.

[0066] The cargo characteristics stored in a centralized manner include 12 items: cargo packaging type, cargo identification data, cargo size data, cargo coordinates, cargo spacing, cargo feature vector, quantity feature data, packaging damage data, percentage of damaged area, damage type data, cargo location number after landing, and relative position information between cargo and cargo location boundary.

[0067] Using a 3D point cloud simplification algorithm and coordinate calculation algorithm, key data and abrupt change data, selected according to a preset ratio, are extracted to obtain the cargo tilt angle, center of gravity offset, and swing amplitude. The abrupt change data is compared with the core data to obtain key parameters of abnormal posture. A combination of target counting and gap measurement algorithms is used to extract abrupt change data to obtain the number of bindings, maximum gap value of tightness, and damage indicators of binding locations. The abrupt change data is compared with the core data to obtain binding safety anomaly indicators. Coordinate calibration and distance calculation algorithms are used to extract abrupt change data to obtain the safety zone boundary coordinates, personnel-related coordinates, and the shortest distance between personnel and the lifting equipment's trajectory. The safety dataset is then summarized, and feature analysis and extraction are performed. The lifting order number is bound to the data type to obtain a safety feature set.

[0068] The safety features stored centrally include eight items: cargo tilt angle, center of gravity offset, swing amplitude, number of bindings, maximum gap value of tightness, as well as markings of damage to binding parts, abnormal binding safety indicators, boundary coordinates of the safety area, and the shortest distance between personnel and the lifting equipment.

[0069] The indicators from the cargo feature set and the security feature set are aggregated and then compressed to remove redundant fields, thereby reducing the data volume, achieving lightweight data processing, and improving data transmission and storage efficiency. At the same time, the feature vectors corresponding to the mutation data are encrypted to ensure the security of abnormal data.

[0070] In practical use, the main focus is on verifying the completeness, accuracy, and standardization of the feature vectors; feature vectors corresponding to mutation data are verified first, with particular emphasis on verifying the accuracy of anomalous features.

[0071] If a feature vector is found to have missing features, abnormal parameters, or incorrect labeling, the system will immediately trigger a feature acquisition instruction. At this time, the corresponding data acquisition device needs to re-capture the image or acquire the data and re-extract the relevant features. After the data acquisition is completed, the normal / mutation label needs to be re-labeled. If the acquisition operation fails, it will be marked as abnormal and associated with the original data of the corresponding hoisting for review.

[0072] Validated feature vectors will proceed to the next stage of comprehensive analysis; feature vectors that fail validation will be marked as feature anomalies and associated with the corresponding normal / mutation identifiers and original image / video data, awaiting manual review. During manual review, feature anomalies related to mutation data should be prioritized to improve review efficiency.

[0073] Cargo Analysis Module: Based on preset cargo handling rules and convolutional neural networks, it processes data in the cargo feature set, performs accounting and calibration, and obtains a cargo list;

[0074] The data processing based on preset cargo handling rules and convolutional neural networks includes: normalizing the feature data in the cargo feature set, unifying the units and value ranges of feature parameters to obtain a normalized core feature matrix for cargo handling, and removing data with normal data types from the cargo dataset; using convolutional neural networks to compare the core feature matrix for cargo handling, the preset cargo handling rules, and the cargo dataset after removing normal data. Specifically, the convolutional neural network is used to compare the physical cargo features of similar historical cargo, combined with the preset cargo handling rules, to determine the specific category of the cargo and label key data of the cargo category; based on quantitative feature data, cargo coordinates, and cargo spacing, a... The contour matching algorithm, combined with cargo counting rules, completes the preliminary statistics of cargo quantity; it focuses on handling quantity anomalies involved in sudden data changes, comparing key data and historical similar sudden data cases to correct the deviation in quantity accounting; based on packaging damage data and the proportion of damaged area, the area proportion analysis algorithm is used, combined with the damage standards in the preset cargo handling rules, to determine the damage status of the cargo, and mark the damage type, damage degree and damage location; based on the cargo location association characteristics, it matches the lifting information to verify the accuracy of the cargo landing location, associates the cargo location coordinate information, and obtains the cargo handling results, anomaly report, parameter comparison table and data list, and conducts verification work. After verification and calibration, the cargo list is generated.

[0075] When conducting verification work, a weight re-verification is required. Specifically, the quantity of goods being analyzed is multiplied by the standard weight of a single item, and the result is compared with the lifting weight collected by the lifting device's sensors to verify whether the deviation between the two is within a preset threshold range. For data lists with inconsistent weight re-verification results, it is necessary to call up abrupt change data, along with corresponding audio and video evidence, packaging damage data, damaged area percentage, damage type data, the cargo location number after the goods are landed, and the relative position information between the goods and the cargo location boundary. Using a convolutional neural network backpropagation algorithm, and combining it with historical similar data cases, the core reasons for the deviation are analyzed, such as weight deviation due to damage, information anomalies caused by cargo location offset, and collection errors. Then, a second correction is completed. Combined with the verification and comparison of key data, the final verification of the cargo handling results is performed to ensure that the four parameters of cargo category, quantity, damage, and cargo location are consistent, and that the data type labeling is accurate.

[0076] This invention utilizes machine vision inspection technology to solve problems such as untimely and inaccurate damage identification, difficulty in tracing damage links, and difficulty in defining responsibility in the current cargo unloading and tallying process. It achieves real-time identification, graded judgment, and full-process traceability of damaged goods. By collecting images and video data in real time during the loading and unloading process through cameras, and using machine vision inspection algorithms, it identifies damage conditions such as breakage, collision, dampness, and deformation on the surface of the goods, generates damage reports in real time, and pushes them to relevant personnel so that remedial measures can be taken in a timely manner. Compared with the manual photo recording mode used in the prior art, it shortens the response time of damage identification, improves the accuracy of damage judgment, effectively reduces the omission of damaged goods, provides a basis for claims and rectification of damaged goods, and reduces the incidence of disputes between ports, cargo owners, and shipping companies.

[0077] Security Analysis Module: Based on recurrent neural networks, it processes security feature sets and historical risk case datasets to obtain security risk scores, and assesses risk levels based on these scores.

[0078] The process of processing safety feature sets and historical risk case datasets using recurrent neural networks includes: standardizing the feature data in the safety feature set, unifying the units and value ranges of feature parameters to obtain a standardized core safety feature matrix, and removing data with normal data types from the safety dataset; using feature label matching technology to associate cargo feature vectors and quantity feature data from the cargo feature set; filtering and loading a preset safety feature weight allocation standard, specifically 40% for personnel intrusion, 35% for binding security, and 25% for posture security; and using a recurrent neural network to compare the core safety feature matrix with the historical risk case dataset for weighted scoring. Specifically, the recurrent neural network iteratively calculates the corresponding scores for each safety feature by comparing it with historical risk cases, and then uses a weighted summation algorithm according to the preset weight allocation standard to calculate the safety risk score.

[0079] The risk level assessment based on the security risk score includes: mapping the security risk score to the preliminary risk level according to the corresponding mapping rules, verifying the authenticity of the abnormal features by comparing the corresponding images and videos with the mutation data, and after the verification is passed, analyzing the number of abnormal features. If there is a single abnormal feature, the preliminary risk level is determined to be a risk level.

[0080] If multiple abnormal features exist, such as the simultaneous presence of loose bindings and tilted cargo, the risk level upgrade mechanism will be triggered, the initial risk level will be upgraded, and the upgraded initial risk level will be determined as the risk level.

[0081] If the verification fails, the security risk score will be recalculated.

[0082] After verifying the authenticity of the abnormal characteristics, we can also refer to the risk handling effect data of similar scenarios in the past to review and calibrate the current risk level, so as to ensure that the assessment results are consistent with the actual operation scenario. At the same time, we can combine normal data to verify the rationality of the risk level and eliminate misjudgments caused by extreme data.

[0083] When assessing the risk level, a risk assessment report can be generated simultaneously. The risk assessment report includes a safety risk score, risk level, calculation process, mapping process between score and level, correction and verification process, etc., to facilitate subsequent assessment.

[0084] This invention combines 3D structured light visual inspection technology with recurrent neural network technology to solve problems in existing technologies such as the lack of a dedicated safety management and control mechanism for shipside unloading and tallying, delayed early warnings, inaccurate risk level assessments, and inability to prevent potential hazards in advance. It achieves real-time monitoring, precise early warning, and scientific management of safety risks throughout the entire shipside unloading and tallying process. It collects real-time three-dimensional data such as cargo posture, binding status, and personnel location, and uses an infrared thermal imaging camera to capture potential risks. Recurrent neural network technology extracts features and identifies anomalies, and combined with preset standards and historical cases, calculates risk scores and determines risk levels. It accurately identifies various potential hazards and automatically triggers corresponding early warning mechanisms. Compared with existing solutions, it can shorten the identification response time, improve the accuracy of judgment, provide early warnings of potential hazards, avoid safety accidents, and record the entire risk handling process. This enhances the refinement of safety management, reduces port operation risks, and ensures the safety of personnel, cargo, and equipment, meeting the safety management and control needs of port cargo shipside operations.

[0085] Early warning analysis module: Obtains risk level and cargo list, executes corresponding early warning and emergency response strategies based on risk level, compares cargo list with original list, performs cargo statistical analysis, and if cargo statistical analysis does not meet preset indicators, re-executes accounting and calibration until termination conditions are met.

[0086] The early warning and emergency response strategies are as follows: When the risk is determined to be no, no action is required; when the risk is determined to be low, the yellow indicator light on the terminal device will flash, accompanied by a buzzer; when the risk is determined to be medium, the orange indicator light on the terminal device will remain on, continuously emit a buzzer, and simultaneously pop up a reminder window with corresponding images and videos of the mutation data as supporting evidence, and show the cause of the anomaly; when the risk is determined to be high, the red indicator light on the terminal device will flash loudly, issue a high-decibel alarm, send an emergency stop or deceleration signal to the lifting device, and push detailed information on the mutation data, complete image and video evidence, the urgency of the risk, and the cause of the anomaly.

[0087] When in use, this system will synchronize cargo handling results and safety status information in real time with the terminal production management system, ship terminal equipment, and cargo owner remote terminal equipment. At the same time, it will push abnormal information, supporting materials, and brief analysis results of deep learning algorithms related to mutation data. In addition, this system will also synchronize data related to safety risks, mutation data, and algorithm early warning records with the terminal safety management system to provide data support for safety management work, so that managers can grasp the abnormal situation of operations and the running effect of the algorithm in real time.

[0088] After use, the cloud database categorizes and stores various types of data. The specific stored content includes compressed feature vectors, final cargo handling results, safety risk records, data from the analysis process of deep learning algorithms, and images and videos of key and mutation events. During storage, the data is categorized and named according to rules such as ship name, voyage number, unloading date, crane number, and normal / mutation identifiers to facilitate quick querying and traceability.

[0089] The data is stored in an off-site backup manner, and a full data backup is completed every day in the early morning.

[0090] During use, the system monitors various abnormal situations in real time, focusing on anomalies corresponding to sudden data changes. It also monitors anomalies such as cargo handling data conflicts and data acquisition equipment malfunctions. Once an anomaly is detected, it automatically marks the anomaly type, the associated hoisting number, and the corresponding normal / sudden change identifier. The anomaly information is then pushed to the cargo handling personnel's handheld tablet and the dispatcher's terminal device. The push information clearly marks the location of the anomaly, the anomaly type, the associated normal / sudden change data identifier, and the corresponding key images and video clips, so that staff can quickly arrive at the scene to verify the situation.

[0091] After receiving an anomaly warning, the inventory personnel must go to the site of the anomaly within a set time to verify the actual situation. They should prioritize handling high-risk anomalies related to mutation data. After verification, they should use a handheld tablet to take pictures of the site, record short videos, and upload them to the system, marking the supplementary data with normal / mutation labels to provide a basis for anomaly handling.

[0092] After on-site verification, the inventory personnel enter the actual test data and label the data as normal / abnormal. The system, combined with the on-site images and videos uploaded by the inventory personnel, re-extracts features, corrects the inventory results and risk levels, and updates the normal / abnormal labels of the corresponding data to ensure data consistency. Among these, anomalies related to abrupt changes are prioritized for handling and correction. If equipment-related anomalies occur, the system switches to the backup data collection plan to ensure the continuity of normal / abnormal data collection. After receiving equipment repair reports, maintenance personnel must arrive on-site within a set time to repair the equipment. During the repair, they use a handheld tablet to collect relevant data, take pictures and videos, and manually label the data as normal / abnormal to avoid data loss.

[0093] When the risk level is medium, the dispatcher sends a deceleration command to the spreader through the terminal device. The tally personnel arrive at the site to handle the situation. After the situation is handled, the system re-collects relevant data, takes pictures and videos, re-identifies the normal / abrupt status of the data, reassesses the safety status, and resumes normal operation after confirming that the risk has been eliminated.

[0094] When the risk level is high, the spreader immediately stops operating to prevent safety accidents. The tallying personnel and dispatchers arrive at the scene together to handle the situation. After the hidden danger is completely eliminated, the system re-collects data, takes pictures and videos, identifies and marks them as key data, reassesses the safety status, and restarts the unloading and tallying operation after confirming that there are no errors.

[0095] After an anomaly is handled, the tallying personnel enter the detailed handling process and results into the terminal device, and mark the data type changes before and after the handling. The system automatically links all relevant data for that anomaly, including the abnormal data before handling, the key data after handling, the original images and videos, feature vectors, and early warning records, to generate an anomaly handling record table. The record table contains detailed information such as the anomaly type, the time of the anomaly, the handling time, the handling personnel, the handling process, the result verification, and the data type changes. The anomaly handling record table is synchronized to the cloud database for archiving and storage, and is included as a data source for work review and analysis. It is mainly used as a case study for handling abnormal data anomalies, providing a reference for subsequent optimization of anomaly prevention and control measures and adjustment of the classification standards for normal / abnormal data.

[0096] After all cargo handling and unloading operations are completed, the system automatically summarizes all relevant data from all lifting operations on the vessel, focusing on abrupt changes and key data, and generates cargo handling statistics reports and safety statistics reports. The cargo handling statistics reports include key indicators such as the summary of cargo types and quantities, cargo damage rate, operational efficiency, and cargo handling deviations caused by abrupt changes. The safety statistics reports include key information such as the distribution of safety risk levels, the number of abnormal handling incidents, high-frequency abrupt change data types, and the proportion of risks caused by abrupt changes. Normal data is only used for sampling analysis and is not included in the main statistical indicators.

[0097] The system analyzes the operational data of all acquisition devices, and combines the acquisition of normal / abnormal data to statistically analyze indicators such as device failure rate, image and video recognition accuracy, and data acquisition accuracy. It then marks a list of devices that require maintenance and notifies maintenance personnel to carry out maintenance work in a timely manner. At the same time, it analyzes the recognition effect of the image and video recognition model, identifies the weak links in the recognition of high-frequency abrupt data, and marks optimization directions for subsequent optimization.

[0098] This invention synergistically utilizes visual acquisition, deep learning, 3D structured light detection, and recurrent neural network technologies to overcome the shortcomings of existing patents in port cargo ship unloading and tallying, such as high reliance on manual labor, poor operational coordination, difficulty in damage tracing, and delayed safety warnings. It achieves intelligent, refined, efficient, and safe management of the entire port cargo ship unloading and tallying process. This invention achieves automated and accurate counting through the combination of visual acquisition and deep learning, solving the problem of manual counting errors; it combines machine vision and blockchain to achieve real-time damage identification and full-process traceability, clarifying responsibilities; and it combines 3D structured light and recurrent neural network technologies to achieve real-time monitoring and accurate early warning of safety risks, preventing accidents. This invention automates the entire process of cargo loading, unloading, and counting, reducing manual intervention, lowering labor intensity and costs, improving the accuracy of tallying data, operational coordination, the effectiveness of damage control, and the ability to prevent safety risks. It enhances various accuracy rates and operational efficiency, avoids safety accidents and operational disputes, standardizes operational processes, compensates for the shortcomings of existing patents, promotes the digital transformation of tallying, provides decision-making support for port management, enhances the core competitiveness of port cargo ship unloading and tallying, and adapts to the needs of port development.

[0099] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented in software, the above embodiments can be implemented, in whole or in part, as a computer program product. Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution.

[0100] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0101] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A port tallying operation management system, characterized by, Comprise: Data acquisition module: collect real-time port unloading site shooting video and image data, and perform data preprocessing to form analysis dataset; Feature analysis module: classify the data in the analysis dataset, then perform feature analysis and extraction to obtain cargo feature set and safety feature set; Cargo analysis module: process the data in the cargo feature set based on the preset cargo tally rules and convolutional neural network, perform accounting calibration to obtain the cargo manifest; Safety analysis module: process the safety feature set and historical risk case dataset based on the recurrent neural network to obtain the safety risk score, and evaluate the risk level based on the safety risk score; Early warning analysis module: obtain the risk level and cargo manifest, execute the corresponding early warning and emergency handling strategy based on the risk level, compare the cargo manifest with the original manifest, perform cargo statistical analysis, if the cargo statistical analysis does not meet the preset indicators, re-execute the accounting calibration until the termination condition is met.

2. The port tally management system according to claim 1, wherein: The video and image data shot at the port unloading site include: image, video and numerical data collected by all handling equipment and recording equipment during unloading operation; cargo overall appearance image collected by wide-angle camera, cargo side detail image and whole process video shot by high-speed snapshot camera during tallying operation; three-dimensional size data of cargo collected by 3D structured light camera, close-up image of cargo binding part shot by macro HD camera, and monitoring video recorded by infrared thermal imaging camera for real-time monitoring of safety area.

3. The port tally management system according to claim 1, wherein: The preprocessing for data classification includes preprocessing of image and video and preprocessing of numerical data; Preprocessing of image and video: running image and video screening and recognition algorithm to process video and image data, and eliminating invalid video and image data that do not meet the preset screening rules; According to the preset picture video processing rules, the screened video and image data are subjected to recognition processing, a plurality of groups of visual features are extracted, the plurality of groups of visual features are processed through the preset state classification scheme, the data types at each time node are analyzed and marked, the key nodes are set, N frames of effective image and video of cargo lifting, running process, landing moment and key nodes are retained, and the data types include mutation, key and normal; The preprocessing of numerical data includes: comparing the numerical value with the preset screening threshold, eliminating the numerical data that does not meet the screening threshold, then extracting the numerical data corresponding to cargo lifting, running process, landing moment and key nodes, performing numerical value and N frames of effective image and video association binding operation, summarizing the associated and bound data, and constructing analysis dataset.

4. The port tally management system according to claim 3, wherein: Classifying the data in the analysis dataset includes preprocessing the data in the analysis dataset, removing redundant marks, completing missing lift number and collection timestamp, then supplementing data source marks by using label encoding technology, and then classifying and arranging according to cargo data, posture data, binding data, personnel data and auxiliary data; The feature correlation mapping technology is adopted to classify the cargo data and auxiliary data as a cargo data set, and the attitude data, binding data and personnel data as a safety data set, the hoisting round correlation technology is adopted to bind all data in the cargo data set and the safety data set with the corresponding hoisting round number, and an identification system is constructed.

5. The port tally management system according to claim 1, wherein: The feature analysis and extraction on the cargo data set includes: using a texture matching algorithm, a contour feature extraction algorithm, a contour clustering algorithm and a multi-frame consensus algorithm to extract data in the cargo data set, to obtain cargo packaging types, cargo identification data, cargo size data, cargo coordinates and cargo spacing, and to form a cargo feature vector; separately labeling data of a mutation type; extracting data of a key type to obtain quantity characteristic data; using an edge detection algorithm and an area ratio algorithm to extract the labeled data to obtain packaging damage data, residual damage area ratio and residual damage type data; using a coordinate matching algorithm to extract the labeled data to obtain the cargo location number after the cargo falls to the ground, and the relative position information of the cargo and the cargo location boundary; binding the hoisting round number and the data type, and forming a cargo feature set by summarizing the data obtained by the feature analysis and extraction on the cargo data set.

6. The port tally management system according to claim 5, wherein: The feature analysis and extraction on the safety data set includes: using a 3D point cloud simplification algorithm and a coordinate calculation algorithm to extract data of a key type and data of a mutation type selected according to a preset proportion, to obtain cargo inclination angle, gravity center offset and swing amplitude; using a target counting algorithm combined with a gap measurement algorithm to extract data of a mutation type to obtain binding quantity, maximum gap value of tightness and binding part damage identification; using a coordinate calibration algorithm and a distance calculation algorithm to extract data of a mutation type to obtain safety region boundary coordinates, personnel related coordinates and the shortest distance between the personnel and the hoist running track; binding the hoisting round number and the data type, and further obtaining a safety feature set by summarizing the data obtained by the feature analysis and extraction on the safety data set.

7. The port tally management system according to claim 6, wherein: Processing the data in the cargo feature set based on a preset tally rule and a convolutional neural network includes: performing normalization processing on the feature data in the cargo feature set to obtain a normalized tally core feature matrix, and eliminating data of a normal type in the cargo data set; using a convolutional neural network to compare and process the tally core feature matrix, the preset tally rule and the cargo data set after eliminating the normal data, to obtain a tally result, an abnormal report, a parameter comparison table and a data list, to carry out verification work, and to generate a cargo list after verification and calibration.

8. The port tally management system according to claim 7, wherein: Processing the safety feature set and a historical risk case data set based on a recurrent neural network includes: performing standardization processing on the feature data in the safety feature set to obtain a standardized safety core feature matrix, and eliminating data of a normal type in the safety data set; using a feature label matching technology to associate the cargo feature vector and the quantity characteristic data in the cargo feature set; using a recurrent neural network to compare the safety core feature matrix with the historical risk case data set based on the safety core feature matrix to obtain a safety risk score.

9. The port tally management system according to claim 1, wherein: The risk level is evaluated based on the security risk score, including: mapping the security risk score to a preliminary risk level according to a corresponding mapping rule, verifying the authenticity of abnormal features by checking the pictures and video evidence corresponding to the mutation data, and if the verification is passed, analyzing the number of abnormal features, and if there is a single abnormal feature, determining that the preliminary risk level is the risk level; If there are multiple abnormal features, trigger the risk level promotion mechanism to promote the preliminary risk level, and determine that the promoted preliminary risk level is the risk level; If the verification fails, recalculate the security risk score.

10. The port tally management system according to claim 9, wherein: The early warning and emergency handling strategy is as follows: when it is determined to be no risk, no processing is required; when it is determined to be low risk, the yellow indicator light of the terminal device flashes and is accompanied by a buzzer prompt; when it is determined to be medium risk, the orange indicator light of the terminal device is always on, continuously emits a buzzer prompt, and pops up a reminder window, the window is accompanied by pictures and videos of corresponding mutation data as evidence, and abnormal reasons are displayed; when it is determined to be high risk, the red indicator light of the terminal device flashes, a high-decibel alarm is emitted, a signal of emergency stop or slow running is sent to the spreader, and detailed information of the mutation data, complete picture and video evidence, risk emergency degree and abnormal reason are pushed.

Images