Port loading and unloading planning support method and system based on multi-modal large model
By integrating multi-source data and constructing a knowledge graph through the multimodal large model X-CLIP, the problems of data integration and constraint modeling in port loading and unloading planning are solved, and efficient, safe and dynamic optimization of port loading and unloading is achieved.
Patent Information
- Application Number
- CN202511667247.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-14
- Publication Date
- 2026-03-03
AI Technical Summary
Traditional port loading and unloading planning methods suffer from problems such as difficulty in integrating multi-source heterogeneous data, insufficient ability to model complex constraints, and poor dynamic adaptability, leading to loading and unloading plan failures and resource waste.
The X-CLIP multimodal large model is used for data cleaning, feature extraction and fusion to construct a structured knowledge graph. Combined with a multi-objective optimization algorithm, the optimal loading and unloading time window sequence is generated, and the rules are automatically parsed and dynamically optimized.
It has achieved efficient integration and dynamic planning of multi-source data, improved port loading and unloading efficiency, reduced cargo delays and resource waste, and enhanced operational efficiency and safety.
Smart Images
Figure CN121599345A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of intelligent scheduling technology for port logistics, and in particular relates to a port loading and unloading planning support method and system based on a multimodal large model. Background Technology
[0002] In modern port logistics systems, precise planning of loading and unloading time windows is a crucial link in cargo turnover, directly impacting port operational efficiency and supply chain responsiveness. Traditional planning methods primarily rely on the experience and judgment of dispatchers and rule-based calculation systems based on static databases, which have significant limitations when dealing with highly dynamic logistics environments.
[0003] Currently, the following problems exist in the intelligent management of port logistics:
[0004] Insufficient multi-source data integration capability: Port logistics data covers a variety of types, including satellite remote sensing images, ship AIS dynamic data, meteorological and hydrological data, equipment status data, and freight documents. Traditional systems have difficulty achieving real-time fusion of heterogeneous data, resulting in serious data silos. For example, if a ship's arrival time is delayed due to sudden weather, the static system cannot promptly correlate and analyze the meteorological data with the ship's dynamic data, causing the loading and unloading plan to fail.
[0005] Modeling complex constraints is difficult: Port operations have a large number of special constraints, such as the different requirements of container and bulk cargo for loading and unloading equipment, environmental restrictions for loading and unloading dangerous goods, and time window restrictions for equipment maintenance. Traditional systems use hard-coded methods to store constraint rules, which cannot adapt to the dynamic changes of rules.
[0006] Poor dynamic adaptability: When unexpected situations such as equipment failure, labor shortage, and yard congestion occur, the static solution adjustment of traditional systems takes up to several hours, resulting in problems such as cargo delays and ship delays.
[0007] Therefore, in order to address the above problems, there is an urgent need for an intelligent planning method that can process multi-source data, automatically parse rules, and perform dynamic optimization, so as to improve the efficiency and adaptability of port loading and unloading planning. Summary of the Invention
[0008] This invention aims to solve the problems of difficulty in integrating multi-source heterogeneous data, insufficient ability to model complex constraints, and poor adaptability to dynamic environments in traditional port loading and unloading planning methods. It provides an intelligent planning support method based on a multimodal large model to achieve accurate planning and dynamic optimization of loading and unloading time windows.
[0009] The present invention adopts the following technical solution:
[0010] A port loading and unloading planning support method based on a multimodal large model, the method comprising the following steps:
[0011] Data acquisition and processing: Construct a thematic database according to the port loading and unloading business scenarios, and clean, repair missing data and standardize the raw data;
[0012] Construct a multimodal large model X-CLIP and pre-train it. Use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors.
[0013] Construct a structured knowledge graph and extract key constraints from unstructured logistics documents;
[0014] We construct a physical-temporal layer and an operational rule layer, transform the constraints into mathematical models, and combine them with a multi-objective optimization algorithm to generate the optimal loading and unloading time window sequence.
[0015] Furthermore, the method for obtaining the fused multimodal feature vectors using the pre-trained multimodal large model X-CLIP is as follows:
[0016] Key features of meteorological texts are extracted using the text encoding module of the pre-trained multimodal large model X-CLIP and converted into numerical feature vectors.
[0017] The visual encoding module of the pre-trained multimodal large model X-CLIP is used to perform target detection on port images, identify the number of containers stacked, the coordinates of empty spaces, and the location of equipment, and generate spatial feature vectors.
[0018] The ship's position and speed sequences are encoded using the temporal coding module of the pre-trained multimodal large model X-CLIP, and the ship's dynamic feature vector is output.
[0019] The correlation between different data is calculated by an attention layer to generate a fused multimodal feature vector.
[0020] Furthermore, when obtaining the optimal loading / unloading time window sequence, the multi-objective optimization algorithm includes:
[0021] Minimize cargo turnaround time: The objective function is to minimize the average time for all cargo from arrival to departure from the port.
[0022] Highest equipment utilization: The objective function is to maximize equipment utilization.
[0023] Minimize safety risk: The objective function is to minimize the risk score.
[0024] Furthermore, the method for extracting key constraints from unstructured logistics documents is as follows:
[0025] Obtain the rule terms as training data and validation set data, and manually annotate the key constraint rules;
[0026] The multimodal large model X-CLIP was fine-tuned using the acquired training and validation set data. Semantic features were extracted through a text encoder, and a classification head was added for rule recognition.
[0027] Fine-tuning employs the cross-entropy loss function and the Adam optimizer, adjusting hyperparameters on the validation set to ensure accuracy;
[0028] The X-CLIP multimodal large model with fine-tuning is used to parse the loading and unloading rule text, automatically extract the constraint rules, and formalize them into logical rules.
[0029] Furthermore, when constructing a physical-temporal layer and an operational rule layer to transform constraints into mathematical models, the constraints include equipment capacity constraints, cargo characteristic constraints, environmental safety constraints, and process timing constraints.
[0030] Furthermore, when constructing the thematic database according to port loading and unloading business scenarios, the data in the database includes container loading and unloading data, bulk cargo handling data, dangerous goods transportation data, equipment maintenance data, ship AIS data, meteorological data, and rule text data.
[0031] Furthermore, the methods for cleaning, missing data repair, and standardization of the raw data are as follows:
[0032] Data cleaning is achieved through a rules engine. Value range validation rules are set for numerical data, and regular expressions are used to match a standard terminology library to correct synonym ambiguity for text data.
[0033] Missing data was repaired using a sample-generating machine learning algorithm.
[0034] Data standardization is achieved by building data meta-models and automatic mapping tools.
[0035] A port loading and unloading planning support system based on a multimodal large model, the system comprising:
[0036] The data acquisition and processing module is used to build a thematic database according to the port loading and unloading business scenarios, and to clean, repair missing data and standardize the raw data.
[0037] The multimodal feature vector acquisition module is used to construct a multimodal large model X-CLIP and pre-train it, and use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors.
[0038] The knowledge graph construction and constraint extraction module is used to construct a structured knowledge graph and extract key constraints from unstructured logistics documents.
[0039] The module for obtaining the optimal loading / unloading time window sequence is used to construct the physical spatiotemporal layer and the operation rule layer, transform the constraints into mathematical models, and generate the optimal loading / unloading time window sequence by combining multi-objective optimization algorithms.
[0040] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a port loading and unloading planning support method based on a multimodal large model as described above.
[0041] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a port loading and unloading planning support method based on a multimodal large model as described above.
[0042] The beneficial technical effects of this invention are as follows:
[0043] 1. This invention effectively integrates multimodal heterogeneous data through a multi-source data fusion mechanism, solving the problem of insufficient data integration capabilities of traditional methods;
[0044] 2. This invention utilizes an automatic rule parsing engine to extract computable constraint rules from professional documents, thereby improving the ability to model complex constraints.
[0045] 3. This invention designs a spatiotemporal reasoning optimization algorithm, which transforms the planning problem into a multi-objective optimization model. The optimization objectives include: hard constraints: operational safety (e.g., dangerous goods loading and unloading meet time requirements) and equipment capacity matching (e.g., not exceeding the maximum load of equipment); soft constraints: minimum cargo turnaround time, maximum equipment utilization, and minimum operational risk.
[0046] 4. This invention generates the optimal time window sequence through algorithm iteration and can dynamically adjust the planning scheme according to real-time data changes (such as sudden equipment failures), which improves the accuracy and dynamic adaptability of time window planning, effectively balances loading and unloading efficiency, operational safety and resource constraints, reduces cargo delays and resource waste, and significantly improves port operation efficiency. Attached Figure Description
[0047] Figure 1 This is a technical roadmap for data fusion and knowledge base construction provided by embodiments of the present invention. Detailed Implementation
[0048] The following, in conjunction with the accompanying drawings, provides a clearer and more complete description of the power supply capacity adjustment and optimization method and system for a multi-source long-distance traction power supply system provided by the present invention:
[0049] Example 1
[0050] This invention provides a port loading and unloading planning support method based on a multimodal large model, the method comprising the following steps:
[0051] Data acquisition and processing: Construct a thematic database according to the port loading and unloading business scenarios, and clean, repair missing data and standardize the raw data;
[0052] Construct a multimodal large model X-CLIP and pre-train it. Use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors.
[0053] Construct a structured knowledge graph and extract key constraints from unstructured logistics documents;
[0054] We construct a physical-temporal layer and an operational rule layer, transform the constraints into mathematical models, and combine them with a multi-objective optimization algorithm to generate the optimal loading and unloading time window sequence.
[0055] Specifically:
[0056] Data processing is the foundation of this invention, aiming to ensure the accuracy, integrity, and consistency of data, providing reliable data support for subsequent data fusion and optimization decisions. Specific measures are as follows:
[0057] Thematic database construction includes: container loading and unloading data, including fields such as container type, size, weight, and destination; bulk cargo handling data, including fields such as cargo type, density, and storage requirements; data on dangerous goods transportation, including fields such as dangerous goods class and transportation restrictions; equipment maintenance data, including fields such as equipment model, maintenance cycle, and fault records; port satellite imagery data; ship AIS data, including fields such as ship position, speed, and estimated arrival time; meteorological data, including fields such as wind speed, precipitation, visibility, and tidal height; and rule text data, including documents such as "Port Loading and Unloading Rules," "Ship Berthing Management Regulations," and "Safety Regulations for Dangerous Goods Operations."
[0058] Data cleaning operations: For numerical data, set a reasonable value range, and mark data that exceeds the range as abnormal and perform secondary verification; for text data, use regular expressions to correct synonyms; for image data, perform operations such as dehazing, noise reduction, and distortion correction.
[0059] Data missing data repair: When data is missing, the missing data is generated using a generative adversarial network based on historical load records, cargo types, and route information of similar vessels.
[0060] Data standardization: Construct a model containing core data elements such as "ship ID", "arrival time" and "cargo ID" to unify the timestamp format across different systems.
[0061] When integrating multi-source data: First, data from different modalities are encoded into a unified vector representation. The text encoding module of the pre-trained multimodal large model X-CLIP is used to extract key features from meteorological texts and convert them into numerical feature vectors. A visual encoding module is used to perform target detection on port images, identifying container stacking quantities, empty space coordinates, equipment locations, etc., generating a "spatial feature vector." A temporal encoding module is used to encode ship positions and speed sequences, outputting a "ship dynamic feature vector." Then, the correlation between different data is calculated through an attention layer to generate a fused multimodal feature vector. This process allows the model to automatically learn the implicit temporal, spatial, and semantic relationships between data, providing input for subsequent knowledge graph construction. Here, a cross-modal attention fusion mechanism is used to process structured data, semi-structured data, and unstructured text. Hierarchical feature extraction identifies the relationships between semantic rules, achieving a unified knowledge representation in the joint embedding space. The hierarchical feature extraction includes bottom-level feature extraction (data format conversion, basic semantic parsing), mid-level feature extraction (entity relationship recognition, temporal feature modeling), and high-level feature extraction (rule conflict detection, constraint weight allocation).
[0062] Knowledge Graph Construction: By treating ships, cargo, equipment, and port areas as entity nodes and the relationships between entities as edges, a structured knowledge graph is constructed to make the "entity-relationship" of port loading and unloading elements explicit. This provides structured knowledge support for the next step of spatiotemporal reasoning and time window planning, including the spatial association between equipment and cargo and the matching relationship between equipment capabilities and cargo requirements.
[0063] like Figure 1 As shown, Figure 1 This embodiment presents a technical roadmap for data fusion and knowledge base construction, where: input data includes multimodal data such as historical logistics data, real-time port intelligence, and text reports; the data processing layer performs data cleaning, standardization, and feature extraction; the fusion layer achieves multi-source data fusion through a cross-modal attention mechanism; and the knowledge output layer generates a structured knowledge graph.
[0064] Rule Extraction: The purpose of rule extraction is to automatically obtain constraint rules from text specifications, improving the efficiency of rule extraction for subsequent rule layer construction. First, rule clauses are collected as training and validation data, and key constraint rules (including constraint subjects and conditions) are manually labeled. This data is used to fine-tune the CLIP model, extracting semantic features through a text encoder and adding a classification head for rule recognition. Fine-tuning employs the cross-entropy loss function and the Adam optimizer, adjusting hyperparameters on the validation set to ensure accuracy. The F1 score (precision + recall) is used as the evaluation metric; fine-tuning stops when the validation set F1 score ≥ 0.9.
[0065] The fine-tuned multimodal model is used to parse the loading and unloading rule text, automatically extract the constraint rules, and formalize them into logical rules and store them in the knowledge base.
[0066] Physical spatiotemporal layer construction: Semantic segmentation technology is used on the "spatial feature vector" of satellite imagery to identify geographical elements such as the location of berths, yard zoning, and equipment distribution; then, real-time sensor data is integrated to update dynamic environmental parameters such as wind speed, tidal height, and visibility.
[0067] Operational rule layer construction: Integrating constraints from the physical-temporal layer and the knowledge base to provide a rule-based foundation for optimization problems. This includes: equipment capacity constraints, such as maximum lifting weight, maximum efficiency, and downtime maintenance constraints; cargo characteristic constraints, such as hazardous materials isolation constraints, refrigerated cargo constraints, and special cargo loading and transportation efficiency constraints; environmental safety constraints, such as weather conditions limiting operations and tidal conditions limiting ship berthing; and process timing constraints, such as hazardous materials loading and unloading time restrictions and cargo priority constraints.
[0068] Multi-objective optimization:
[0069] ① Minimize cargo turnaround time: The objective function is to minimize the average time for all goods from arrival to departure from the port, i.e. in, The departure time of cargo i. Let N be the arrival time of cargo i, and N be the total number of cargoes.
[0070] ② Maximize equipment utilization: The objective function is to maximize equipment utilization, i.e. Where M represents the number of devices. For equipment operating time, Total time;
[0071] ③ Minimize safety risk: The objective function is to minimize the risk score, calculated based on the hazard level, environmental factors, etc. Where K is the number of jobs, r k Risk value;
[0072] These objectives are combined into a single objective function using a weighted summation method.
[0073] A genetic algorithm is used to solve the problem. First, the time window sequence is encoded as chromosomes (each gene represents the start time of a task). Then, the population is initialized and evolved through selection, crossover, and mutation operations (generating initial solutions based on a knowledge graph). The fitness function is based on the objective function mentioned above, and constraints are added. The algorithm runs until convergence and outputs the Pareto optimal solution set. Finally, the decision-maker selects the optimal time window sequence.
[0074] It should be noted that in this embodiment, the physical spatiotemporal layer covers the port's geographical layout and dynamic environmental parameters; the operation rule layer includes equipment capacity constraints, cargo characteristic constraints, and time constraints; the automatic conversion from natural language constraints to formal descriptions is achieved through a multi-granularity spatiotemporal constraint expression language; the multi-objective optimization model aims to maximize logistics benefits while satisfying loading and unloading efficiency, operational safety, and resource constraints.
[0075] Example 2
[0076] The present invention also adopts the following technical solutions:
[0077] A port loading and unloading planning support system based on a multimodal large model, the system comprising:
[0078] The data acquisition and processing module is used to build a thematic database according to the port loading and unloading business scenarios, and to clean, repair missing data and standardize the raw data.
[0079] The multimodal feature vector acquisition module is used to construct a multimodal large model X-CLIP and pre-train it, and use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors.
[0080] The knowledge graph construction and constraint extraction module is used to construct a structured knowledge graph and extract key constraints from unstructured logistics documents.
[0081] The module for obtaining the optimal loading / unloading time window sequence is used to construct the physical spatiotemporal layer and the operation rule layer, transform the constraints into mathematical models, and generate the optimal loading / unloading time window sequence by combining multi-objective optimization algorithms.
[0082] A non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a port loading and unloading planning support method based on a multimodal large model as described above.
[0083] Furthermore, the present invention adopts the following technical solution:
[0084] An electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements a port loading and unloading planning support method based on a multimodal large model as described above.
[0085] From the above description of the embodiments, those skilled in the art will clearly understand that the facilities of the present invention can be implemented using software plus necessary general-purpose hardware platforms. Embodiments of the present invention can be implemented using existing processors, or by dedicated processors used for this or other purposes for suitable systems, or by hardwired systems. Embodiments of the present invention also include non-transitory computer-readable storage media, comprising machine-readable media for carrying or having machine-executable instructions or data structures stored thereon; such machine-readable media can be any available medium accessible by a general-purpose or special-purpose computer or other machine with a processor. For example, such machine-readable media can include RAM, ROM, EPROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to carry or store the required program code in the form of machine-executable instructions or data structures and is accessible by a general-purpose or special-purpose computer or other machine with a processor. When information is transmitted or provided to a machine via a network or other communication connection (hardwired, or wireless, or a combination of hardwired and wireless), that connection is also considered a machine-readable medium.
[0086] The technical solution of the present invention has been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of the present invention is obviously not limited to these specific embodiments. Without departing from the principles of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after such changes or substitutions will all fall within the scope of protection of the present invention.
Claims
1. A port loading and unloading planning support method based on a multimodal large model, characterized in that, The method includes the following steps: Data acquisition and processing: Construct a thematic database according to the port loading and unloading business scenarios, and clean, repair missing data and standardize the raw data; Construct a multimodal large model X-CLIP and pre-train it. Use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors. Construct a structured knowledge graph and extract key constraints from unstructured logistics documents; We construct a physical-temporal layer and an operational rule layer, transform the constraints into mathematical models, and combine them with a multi-objective optimization algorithm to generate the optimal loading and unloading time window sequence.
2. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, The method for obtaining the fused multimodal feature vectors using the pre-trained multimodal large model X-CLIP is as follows: Key features of meteorological texts are extracted using the text encoding module of the pre-trained multimodal large model X-CLIP and converted into numerical feature vectors. The visual encoding module of the pre-trained multimodal large model X-CLIP is used to perform target detection on port images, identify the number of containers stacked, the coordinates of empty spaces, and the location of equipment, and generate spatial feature vectors. The ship's position and speed sequences are encoded using the temporal coding module of the pre-trained multimodal large model X-CLIP, and the ship's dynamic feature vector is output. The correlation between different data is calculated by an attention layer to generate a fused multimodal feature vector.
3. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, When obtaining the optimal loading / unloading time window sequence, multi-objective optimization algorithms include: Minimize cargo turnaround time: The objective function is to minimize the average time for all cargo from arrival to departure from the port. Highest equipment utilization: The objective function is to maximize equipment utilization. Minimize safety risk: The objective function is to minimize the risk score.
4. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, The method for extracting key constraints from unstructured logistics documents is as follows: Obtain the rule terms as training data and validation set data, and manually annotate the key constraint rules; The multimodal large model X-CLIP was fine-tuned using the acquired training and validation set data. Semantic features were extracted through a text encoder, and a classification head was added for rule recognition. Fine-tuning employs the cross-entropy loss function and the Adam optimizer, adjusting hyperparameters on the validation set to ensure accuracy; The X-CLIP multimodal large model with fine-tuning is used to parse the loading and unloading rule text, automatically extract the constraint rules, and formalize them into logical rules.
5. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, When constructing the physical-temporal layer and the operational rule layer, and transforming the constraints into mathematical models, the constraints include equipment capacity constraints, cargo characteristic constraints, environmental safety constraints, and process timing constraints.
6. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, When constructing thematic databases according to port loading and unloading business scenarios, the data in the databases include container loading and unloading data, bulk cargo handling data, dangerous goods transportation data, equipment maintenance data, ship AIS data, meteorological data, and rule text data.
7. The port loading and unloading planning support method based on a multimodal large model according to claim 1, characterized in that, The methods for cleaning, missing data repair, and standardization of raw data are as follows: Data cleaning is achieved through a rules engine. Value range validation rules are set for numerical data, and regular expressions are used to match a standard terminology library to correct synonym ambiguity for text data. Missing data was repaired using a sample-generating machine learning algorithm. Data standardization is achieved by building data meta-models and automatic mapping tools.
8. A port loading and unloading planning support system based on a multimodal large model, characterized in that, The system includes: The data acquisition and processing module is used to build a thematic database according to the port loading and unloading business scenarios, and to clean, repair missing data and standardize the raw data. The multimodal feature vector acquisition module is used to construct a multimodal large model X-CLIP and pre-train it, and use the pre-trained multimodal large model X-CLIP to obtain the fused multimodal feature vectors. The knowledge graph construction and constraint extraction module is used to construct a structured knowledge graph and extract key constraints from unstructured logistics documents. The module for obtaining the optimal loading / unloading time window sequence is used to construct the physical spatiotemporal layer and the operation rule layer, transform the constraints into mathematical models, and generate the optimal loading / unloading time window sequence by combining multi-objective optimization algorithms.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements a port loading and unloading planning support method based on a multimodal large model as described in any one of claims 1 to 7.
10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements a port loading and unloading planning support method based on a multimodal large model as described in any one of claims 1 to 7.