Multi-modal port logistics adaptive scheduling system and method

Abstract

The invention discloses a multi-modal port logistics adaptive scheduling system and method. The system comprises a data acquisition module, a semantic alignment module, a situation analysis module, a decision generation module, a multi-objective optimization module, an instruction generation module and an execution monitoring module. Multi-modal data collected by the data collection module generates a unified representation through the semantic alignment module, the situation analysis module calculates KPI and detects abnormity based on the unified representation, the decision generation module generates a scheduling decision according to situation information, the multi-target optimization module optimizes the decision to obtain the scheduling decision, the instruction generation module converts an optimization scheme into an executable instruction, and the instruction generation module sends the executable instruction to the data collection module. And feedback of the monitoring module is executed and returned to the data acquisition layer to update the data source, and meanwhile, a historical corpus is updated for continuous learning of the decision model. According to the method, self-adaptive scheduling of port operation is realized by constructing a port space-time semantic model, fusing multi-source heterogeneous data, performing real-time situation analysis and performing intelligent decision generation.

Description

Technical Field

[0001] This invention relates to the field of port logistics scheduling and artificial intelligence applications, specifically to a multimodal port logistics adaptive scheduling system and method, which aims to solve problems such as complex port collection and distribution scheduling, serious information silos, and difficulties in resource coordination, and to achieve intelligent, efficient and low-carbon scheduling of port operations. Background Technology

[0002] With the continuous expansion of global trade, ports are playing an increasingly important role as international logistics hubs. Modern ports involve multiple operational stages, including quay crane loading and unloading, yard management, truck dispatching, and intermodal transport. These stages generate diverse data types, including video surveillance, AIS vessel tracking, IoT sensor data, voice intercom, and text dispatch instructions. However, existing port dispatching systems suffer from the following technical problems:

[0003] Severe information silos exist: data formats are inconsistent between different systems (TOS terminal operating system, GOS gate system, equipment monitoring system), making cross-system data fusion and collaborative decision-making difficult. Video, sensor, and text data from various operational stages are stored in a scattered manner, lacking a unified spatiotemporal alignment mechanism.

[0004] Delayed scheduling decisions: Traditional scheduling methods rely mainly on manual experience and static rules, which cannot respond in real time to dynamic changes such as yard congestion, equipment failure, and sudden weather changes. The scheduling instruction generation cycle is long, making it difficult to adapt to the high-frequency demand for ship arrivals and cargo transfers.

[0005] The resource coupling relationships are complex: there are strong coupling relationships among resources such as quay cranes, yard cranes, container trucks, and container bays, and changes in the scheduling of a single resource can trigger a chain reaction. The existing system lacks the ability to globally model the resource coupling relationships, resulting in the problem of local optimization but global suboptimal.

[0006] Insufficient anomaly identification capability: Port operations may encounter various anomalies such as empty container stacking, misplacement of hazardous materials, and channel congestion. Existing systems mainly rely on manual inspections, resulting in low identification efficiency, slow response speed, and potential safety hazards.

[0007] Lack of energy consumption and carbon emission management: Under the "dual carbon" target, port scheduling needs to balance operational efficiency and energy consumption optimization. Existing scheduling systems mostly focus on minimizing operation time as the sole objective, failing to incorporate indicators such as energy consumption and carbon emissions into the scheduling decision-making system.

[0008] In existing technologies, some ports use rule-based expert systems or single optimization algorithms (such as genetic algorithms and particle swarm optimization) for scheduling, but these methods have the following limitations:

[0009] 1. The rule system struggles to cover complex and ever-changing port scenarios, exhibiting poor generalization capabilities;

[0010] 2. Single optimization algorithms struggle to handle multimodal data fusion problems;

[0011] 3. Lack of mechanisms for learning from and reusing historical scheduling experience;

[0012] 4. Unable to generate natural language and interact with voice commands for scheduling instructions.

[0013] Therefore, there is an urgent need for an intelligent scheduling system that can integrate multimodal data, perceive the port situation in real time, adaptively generate scheduling decisions, and support human-machine collaboration, so as to improve port operation efficiency, reduce energy consumption, and ensure safety. Summary of the Invention

[0014] To address the aforementioned issues, this invention provides a multimodal port logistics adaptive scheduling system and method. By constructing a port spatiotemporal semantic model, integrating multi-source heterogeneous data, and performing real-time situational analysis and intelligent decision generation, adaptive scheduling of port operations is achieved. The system accesses multimodal data sources, including quay crane operation videos, AIS trajectories, UAV yard top-down views, equipment IoT sensors, intercom voice, shipping company manifests, railway / highway scheduling text, and meteorological / tidal data. Through a multimodal alignment mechanism driven by port business ontology, heterogeneous data is mapped to a unified spatiotemporal semantic space. Based on this, an intelligent decision-making model for instruction-based scheduling fine-tuning is used to identify anomalies such as congestion, empty container stacking, and hazardous material misplacement in real time. It automatically generates decision suggestions for quay crane scheduling, stacking position adjustments, and barge connection, and outputs voice / text dual-modal scheduling commands, forming a closed-loop scheduling system of "data acquisition - situational analysis - decision generation - execution feedback."

[0015] The specific plan is as follows:

[0016] A multimodal port logistics adaptive scheduling system includes a data acquisition module, a semantic alignment module, a situation analysis module, a decision generation module, a multi-objective optimization module, an instruction generation module, and an execution monitoring module; the data acquisition module collects multimodal data. A unified representation is generated by the semantic alignment module. The situation analysis module is based on The KPIs are calculated and anomalies are detected. The decision generation module generates scheduling decisions based on the situation information. The multi-objective optimization module optimizes the decision to obtain... The instruction generation module converts the optimization scheme into executable instructions, and the feedback from the execution monitoring module is fed back to the data acquisition layer to update the data source. At the same time, the historical corpus is updated for the decision model to learn continuously. The modules form a complete closed loop through data flow and feedback mechanisms.

[0017] A multimodal port logistics adaptive scheduling method includes the following steps:

[0018] S1. Multi-source data acquisition and spatiotemporal alignment;

[0019] Access to the port's multimodal data sources is achieved through the data acquisition layer, including:

[0020] Video data stream: High-definition video surveillance of key areas such as quay crane operation area, storage yard, and gate, with a frame rate of 25fps and a resolution of 1920×1080, is transmitted to the data processing center in real time;

[0021] AIS vessel trajectory data: Receives information such as vessel position, speed, and heading broadcast by the Automatic Identification System (AIS), updated every 2-10 seconds, covering the port area and surrounding waters;

[0022] Drone inspection data: Drones take off regularly or on demand to collect overhead images of the yard, with a resolution of 4K, for use in yard container location identification and anomaly detection;

[0023] IoT sensor data: Operating status sensors for equipment such as quay cranes, yard cranes, and container trucks collect parameters such as current, temperature, vibration, and position at a sampling frequency of 1Hz;

[0024] Voice communication data: The voice communication between the dispatcher and the on-site operators, with a sampling rate of 16kHz, is transcribed into text in real time;

[0025] Business system data: vessel plans, manifest information, and stacking space allocation records from the TOS system; truck entry and exit records from the GOS system; and intermodal transport plan texts from the railway / highway dispatching system.

[0026] Meteorological and tidal data: Wind speed, wind direction, rainfall, and visibility data provided by weather stations; Tide level and tidal time data provided by tidal forecasting systems;

[0027] Since the aforementioned data sources differ in their time base, spatial coordinate system, and data format, the system first performs spatiotemporal alignment processing to define a unified time axis. Based on UTC time, all data is timestamped. Perform indexing; for spatial data, establish a unified coordinate system for the port area. Convert AIS latitude and longitude, video pixel coordinates, and device GPS coordinates to a unified format. Coordinate system;

[0028] The spatiotemporally aligned multimodal data is represented as follows:

[0029]

[0030] Among them, For a moment A collection of video frames; For a moment The set of AIS trajectory points; For a moment The set of sensor readings; For a moment The text data set (including speech-to-text, dispatch instructions, and manifest information); For a moment Meteorological and tidal data.

[0031] S2, Port Business Ontology-Driven Multimodal Semantic Alignment

[0032] To achieve semantic fusion of collected multimodal data, a port business ontology database is constructed. It includes loading and unloading processes, equipment types, cargo attributes, operational status, and their relationships; the ontology is represented in triplet form:

[0033]

[0034] in, It is a collection of entities, including equipment entities (quay crane QB01, yard crane YC05, etc.), cargo entities (container TCLU1234567, etc.), and area entities (yard A area, berth #3, etc.). This is a set of relations, including "serves", "is located in", and "depends on" relations; Entities in the ontology library; For entities and The relationship between them;

[0035] Based on the ontology library, spatiotemporally aligned multimodal data Perform scene-level annotation; for video frames The system uses object detection models to identify equipment, containers, and vehicles in the image and associates them with corresponding entities in the ontology database; for text data... The device number, box number, and location entity are extracted using the Named Entity Recognition (NER) model and mapped to the ontology library.

[0036] Multimodal semantic alignment is achieved through cross-modal embedding; a visual encoder is defined. Text encoder Sensor encoder AIS encoder Weather encoder This maps data from different modalities to a unified semantic embedding space. :

[0037]

[0038] in These are semantic embedding vectors for vision, text, sensors, AIS, and meteorology, respectively. For the embedded dimension;

[0039] Through a contrastive learning mechanism, the distance between different modal embeddings of the same entity at the same time in the semantic space is minimized, while the embedding distance between different entities is maximized; the aligned multimodal semantic representation is calculated through a multi-head attention fusion mechanism.

[0040]

[0041] in, For query matrix; The key matrix; It is a value matrix; It is a learnable linear transformation matrix; For the embedded dimension;

[0042] The fused semantic representation Includes time The semantic information of the overall situation of the port serves as a unified input for subsequent situation analysis and decision generation.

[0043] S3. Real-time Port Status Analysis and Anomaly Detection

[0044] Based on generated multimodal semantic representation A port situation analysis module is constructed to calculate key performance indicators (KPIs) in real time. These KPIs extract the operational status of various areas and equipment in the port from semantic representations, and quantitatively assess the port's operational status.

[0045] Stockyard saturation: Defines the stockyard area The saturation is:

[0046]

[0047] in, For a moment storage yard area The number of container slots already occupied; For storage yard area Total container capacity

[0048] Crane utilization rate: defining the quay crane In the time window The utilization rate of the interior is:

[0049]

[0050] in, for shore bridge The actual working time within the time window; The total duration of the time window ;

[0051] Channel Congestion Index: Defining a Channel The congestion index is:

[0052]

[0053] in, For channel Current average vehicle speed; For channel Free-flow speed (speed when there is no congestion). For channel Current number of vehicles in the queue; For channel Design capacity (number of vehicles that can be accommodated at the same time); The queuing effect affects the weighting coefficient;

[0054] The anomaly detection module is based on the aforementioned KPIs and multimodal semantic representations. Identify the following abnormal situations:

[0055] abnormal congestion in the storage yard: when Triggered at time, where This is the saturation threshold (e.g., 0.85).

[0056] Equipment malfunction: When sensor data Triggered when device parameters exceed the normal range;

[0057] Dangerous goods misplacement anomaly: By comparing video recognition with the manifest, it is detected whether dangerous goods containers are placed in the designated area;

[0058] Abnormal congestion in the passageway: When Triggered at time, where The congestion threshold (e.g., 1.5);

[0059] Anomaly detection results output risk level ; Transfer KPI vector Risk level It is also transmitted to the scheduling decision module as a key input for decision generation.

[0060] S4. Command-based scheduling decision generation and constraint verification

[0061] The scheduling decision generation module receives semantic representations. Based on the situational analysis results and historical dispatch corpus, a dispatch decision scheme is generated; a large language model with imperative fine-tuning is used as the core of the decision-making process, and the model input is:

[0062]

[0063] in, For multimodal semantic representation; These are vectors representing the saturation, utilization rate, and congestion index of each region / equipment. Risk level; It is a corpus of historical scheduling instructions and execution results (continuously updated by execution feedback);

[0064] Model outputs scheduling decision ,include:

[0065] Shore crane scheduling plan: Assign quay cranes and operation time slots to each vessel waiting for operation;

[0066] Stacking space adjustment plan: Allocate stacking spaces for newly arriving containers, or optimize and adjust existing stacking spaces;

[0067] Truck dispatching scheme: Assigning transportation tasks and routes to trucks;

[0068] Barge transfer plan: Arrange the transfer time and berths between the barge and the mother ship;

[0069] To ensure the feasibility and security of the scheduling scheme, the system processes the generated decisions. Perform constraint verification:

[0070] Safety distance constraints: Minimum safety distances must be met between equipment and between equipment and containers. ;

[0071] Equipment capacity constraints: quay crane Maximum lifting capacity Maximum stacking height of the yard bridge ;

[0072] Tide window constraints: Ships entering and leaving the port must meet the tide level requirements, i.e. ,in For a moment The tide level, For the ship's draft, For safety margin;

[0073] Carbon emission constraints: The total carbon emissions of the scheduling scheme must meet the following requirements. ,in To predict carbon emissions, For carbon emission quotas;

[0074] Constraint verification is performed collaboratively by the rule engine and the optimization solver. If a decision violates constraints, the parameters are automatically adjusted or the decision is regenerated until all constraints are met. The verified scheduling scheme is then considered valid. Entering the multi-objective optimization stage, further balancing operational efficiency and energy consumption.

[0075] S5, Congestion-Energy Consumption Co-optimization

[0076] Traditional port scheduling primarily aims to minimize operation time. This invention, based on constraint verification, introduces energy consumption and carbon emission optimization to construct a multi-objective optimization model; and defines scheduling schemes. The overall objective function is:

[0077]

[0078] in, The total operation time of the scheduling plan; The total energy consumption of the scheduling scheme; Costs related to ship delays; For the weighting coefficients, satisfying ;

[0079] Total energy consumption The energy consumption of equipment including quay cranes, yard cranes, and container trucks is calculated using the following formula:

[0080]

[0081] in, This refers to a collection of quay cranes, yard cranes, and container trucks. They are quay bridges , field bridge Average power; They are quay bridges , field bridge In the plan The homework time in the middle; respectively container trucks Power under heavy load and no load; respectively container trucks Driving distance under heavy load and unloaded conditions - respectively container trucks Average speed under heavy load and no load;

[0082] By adjusting the weighting coefficients It can achieve a flexible balance between operational efficiency and energy consumption; during off-peak hours or when carbon emission quotas are tight, it can increase... To reduce energy consumption; to increase capacity when ships arrive in port in large numbers. To improve operational efficiency; optimized scheduling scheme The instructions are passed to the instruction generation module and converted into executable scheduling instructions.

[0083] S6, Dual-modal scheduling instruction generation and human-machine collaboration

[0084] Optimized scheduling scheme It needs to be converted into executable scheduling instructions; the instruction generation module supports dual-modal output of text and speech:

[0085] Text instructions: Structured scheduling instructions are pushed to the TOS / GOS system and equipment control terminal, in a format such as "The quay crane QB01 will start working on vessel COSCO001 at 14:30, with an estimated operation time of 120 minutes".

[0086] Voice commands: Using text-to-speech (TTS) technology, dispatch commands are converted into voice broadcasts and sent to on-site personnel via the intercom system;

[0087] To support human-machine collaboration, voice interaction functionality is provided; dispatchers can query the port status (including KPIs and risk levels), modify dispatching plans, and confirm instruction execution in real time via voice intercom; voice input is transcribed into text by automatic speech recognition (ASR) and then input into the decision-making model for understanding and response;

[0088] The system records the dispatcher's verbal verification and confirmation information as feedback data for model fine-tuning. When the dispatcher modifies the instructions generated by the model, the system automatically extracts the differences before and after the modification, constructs new instruction samples, and stores them in the historical corpus. This is used for continuous learning and optimization of the model. The generated instructions are pushed to the execution layer for execution monitoring and feedback.

[0089] S7. Perform closed-loop and rolling optimization.

[0090] After the scheduling command is pushed to the execution layer, the execution status of the command is tracked in real time through the TOS / GOS interface; the execution feedback data includes: the actual start / end time of the equipment operation, the actual stacking position of the container, the actual driving route and time of the truck, and abnormal event records (equipment failure, weather interruption, etc.).

[0091] The feedback data is fed back to the data acquisition layer as new sensor data. Data from business systems Part of the port status is updated after semantic alignment. This forms a complete closed loop; the system evaluates the actual effect of the scheduling scheme and calculates the prediction deviation based on execution feedback.

[0092]

[0093] in, This is the vector of actual operating time of the equipment; This is the vector of job times predicted by the decision model; It is the vector norm;

[0094] When prediction deviation Exceeding the threshold When this happens, rolling optimization is triggered, and the process of situation analysis, decision generation, multi-objective optimization, and command output is re-executed to generate a new scheduling scheme. Rolling optimization adopts a sliding time window mechanism, which adjusts the scheduling scheme based on the latest port situation at fixed time intervals (e.g., 30 minutes) or when an abnormal event occurs. Re-plan the scheduling scheme for future time periods to achieve dynamic adaptive scheduling;

[0095] The execution feedback data also serves as a sample source for model fine-tuning; the system periodically (e.g., daily) extracts scheduling instructions, execution results, and exception event information from the execution log, constructs instruction-response pairs, and updates the historical corpus. This is used for incremental learning of the decision-making model, continuously improving the model's scheduling and decision-making capabilities. This completes the closed loop of "data acquisition → semantic alignment → situational analysis → decision generation → multi-objective optimization → instruction output → execution feedback".

[0096] The beneficial effects of this invention are as follows:

[0097] 1. This invention uses a multimodal alignment mechanism driven by port business ontology to uniformly map heterogeneous data such as video, AIS, sensors, text, and voice to the semantic space, breaking down the information silos of traditional port systems and realizing deep data fusion across systems and modalities, providing a complete information foundation for global scheduling decisions.

[0098] 2. This invention, based on real-time situational analysis and intelligent decision-making models, can generate scheduling plans within a minute-level timescale, improving decision response speed by more than 10 times compared to traditional manual scheduling or static rule systems. Through instruction-based fine-tuning, the model understands port terminology, significantly improving the accuracy and executability of the generated scheduling plans.

[0099] 3. This invention utilizes multimodal semantic analysis to automatically identify various abnormal situations such as yard congestion, equipment malfunction, hazardous material misplacement, and passageway blockage, achieving an accuracy rate of over 95%. Compared to traditional manual inspections, the anomaly identification efficiency is increased by 5 times, and the response time is shortened to minutes, effectively reducing safety risks.

[0100] 4. This invention incorporates energy consumption and carbon emission indicators into the scheduling objective function. Through adjustable weighting coefficients, it reduces port energy consumption by 10%-15% and carbon emissions by 10%-15% while ensuring operational efficiency, thus helping ports achieve their "dual carbon" goals. During off-peak hours, energy consumption can be reduced by more than 20% through optimized scheduling schemes.

[0101] 5. This invention, through the implementation of a closed-loop and rolling optimization mechanism, can dynamically adjust the scheduling scheme based on real-time execution feedback, adapting to emergencies such as equipment failure, sudden weather changes, and ship delays. Compared to static scheduling schemes, dynamic adaptive scheduling increases the port operation plan completion rate by 15% and equipment utilization by 10%.

[0102] 6. This invention utilizes a continuous learning mechanism to automatically extract knowledge from historical scheduling experience, reducing reliance on the experience of human dispatchers. New dispatchers can be deployed after brief training, achieving scheduling quality comparable to senior dispatchers, and reducing labor costs by more than 30%.

[0103] 7. This invention supports voice interaction and bimodal command output. Dispatchers can query port status and modify dispatching plans via natural language, eliminating the need for cumbersome system operations. Voice command broadcasts enable on-site personnel to quickly receive dispatching information, improving operational response speed by 20%.

[0104] 8. The technical solution of the present invention does not depend on a specific port layout or equipment type. By configuring the ontology library and adjusting the model parameters, it can be quickly adapted to ports of different sizes and types (container terminals, bulk cargo terminals, rail-water intermodal ports, etc.), and has good scalability and versatility. Attached Figure Description

[0105] Figure 1 This is a flowchart of the method in this invention. Detailed Implementation

[0106] The present invention will be further illustrated below with reference to the accompanying drawings and specific embodiments. It should be understood that the following specific embodiments are for illustrative purposes only and are not intended to limit the scope of the invention.

[0107] As shown in the figure, this embodiment provides a multimodal port logistics adaptive scheduling system and method. Taking a large container terminal as an example, the terminal has 6 berths, 12 quay cranes, 30 yard cranes, and 80 container trucks. The storage yard is divided into three areas: A, B, and C, with a total container capacity of 15,000 TEU. The specific solution is as follows:

[0108] S1. Multi-source data acquisition and spatiotemporal alignment

[0109] The system deploys data acquisition nodes and connects to the following data sources:

[0110] Video data: A total of 50 high-definition cameras are deployed in 6 berths, 3 yard areas and 2 gates. The video stream is transmitted to the data center through a fiber optic network, with a total bandwidth requirement of approximately 500Mbps.

[0111] AIS data: Deployed AIS base stations receive AIS signals from ships within a 20-nautical-mile radius of the port area, receiving an average of about 100 AIS messages per second.

[0112] Drone data: Two drones are deployed to conduct three scheduled inspections of the yard daily, collecting approximately 5,000 4K images per inspection.

[0113] IoT sensors: Sensors are deployed on 12 quay cranes, 30 yard cranes, and 80 container trucks to collect parameters such as current, temperature, and GPS location, generating approximately 1,000 sensor records per second.

[0114] Voice intercom: Voice communication between the dispatch center and on-site personnel is collected through a digital intercom system, averaging about 500 voice messages per day.

[0115] Business systems: Connect to the TOS system to obtain vessel plans, manifests, and stacking space allocation data; connect to the GOS system to obtain truck entry and exit records; connect to the railway dispatching system to obtain rail-water intermodal transport plans.

[0116] Meteorological tides: Access meteorological stations to obtain real-time wind speed, wind direction, and rainfall data; access the tide forecast system to obtain tide level forecasts for the next 72 hours.

[0117] After data collection, the system performs spatiotemporal alignment to unify the timeline. Using UTC+8 time zone, all data is timestamped in seconds. Index. Spatial coordinate system A Cartesian coordinate system is established with the center of the wharf as the origin. AIS latitude and longitude are transformed through Mercator projection, video pixel coordinates are transformed through camera calibration matrix, and equipment GPS coordinates are directly transformed to the wharf coordinate system.

[0118] The spatiotemporally aligned multimodal data organization is as follows The sensor data is stored in a data lake. A time-series database (InfluxDB) is used to store the sensor data. MinIO object storage stores video. With images, relational databases (PostgreSQL) store business data. Graph database (Neo4j) stores ontology. These aligned data serve as input to the semantic alignment module.

[0119] S2, Port Business Ontology-Driven Multimodal Semantic Alignment

[0120] Receive spatiotemporally aligned multimodal data Build a port business ontology library It includes the following entity types (entity collections) ):

[0121] Equipment consists of 12 quay cranes (QB01-QB12), 30 yard cranes (YC01-YC30), and 80 container trucks (TK001-TK080).

[0122] Area entities: 6 berths (Berth1-Berth6), 3 yard areas (YardA, YardB, YardC), and 2 gates (Gate1, Gate2).

[0123] Cargo entities: Containers (identified by container number), Vessels (identified by vessel name / IMO number)

[0124] Operational entities: loading and unloading operations, container handling in the storage yard, and truck transportation.

[0125] Relations (sets of relations) defined in the ontology library )include:

[0126] "Serving": The quay crane QB01 serves berth 1.

[0127] "Located at": Container TCLU1234567 is located in Yard A-03-05-02 (Area A, Row 3, Column 5, Level 2).

[0128] "Depends on": Loading operations depend on the coordination of quay cranes, yard cranes, and container trucks.

[0129] For video data The YOLOv8 object detection model is used to identify quay cranes, yard cranes, trucks, and containers in the image, achieving an accuracy rate of over 95%. Detected entities are associated with an ontology database through features such as equipment number and container number.

[0130] For text data The BERT-NER model was used to extract entities such as equipment number, box number, and location, achieving an F1 score of over 92%. The extracted entities were also mapped to the ontology library.

[0131] Multimodal semantic alignment employs a CLIP architecture cross-modal encoder. (Visual encoder) Text encoder based on ResNet-50 Based on BERT-Base, sensor encoder AIS encoder using Transformer architecture and weather encoder An MLP structure is used. The embedding dimension of the encoder output. .

[0132] The modal data is encoded as follows:

[0133]

[0134] The encoder is trained by contrastive learning, which embeds different modalities of the same entity into the semantic space. Cosine similarity is maximized. The training dataset contains 100,000 labeled samples, each containing a video frame, text description, and sensor readings for the same entity. After training, the cross-modal embedding cosine similarity for the same entity reaches above 0.85.

[0135] Multimodal fusion employs a multi-head attention mechanism:

[0136]

[0137] Wherein: each modality is embedded and concatenated into a matrix. Query matrix Key matrix Value matrix Linear transformation matrix This was achieved through comparative learning training; the number of attention heads was set to 8, and each head had a dimension of 64-. For the embedding dimension. The fused semantic representation. It contains semantic information about the overall situation of the port at the current moment and is transmitted to the situation analysis module.

[0138] S3. Real-time Port Status Analysis and Anomaly Detection

[0139] Based on the multimodal semantic representation generated above The system calculates key performance indicators every minute:

[0140] Yard saturation: Total container space in Yard Area A TEU, currently occupied TEU, calculate saturation:

[0141]

[0142] Saturation of 0.84 is close to the threshold. The system issued a warning.

[0143] Quay crane utilization rate: Quay crane QB01 in the past 1 hour time window ( Actual working time within minutes Minutes, calculate utilization rate:

[0144]

[0145] The utilization rate of 0.80 is within the normal range.

[0146] Channel congestion index: Main channel from yard A to berth 1, free-flow speed km / h, current average vehicle speed km / h, number of vehicles in queue Vehicles, channel design capacity Vehicles, queuing affects weight Calculate the congestion index:

[0147]

[0148] The congestion index of 2.33 exceeds the threshold. The system triggered an alarm, suggesting that the truck route be adjusted or the channel capacity be increased.

[0149] The anomaly detection module is based on the above KPIs and semantic representations. The following anomalies were detected:

[0150] 1. Abnormal congestion in the storage yard: Saturation of storage yard area A. This triggered an alert, and it is recommended that some containers be moved to Area B of the yard.

[0151] 2. Abnormal congestion in the main corridor: Congestion index of the main corridor This triggered an alarm, and it is recommended that some trucks switch to the alternative route.

[0152] 3. Hazardous materials misplacement anomaly: Video recognition detected a hazardous materials container (UN1203, gasoline) placed in a non-hazardous materials area, triggering an emergency alarm and requiring immediate relocation.

[0153] System comprehensive risk level assessment The situation information (semantic representation) KPI vector Risk level The message is pushed to the scheduling decision module.

[0154] S4. Command-based scheduling decision generation and constraint verification

[0155] The scheduling decision module receives situational information and constructs a model input. :

[0156] Multimodal semantic representation

[0157] Stockyard saturation vector (Corresponding to areas A, B, and C)

[0158] quay crane utilization vector (12 shore cranes)

[0159] Channel congestion index (3 main routes)

[0160] Risk level

[0161] Historical scheduling corpus (Includes scheduling instructions and execution results from the past 3 months, continuously updated by execution feedback)

[0162] The decision model is based on the GPT architecture, with 7 bytes of parameters, and is fine-tuned using an instruction-based approach on a port scheduling corpus. The fine-tuning dataset contains 50,000 scheduling instruction samples, each in the following format:

[0163] Input: Yard A area saturation 0.84, main aisle congestion index 2.33, dangerous goods container UN1203 misaligned.

[0164] Output: 1. Transfer 100 containers from Yard A to Yard B; 2. Detach trucks TK010-TK020 to the alternate lane; 3. Immediately dispatch yard crane YC05 to transfer dangerous goods container UN1203 to the designated dangerous goods area.

[0165] Model outputs scheduling decision :

[0166] 1. Container relocation: 100 empty containers near the gate in area A of the yard will be moved to area B of the yard, reducing the saturation level in area A to 0.82.

[0167] 2. Truck route adjustment: Trucks TK010-TK020 will be rerouted to use the alternative route, which is expected to reduce the congestion index on the main route to 1.4.

[0168] 3. Dangerous Goods Transfer: Dispatch yard crane YC05 shall immediately transfer the dangerous goods container TCLU1234567 (UN1203) to the dedicated dangerous goods area YardC-DG-01.

[0169] The constraint verification module verifies the above decisions. Verification required:

[0170] Safety distance constraint: When yard crane YC05 is transferring dangerous goods containers, the distance between it and the adjacent yard crane YC06 must be greater than the minimum safety distance. Meters. Querying the current location, the distance between YC05 and YC06 is 15 meters, which satisfies the constraint.

[0171] Equipment capacity constraints: Maximum lifting capacity of YC05 yard crane The dangerous goods container weighs 25 tons, which meets the constraints.

[0172] Tide window constraint: There are currently no vessel entry or exit plans, so there is no need to verify the tide window constraint.

[0173] Carbon emission constraints: Calculate the total energy consumption of the scheduling scheme. This scheme involves yard crane YC05 (hazardous materials transfer), yard cranes YC08-YC10 (stack location adjustment), and container trucks TK010-TK020 (route adjustment). The energy consumption calculation is as follows:

[0174]

[0175] Among them: YC05 power of the yard bridge kW, operating time Hours (dangerous goods transfer); YC08-YC10 yard cranes all have the following power ratings. kW, operating time Hours (stacking location adjustment); Truck heavy-load power kW, no-load power kW; heavy load distance km, unloaded distance km; heavy load speed km / h, unloaded speed km / h; number of trucks: 11 (TK010-TK020).

[0176] The calculation yields:

[0177]

[0178]

[0179] carbon emissions kgCO2 (grid carbon emission factor 0.6 kg / kWh), lower than the current carbon emission quota. kg, satisfying the constraints.

[0180] All constraints passed verification; scheduling decision scheme Once confirmed as feasible, it is passed to the multi-objective optimization module.

[0181] S5, Congestion-Energy Consumption Co-optimization

[0182] The scheduling scheme that passes the constraint verification. Calculate the overall objective function. Currently, it is an off-peak period, so the weighting coefficients are set to... It balances operational efficiency and energy consumption.

[0183] Total operation time: Estimated time for stacking location adjustment is 2 hours; truck route adjustment takes effect immediately; hazardous materials transfer takes 0.5 hours. Hour.

[0184] Total energy consumption: as calculated above, kWh.

[0185] Delay costs: There are currently no ship delays. .

[0186] Calculate the objective function:

[0187]

[0188]

[0189] The system attempted an optimized solution, performing the stacker relocation in batches and completing it during off-peak hours at night. This extended the total operation time to 3 hours, but reduced energy consumption to 310 kWh (utilizing lower electricity prices at night allows for higher equipment operating efficiency and the adoption of more energy-efficient operating modes). The optimized solution... Objective function:

[0190]

[0191] The optimized solution's objective function value The system adopted the optimization solution. It is then passed to the instruction generation module.

[0192] S6, Dual-modal scheduling instruction generation and human-machine collaboration

[0193] Receive the optimized scheduling scheme The system converts these commands into text instructions and pushes them to the TOS system and related device terminals:

[0194] Instruction 1: YC05, immediately transfer container TCLU1234567 (dangerous goods UN1203) from row 05, column 02, row 03 of yard A to the dangerous goods-specific area DG-01 in yard C. The estimated operation time is 30 minutes.

[0195] Instruction 2: Trucks TK010-TK020, effective immediately, shall use the alternative route (route: Yard A area - alternative route - berth 1) to avoid congestion on the main route.

[0196] Instruction 3: Yard cranes YC08-YC10 shall begin at 22:00 today to transfer 100 empty containers near the gate in Yard A to Yard B. The operation is expected to take 3 hours.

[0197] Meanwhile, the system uses TTS technology to convert commands into voice, which is then broadcast to on-site personnel via the intercom system.

[0198] “Attention YC05 operator: Immediately move container TCLU1234567 from row 05, column 03, level 02 of section A in yard to section C, dangerous goods area DG-01. This container contains dangerous goods UN1203. Please strictly follow the safety operating procedures.”

[0199] The dispatcher confirmed the instruction via voice intercom: "Received, YC05 is executing, expected to arrive at the target location in 10 minutes."

[0200] The voice confirmation is transcribed into text via ASR, the system updates the command execution status to "Executing," and starts execution monitoring. The generated command is pushed to the execution layer, entering the execution loop and feedback phase.

[0201] S7. Perform closed-loop and rolling optimization.

[0202] After the dispatch instructions are pushed to the execution layer, the system tracks the execution of the instructions in real time through the TOS interface. YC05 yard crane completed the transfer of the dangerous goods container after 8 minutes. The actual operation time was shorter than the predicted time (30 minutes). The prediction deviation was calculated as follows:

[0203] Actual working time of YC05 yard bridge Minutes, predicted task time Minutes, prediction bias is:

[0204]

[0205] Prediction deviation exceeded the threshold by 22 minutes. After minutes, the system analyzed the cause: the YC05 yard crane was currently close to the target location and there were no other tasks interfering, resulting in a significantly shorter actual operation time than the average. The system stored this execution record (instruction, predicted time, actual time, reason for deviation) as a new sample in the historical corpus. It is used for fine-tuning decision-making models to improve the accuracy of future predictions.

[0206] After the truck route is adjusted, the execution feedback data is fed back to the data acquisition layer as new sensor data. Update. Based on semantic alignment and situational analysis, the main lane congestion index decreased after 15 minutes. It is still higher than the normal value but lower than the alarm threshold. The system continuously monitors the congestion index. If the index does not drop below 1.5 after 30 minutes, rolling optimization will be triggered, and the process of situation analysis, decision generation, multi-objective optimization, and instruction output will be re-executed to generate a new scheduling scheme.

[0207] The stack adjustment plan is scheduled to be executed at 22:00. The system will conduct a final situation assessment at 21:50 based on the latest semantic representation. Confirm the saturation level of storage area B. There is enough space to receive 100 empty containers, and the YC08-YC10 yard cranes have no other urgent tasks, so the plan is being executed as scheduled.

[0208] During execution, the system collects the execution progress every 5 minutes and updates the port status. By 00:00 the following day, the stockpile relocation was completed, and the saturation level of area A in the stockpile yard had decreased to [a certain value]. The data is below the warning threshold. The system assesses that the scheduling plan was successfully executed and stores the execution log (including instructions, execution time, and actual results) in the historical corpus. This is used for subsequent model fine-tuning. At this point, the complete closed loop of "data acquisition → semantic alignment → situational analysis → decision generation → multi-objective optimization → command output → execution feedback" is achieved.

[0209] In summary, this invention constructs a port business ontology library covering loading and unloading processes, equipment types, and cargo attributes, and performs scene-level annotation and semantic mapping on heterogeneous data such as video, text, and sensor data. Through cross-modal encoders and contrastive learning, multimodal data is aligned to a unified semantic embedding space, achieving deep fusion of heterogeneous data. Based on a large language model architecture, it utilizes port-specific corpora such as real scheduling commands, event logs, and anomaly replays for instructional fine-tuning, enabling the model to understand professional terms such as "lead time," "cross-operation," and "tide window," generating scheduling decisions that conform to port operation specifications. This invention also constructs a multi-objective optimization model incorporating operation time, energy consumption, and delay costs, achieving a flexible balance between operational efficiency and energy consumption through adjustable weight coefficients. Parameters such as equipment power and travel distance are included in energy consumption calculations, supporting carbon emission constraint verification. After scheduling decisions are generated, hard constraints such as safety distance, equipment capacity, tide window, and carbon emissions are automatically verified. Through the collaboration of a rule engine and an optimization solver, schemes that violate constraints are automatically adjusted or regenerated, ensuring the feasibility and safety of scheduling schemes.

[0210] This invention tracks the execution status of scheduling commands in real time through the TOS / GOS interface, feeding back execution feedback data to the situation analysis module, forming a closed loop of "data acquisition - situation analysis - decision generation - execution feedback". A sliding time window mechanism is employed to trigger rolling optimization when abnormal events or prediction deviations exceed thresholds, achieving dynamic adaptive scheduling. This invention supports dual-modal scheduling command output (text and voice), enabling voice interaction between the scheduler and the system through ASR / TTS technology. The scheduler's spoken verification and confirmation information is recorded as feedback data for model fine-tuning, continuously improving the model's scheduling decision-making capabilities.

[0211] The above embodiments are merely typical illustrative methods of the present invention, and the scope of protection of the present invention is not limited thereto. All equivalent substitutions and improvements made under the concept of the present invention should fall within the scope of protection. It should be emphasized that any modifications or minor adjustments made by those skilled in the art without departing from the basic principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A multimodal port logistics adaptive scheduling system, characterized in that, It includes a data acquisition module, a semantic alignment module, a situation analysis module, a decision generation module, a multi-objective optimization module, an instruction generation module, and an execution monitoring module; the data acquisition module collects multimodal data. A unified representation is generated by the semantic alignment module. The situation analysis module is based on The KPIs are calculated and anomalies are detected. The decision generation module generates scheduling decisions based on the situation information. The multi-objective optimization module optimizes the decision to obtain... The instruction generation module converts the optimization scheme into executable instructions, and the feedback from the execution monitoring module is fed back to the data acquisition layer to update the data source, while also updating the historical corpus for the decision model to learn continuously. The modules form a complete closed loop through data flow and feedback mechanisms.

2. A multimodal port logistics adaptive scheduling method, based on the system described in claim 1, characterized in that, Includes the following steps: S1. Multi-source data acquisition and spatiotemporal alignment; S2, Port Business Ontology-Driven Multimodal Semantic Alignment; S3. Real-time analysis and anomaly detection of port status; S4. Command-based scheduling decision generation and constraint verification; S5, Congestion-Energy Consumption Co-optimization; S6, Dual-modal scheduling instruction generation and human-machine collaboration; S7. Perform closed-loop and rolling optimization.

3. The multimodal port logistics adaptive scheduling method according to claim 2, characterized in that, In step S1, the port's multimodal data sources are accessed through the data acquisition layer, including video data streams, AIS ship trajectory data, UAV inspection data, IoT sensor data, voice intercom data, business system data, and meteorological and tidal data; firstly, spatiotemporal alignment processing is performed to define a unified timeline. Based on UTC time, all data is timestamped. Perform indexing; for spatial data, establish a unified coordinate system for the port area. Convert AIS latitude and longitude, video pixel coordinates, and device GPS coordinates to a unified format. Coordinate system; The spatiotemporally aligned multimodal data is represented as follows: in, For a moment A collection of video frames; For a moment The set of AIS trajectory points; For a moment The set of sensor readings; For a moment A collection of text data; For a moment Meteorological and tidal data.

4. The multimodal port logistics adaptive scheduling method according to claim 3, characterized in that, In step S2, a port business ontology library is constructed to achieve semantic fusion of the collected multimodal data. It includes loading and unloading processes, equipment types, cargo attributes, operational status, and their relationships; the ontology is represented in triplet form: in, It is a collection of entities, including equipment entities, goods entities, and area entities; This is a set of relations, including "serves", "is located in", and "depends on" relations; Entities in the ontology library; For entities and The relationship between them; Based on the ontology library, spatiotemporally aligned multimodal data Perform scene-level annotation; for video frames The system uses object detection models to identify equipment, containers, and vehicles in the image and associates them with corresponding entities in the ontology database; for text data... The named entity recognition model is used to extract the device number, box number, and location entities, which are then mapped to the ontology library. Multimodal semantic alignment is achieved through cross-modal embedding; a visual encoder is defined. Text encoder Sensor encoder AIS encoder Weather encoder This maps data from different modalities to a unified semantic embedding space. : in These are semantic embedding vectors for vision, text, sensors, AIS, and meteorology, respectively. For the embedded dimension; Through a contrastive learning mechanism, the distance between different modal embeddings of the same entity at the same time in the semantic space is minimized, while the embedding distance between different entities is maximized; the aligned multimodal semantic representation is calculated through a multi-head attention fusion mechanism. in, For query matrix; The key matrix; It is a value matrix; It is a learnable linear transformation matrix; For the embedded dimension; The fused semantic representation Includes time The semantic information of the overall situation of the port serves as a unified input for subsequent situation analysis and decision generation.

5. The multimodal port logistics adaptive scheduling method according to claim 4, characterized in that, In step S3, based on the generated multimodal semantic representation A port situation analysis module is constructed to calculate key performance indicators (KPIs) in real time. These KPIs extract the operational status of various areas and equipment in the port from semantic representations, and quantitatively assess the port's operational status. Stockyard saturation: Defines the stockyard area The saturation is: in, For a moment storage yard area The number of container slots already occupied; For storage yard area Total container capacity Crane utilization rate: defining the quay crane In the time window The utilization rate of the interior is: in, for shore bridge The actual working time within the time window; The total duration of the time window ; Channel Congestion Index: Defining a Channel The congestion index is: in, For channel Current average vehicle speed; For channel Free-flow vehicle speed; For channel Current number of vehicles in the queue; For channel Design capacity; The queuing effect affects the weighting coefficient; The anomaly detection module is based on the aforementioned KPIs and multimodal semantic representations. Identify the following abnormal situations: abnormal congestion in the storage yard: when Triggered at time, where This is the saturation threshold. Equipment malfunction: When sensor data Triggered when device parameters exceed the normal range; Dangerous goods misplacement anomaly: By comparing video recognition with the manifest, it is detected whether dangerous goods containers are placed in the designated area; Abnormal congestion in the passageway: When Triggered at time, where The congestion threshold; Anomaly detection results output risk level ; Transfer KPI vector Risk level It is also transmitted to the scheduling decision module as a key input for decision generation.

6. The multimodal port logistics adaptive scheduling method according to claim 5, characterized in that, In step S4, the scheduling decision generation module receives semantic representation. Based on the situational analysis results and historical dispatch corpus, a dispatch decision scheme is generated; a large language model with imperative fine-tuning is used as the core of the decision-making process, and the model input is: in, For multimodal semantic representation; These are vectors representing the saturation, utilization rate, and congestion index of each region / equipment. Risk level; It is a corpus of historical scheduling instructions and execution results (continuously updated by execution feedback); Model outputs scheduling decision ,include: Shore crane scheduling plan: Assign quay cranes and operation time slots to each vessel waiting for operation; Stacking space adjustment plan: Allocate stacking spaces for newly arriving containers, or optimize and adjust existing stacking spaces; Truck dispatching scheme: Assigning transportation tasks and routes to trucks; Barge transfer plan: Arrange the transfer time and berths between the barge and the mother ship; To ensure the feasibility and security of the scheduling scheme, the system processes the generated decisions. Perform constraint verification: Safety distance constraints: Minimum safety distances must be met between equipment and between equipment and containers. ; Equipment capacity constraints: quay crane Maximum lifting capacity Maximum stacking height of the yard bridge ; Tide window constraints: Ships entering and leaving the port must meet the tide level requirements, i.e. ,in For a moment The tide level, For the ship's draft, For safety margin; Carbon emission constraints: The total carbon emissions of the scheduling scheme must meet the following requirements. ,in To predict carbon emissions, For carbon emission quotas; Constraint verification is performed collaboratively by the rule engine and the optimization solver. If a decision violates constraints, the parameters are automatically adjusted or the decision is regenerated until all constraints are met. A successful scheduling plan is then selected. Entering the multi-objective optimization stage, further balancing operational efficiency and energy consumption.

7. The multimodal port logistics adaptive scheduling method according to claim 6, characterized in that, In step S5, based on constraint verification, energy consumption and carbon emission optimization are introduced to construct a multi-objective optimization model; and a scheduling scheme is defined. The overall objective function is: in, The total operation time of the scheduling plan; The total energy consumption of the scheduling scheme; Costs related to ship delays; Let be the weighting coefficient, satisfying ; Total energy consumption The energy consumption of equipment including quay cranes, yard cranes, and container trucks is calculated using the following formula: in, This refers to a collection of quay cranes, yard cranes, and container trucks. They are quay bridges , field bridge Average power; They are quay bridges , field bridge In the plan The homework time in the middle; respectively container trucks Power under heavy load and no load; respectively container trucks Driving distance under heavy load and unloaded conditions - respectively container trucks Average speed under heavy load and no load; By adjusting the weighting coefficients It can achieve a flexible balance between operational efficiency and energy consumption; during off-peak hours or when carbon emission quotas are tight, it can increase... To reduce energy consumption; to increase capacity when ships arrive in port in large numbers. To improve operational efficiency; optimized scheduling scheme The instructions are passed to the instruction generation module and converted into executable scheduling instructions.

8. The multimodal port logistics adaptive scheduling method according to claim 7, characterized in that, In step S6, the optimized scheduling scheme It needs to be converted into executable scheduling instructions; the instruction generation module supports dual-modal output of text and speech: Text commands: Structured scheduling commands are pushed to the TOS / GOS system and device control terminals; Voice commands: Using text-to-speech technology, dispatch commands are converted into voice broadcasts and sent to on-site personnel via an intercom system; To support human-machine collaboration, voice interaction functionality is provided; dispatchers can query port status, modify dispatch plans, and confirm instruction execution in real time via voice intercom; voice input is automatically transcribed into text by speech recognition and then input into the decision model for understanding and response. The system records the dispatcher's verbal verification and confirmation information as feedback data for model fine-tuning; When the scheduler modifies the instructions generated by the model, the differences between the original and modified instructions are automatically extracted, new instruction samples are constructed, and stored in the historical corpus. It is used for continuous learning and optimization of the model; the generated instructions are pushed to the execution layer and enter the execution monitoring and feedback stage.

9. The multimodal port logistics adaptive scheduling method according to claim 8, characterized in that, In step S7, after the scheduling instruction is pushed to the execution layer, the execution status of the instruction is tracked in real time through the TOS / GOS interface; the execution feedback data includes: the actual start / end time of the equipment operation, the actual stacking position of the container, the actual driving path and time of the truck, and abnormal event records. The feedback data is fed back to the data acquisition layer as new sensor data. Data from business systems Part of the port status is updated after semantic alignment. This forms a complete closed loop; the system evaluates the actual effect of the scheduling scheme and calculates the prediction deviation based on execution feedback. in, This is the vector of actual operating time of the equipment; This is the vector of job times predicted by the decision model; It is the vector norm; When prediction deviation Exceeding the threshold When this occurs, rolling optimization is triggered, and the process of situation analysis, decision generation, multi-objective optimization, and command output is re-executed to generate a new scheduling scheme. Rolling optimization adopts a sliding time window mechanism, which updates the scheme based on the latest port situation at fixed time intervals or when an abnormal event occurs. Re-plan the scheduling scheme for future time periods to achieve dynamic adaptive scheduling; The execution feedback data also serves as a sample source for model fine-tuning; the system periodically extracts scheduling instructions, execution results, and abnormal event information from the execution log, constructs instruction-response pairs, and updates the historical corpus. It is used for incremental learning of decision-making models to continuously improve the model's scheduling and decision-making capabilities.

Images