Container detection scheduling method, device and equipment and storage medium

Abstract

The embodiment of the invention provides a container detection scheduling method and device, equipment and a storage medium. The method comprises the steps of obtaining multi-modal data of a container; performing semantic alignment according to the multi-modal data and the multi-modal large model, and extracting a multi-modal alignment text; generating a task plan according to the multi-modal alignment text and the port operation scheduling model; and port scheduling is carried out and task planning is executed. According to the method, the multi-modal data is aligned into the language modal, the understanding and processing capability of the multi-modal large model is improved, so that the container is identified more accurately, and the job scheduling accuracy is further improved; the influence of data noise or an abnormal value of a single mode is effectively reduced by utilizing the complementary characteristic of each mode, so that the stability and the anti-interference performance of the multi-mode large model are improved; the two key links of container detection and task scheduling are tightly combined, seamless connection of the whole process from detection to scheduling is achieved, and the response time is greatly shortened.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a container detection and scheduling method, device, equipment and storage medium. Background Art

[0002] As containers become the core tool in international trade transportation, transportation companies and ports have an increasing demand for container damage detection. Container damage is directly related to cargo safety and transportation efficiency. In serious cases, it may cause cargo loss or endanger driving safety. Therefore, it is crucial to develop intelligent container damage detection technology.

[0003] Traditional container detection methods rely only on a single mode for detection, which has defects such as insufficient information, weak comprehension, poor robustness and low stability. In addition, the traditional container detection method and the container scheduling method are two independent steps, resulting in the port operation scheduling being unable to respond quickly after the container is detected, thereby reducing the operating efficiency of the entire system. Summary of the invention

[0004] The embodiments of the present invention provide a container detection scheduling method, device, equipment and storage medium, the purpose of which is to provide a container detection method based on multimodal data to effectively improve the accuracy of container detection.

[0005] In a first aspect, an embodiment of the present invention provides a container detection and scheduling method, the method comprising:

[0006] Acquire multimodal data of the container, the multimodal data comprising at least two of text data, image data, infrared data, point cloud data, and radiographic image data of the container;

[0007] According to the multimodal data and the multimodal large model, semantic alignment is performed to extract multimodal aligned text, wherein the multimodal large model is obtained by semantic alignment training using contrastive learning and parameter fine-tuning operations;

[0008] Generate a task plan based on the multimodal alignment text and the port operation scheduling model;

[0009] According to the mission plan, port scheduling is performed and the mission plan is executed.

[0010] Furthermore, the multimodal large model includes a language encoder, a multimodal encoder and a multimodal joint training module. The multimodal large model is obtained by performing semantic alignment training using contrastive learning and parameter fine-tuning operations, and the steps include:

[0011] Inputting a text sample into the language encoder to obtain text features;

[0012] Inputting image samples, infrared samples, point cloud samples and ray image samples into the multimodal encoder, and performing parameter fine-tuning operations to obtain image features, infrared features, point cloud features and ray features;

[0013] Based on the text features, semantic alignment training is performed on the image features, the infrared features, the point cloud features and the ray features to obtain a trained multimodal large model.

[0014] Furthermore, the contrastive learning loss formula of the multimodal large model is as follows:

[0015]

[0016] Among them, L contrast represents the contrastive learning loss, L m2t and L t2m are similarity losses, K is the number of multimodal and text pairs in each batch during the training of the multimodal large model, i and j are both positive integers, and m i is the feature vector of the i-th multimodal data, t j is the feature vector of the j-th text data, T represents the amount of the multimodal data, N represents the amount of the text data, τ is the temperature parameter, and sim represents the cosine similarity.

[0017] Furthermore, the port operation scheduling model includes a domain identification module, a job classifier and a semantic slot filling classifier. The steps of generating a task plan according to the multimodal alignment text and the port operation scheduling model include:

[0018] Inputting the multimodal aligned text into the domain identification module to obtain candidate domains;

[0019] Inputting the candidate fields into the job classifier and the semantic slot filling classifier respectively to obtain the job classification probability of the semantic representation and the prediction probability of the slot filling text;

[0020] The task plan is generated according to the job classification probability and the predicted probability of the slot filling text.

[0021] Furthermore, the calculation formula of the job classification probability is as follows:

[0022] P(y i |T)=softmax(W i ·H cls +b i );

[0023] i∈1…M;

[0024] Among them, P(y i|T) is the job classification probability P of the semantic representation T i , W i represents the classification weight of the i-th port operation, b i represents the classification bias of the i-th port operation, M is the total number of port operation classifications, cls represents a special word, and H cls is the feature vector corresponding to the special word, i and M are positive integers;

[0025] The calculation formula for the predicted probability of the slot filling text is as follows:

[0026]

[0027] n∈1…N;

[0028] in, The predicted probability of filling the slot of the semantic representation T with text W s is the classification weight filled in the sth slot, b s is the graph bias of the sth slot filling, H n is the nth element of the feature vector H, n, N, s are positive integers, and N is the total number of word unit sequences corresponding to the special word unit.

[0029] Furthermore, the method further comprises:

[0030] The damage information of the container is extracted according to the multimodal data and the multimodal large model.

[0031] Furthermore, the steps of performing port dispatch and executing the task plan according to the task plan include:

[0032] Prioritize each task in the task plan and store the priority in the task queue;

[0033] The tasks in the task queue are scheduled to be executed in sequence.

[0034] In a second aspect, an embodiment of the present invention provides a container detection and scheduling device, the device comprising:

[0035] A collection module, used to obtain multimodal data of the container, wherein the multimodal data includes at least two of text data, image data, infrared data, point cloud data and radiographic image data of the container;

[0036] A detection module, used to perform semantic alignment according to the multimodal data and the multimodal large model, and extract multimodal aligned text, wherein the multimodal large model is obtained by semantic alignment training using contrastive learning and parameter fine-tuning operations;

[0037] A generation module, used for generating a task plan according to the multimodal alignment text and the port operation scheduling model;

[0038] The scheduling module is used to perform port scheduling and execute the task plan according to the task plan.

[0039] In a third aspect, an embodiment of the present invention provides a computer device, the device comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0040] Memory, used to store computer programs;

[0041] The processor is used to implement the steps of a container detection and scheduling method provided in the first aspect when executing the program stored in the memory.

[0042] In a fourth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the container detection and scheduling method according to any one of claims 1 to 7.

[0043] The embodiment of the present invention provides a container detection and scheduling method, device, equipment and storage medium. When it is necessary to schedule the container, multimodal data of the container including text, image, infrared, point cloud and radiographic image is collected, and the multimodal data is aligned into language mode. By integrating the multimodal data and relying on the complementary characteristics between the modes, the understanding and processing capabilities of the multimodal large model are improved, so that the container can be identified more accurately and the accuracy of job scheduling can be further improved. Moreover, the multimodal large model can utilize the complementary characteristics between the modes to effectively reduce the influence of data noise or outliers of a single mode, so as to improve the stability and anti-interference of the multimodal large model. Moreover, through semantic alignment processing, the problem can be quickly located and task planning can be formulated, so as to realize intelligent operation process. By closely combining the two key links of container detection and task scheduling, the whole process from detection to scheduling is seamlessly connected, which greatly shortens the response time and improves the efficiency and accuracy of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying any creative work.

[0045] Figure 1 A flowchart of a container detection and scheduling method is provided for an embodiment of the present invention;

[0046] Figure 2 A port task planning collaboration flow chart provided by an embodiment of the present invention;

[0047] Figure 3 A network architecture diagram for semantic alignment of a multimodal large model provided by an embodiment of the present invention;

[0048] Figure 4 A schematic diagram of port task planning and scheduling provided by an embodiment of the present invention;

[0049] Figure 5 A schematic structural diagram of a container detection and scheduling device provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0051] In order to make the description of the disclosed content more detailed and complete, the following is an illustrative description of the implementation mode and specific examples of the present invention; however, this is not the only form of implementing or applying the specific embodiments of the present invention. The implementation mode covers the features of multiple specific embodiments and the method steps and their sequence for constructing and operating these specific embodiments. However, other specific embodiments can also be used to achieve the same or equal functions and step sequences. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0052] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein.

[0053] In the description of the embodiments of the present invention, unless otherwise specified, “ / ” means or, for example, A / B can mean A or B; “and / or” in the text is merely a description of the association relationship of associated objects, indicating that three relationships may exist, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in the description of the embodiments of the present application, “multiple” refers to two or more than two, and other quantifiers are similar. It should be understood that the preferred embodiments described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention, and the embodiments of the present application and the features in the embodiments may be combined with each other without conflict.

[0054] In the existing technology, port artificial intelligence detection systems often use single-mode data for container damage detection, focusing on specific tasks such as container number recognition, damage detection, and seal label recognition. However, it should be noted that existing port artificial intelligence detection systems often separate detection from operation scheduling, which leads to three major problems:

[0055] (1) The limitation of a single modality is the lack of understanding: Traditional port artificial intelligence detection systems rely only on a single modality such as vision, which undoubtedly restricts its understanding ability, resulting in low detection accuracy.

[0056] (2) The separation of detection and scheduling is serious: The traditional port artificial intelligence detection system regards container detection and task scheduling as two independent steps. As a result, after the target is detected, the port operation scheduling system fails to respond quickly, thereby reducing the operating efficiency of the entire system.

[0057] (3) Poor stability and robustness: Affected by environmental variability and target complexity, traditional port artificial intelligence detection systems often face challenges in stability and robustness. Environmental changes, noise interference, target occlusion, etc. may have a negative impact on the detection results.

[0058] In view of the above problems, an embodiment of the present invention provides a container detection and scheduling method. Figure 1 A flowchart of a container detection and scheduling method is provided for an embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0059] S110, acquiring multimodal data of a container, wherein the multimodal data includes at least two of text data, image data, infrared data, point cloud data, and radiographic image data of the container;

[0060] Before implementing the container detection and scheduling method, it is necessary to collect multimodal data of the container, wherein the multimodal data includes at least two of the container text data, container image data, container infrared data, container point cloud data and container radiographic image data. The specific data can be determined according to actual conditions, and the embodiment of the present invention does not make any specific limitation on this.

[0061] For ease of explanation, in the embodiments of the present invention, multimodal data including text data, image data, infrared data, point cloud data and radiographic image data are used as an example for explanation.

[0062] Among them, when collecting multimodal data of containers, it can be achieved with the help of port perception and interaction technology, such as setting up sensors, interactive terminals, ordinary cameras, infrared cameras, lidar and X-ray scanners, etc., and reading the text information of the export freight documents of container goods through the interactive terminal as text data; through the ordinary cameras, infrared cameras, lidar and X-ray scanners set up on the port network, image data, infrared data, point cloud data and radiographic image data, etc.

[0063] The radiographic image data of the container is an X-ray image; the image data may be a video or a photo taken by a common camera, which may be determined according to actual conditions, and the embodiment of the present invention does not specifically limit this.

[0064] The embodiment of the present invention collects multimodal data of containers including text, images, infrared, point clouds and radiographic images, aligns the multimodal data into language modalities, and integrates the multimodal data. With the help of the complementary characteristics between different modalities, the understanding and processing capabilities of the multimodal large model are improved, thereby more accurately identifying containers and further improving the accuracy of job scheduling. Moreover, the multimodal large model can utilize the complementary characteristics between different modalities to effectively reduce the impact of data noise or outliers of a single modality, thereby improving the stability and anti-interference of the multimodal large model.

[0065] S120, performing semantic alignment according to the multimodal data and the multimodal large model, and extracting multimodal aligned text, wherein the multimodal large model is obtained by performing semantic alignment training using contrastive learning and parameter fine-tuning operations;

[0066] Then, the multimodal data is input into the multimodal big model for semantic alignment and extraction of multimodal aligned text. The multimodal big model refers to a model that combines multimodal information such as text, images, videos, and audio for training.

[0067] It is easy to understand that the multimodal large model needs to be trained with sample data before use. In the embodiment of the present invention, contrastive learning and parameter fine-tuning operations are used to perform semantic alignment training on the multimodal large model.

[0068] Semantic alignment refers to establishing a semantic correspondence between different texts, language expressions or knowledge structures so that they can match each other in meaning. In the semantic alignment task, multiple texts that need to be aligned can be input into a multimodal large model to obtain their semantic representations, and then a certain matching algorithm is used to determine the degree of semantic alignment.

[0069] The parameter fine-tuning operation can be carried out using the σ-LoRA (Sigma-Low-Rank Adaptation) technology. σ-LoRA is a technology in the field of efficient fine-tuning of deep learning model parameters. It is an extension and improvement of the LoRA (Low-Rank Adaptation) technology. LoRA mainly represents the update of model weights through low-rank decomposition, thereby reducing the number of trainable parameters when fine-tuning the model.

[0070] The embodiment of the present invention can quickly locate problems and formulate task plans through semantic alignment processing, thereby realizing intelligent operation processes.

[0071] S130, generating a task plan according to the multimodal alignment text and the port operation scheduling model;

[0072] The multimodal aligned text is input into the port operation scheduling model to generate a task plan. In the embodiment of the present invention, the task plan is in JSON format. According to the type of task plan, the execution order of the task queue and the start time of the task plan are determined.

[0073] In the embodiment of the present invention, the port operation scheduling model quickly locates problems and formulates operation scheduling plans based on multimodal alignment text, generates task planning, and realizes intelligent operation processes.

[0074] S140, performing port scheduling and executing the task plan according to the task plan.

[0075] Figure 2 A port task planning collaboration flow chart provided by an embodiment of the present invention is as follows: Figure 2 As shown, after the task plan in JSON format is generated, the task plan is added to the job task queue, the priority sorting agent prioritizes all task plans in the job task queue, and the task execution agent executes each task plan in the task queue in turn.

[0076] The embodiment of the present invention provides a container detection and scheduling method. When the container needs to be scheduled, multimodal data of the container including text, image, infrared, point cloud and radiographic image is collected, and the multimodal data is aligned into a language mode. By integrating the multimodal data and relying on the complementary characteristics between the modes, the understanding and processing capabilities of the multimodal large model are improved, so that the container can be identified more accurately and the accuracy of job scheduling can be further improved. In addition, the multimodal large model can utilize the complementary characteristics between the modes to effectively reduce the influence of data noise or outliers of a single mode, so as to improve the stability and anti-interference of the multimodal large model. Through semantic alignment processing, the problem can be quickly located and task planning can be formulated, so as to realize intelligent operation process. By closely combining the two key links of container detection and task scheduling, the whole process from detection to scheduling is seamlessly connected, the response time is greatly shortened, and the efficiency and accuracy of the overall system are improved.

[0077] As an implementation mode, the multimodal large model includes a language encoder, a multimodal encoder and a multimodal joint training module. The multimodal large model is obtained by performing semantic alignment training using contrastive learning and parameter fine-tuning operations, and the steps include:

[0078] Inputting a text sample into the language encoder to obtain text features;

[0079] Inputting image samples, infrared samples, point cloud samples and ray image samples into the multimodal encoder, and performing parameter fine-tuning operations to obtain image features, infrared features, point cloud features and ray features;

[0080] Based on the text features, semantic alignment training is performed on the image features, the infrared features, the point cloud features and the ray features to obtain a trained multimodal large model.

[0081] Figure 3 A network architecture diagram for semantic alignment of a multimodal large model provided by an embodiment of the present invention, such as Figure 3 As shown in the figure, the multimodal large model proposed in the present invention can align multimodal data, such as optical video / image samples, 3D point cloud samples, X-ray image samples and infrared samples, to a unified language feature space, thereby achieving efficient semantic correspondence between each modality data. This semantic alignment method greatly improves the accuracy and efficiency of multimodal data processing. At the same time, it is also easy to add new modalities without rebuilding the entire alignment mechanism, and has stronger scalability.

[0082] The multimodal large model consists of three main components: language encoder, multimodal encoder, and multimodal joint training. The details are as follows:

[0083] (1) Language encoder: It uses a 48-layer Transformer architecture to generate a 1600-dimensional feature vector.

[0084] (2) Multimodal encoder: For optical image / video samples, the SwinTransformer architecture is used to generate a 1024-dimensional feature vector. For X-ray image and infrared imaging samples, the channel dimension is repeated three times and the processing method is the same as that of optical images. For 3D point cloud samples, the RangViT architecture is used to generate a 1536-dimensional feature vector.

[0085] (3) Multimodal joint training uses a contrastive learning mechanism to map multimodal data into a shared semantic feature space to obtain a trained multimodal large model.

[0086] The data during training are called text samples, image samples, point cloud samples, infrared samples and radiographic image samples. The text samples are sequentially input into the language encoder, Chinese word segmenter and text encoder to obtain text features. The image samples, point cloud samples, infrared samples and radiographic image samples are sequentially input into the multimodal encoder (including image encoder, point cloud encoder, infrared encoder and radiographic encoder) to perform mask vector and σ-LoRA parameter tuning to obtain image features, point cloud features, infrared features and radiographic image features.

[0087] Then, based on the text features, semantic alignment training is performed on the image features, infrared features, point cloud features and ray features to obtain a trained multimodal large model.

[0088] The contrastive learning loss of the multimodal large model is shown in formula (1), (2), (3):

[0089]

[0090] Among them, L contrast represents the contrastive learning loss, L m2t and L t2m are similarity losses, K is the number of multimodal and text pairs in each batch during the training of the multimodal large model, i and j are both positive integers, and m i is the feature vector of the i-th multimodal data, t j is the feature vector of the j-th text data, T represents the amount of the multimodal data, N represents the amount of the text data, τ is the temperature parameter, and sim represents the cosine similarity.

[0091] The training method of the multimodal encoder adopts the σ-LoRA parameter fine-tuning operation, and a good training effect can be obtained through a small number of iterations. The present invention adopts contrastive learning to align optical image data, point cloud data, radiographic data, infrared data and text data.

[0092] Among them, σ-LoRA is a major breakthrough in efficiently training large language models for specific tasks and is widely used in many applications. The embodiment of the present invention uses σ-LoRA for efficient parameter fine-tuning, and the process is as follows:

[0093] (1) Add a bypass next to the original pre-trained model to perform a dimensionality reduction and then dimensionality increase operation to simulate the low-dimensional intrinsic rank r.

[0094] (2) Initialize A with a random Gaussian distribution and initialize B with a zero matrix to ensure that the bypass matrix is ​​still a zero matrix at the beginning of training.

[0095] (3) Use the gradient of AB to update the pre-trained weight matrix W. The gradient of AB is the difference between A*B in two consecutive time steps. The gradient is scaled by the hyperparameter α, and the learning rate γ controls the influence of multimodal joint training on the pre-trained weights.

[0096] The gradient update calculation formula is as follows:

[0097]

[0098] ΔAB=A t+1 B t+1 -A t B t ; (5)

[0099] As an implementation mode, the port operation scheduling model includes a domain identification module, a job classifier and a semantic slot filling classifier. The steps of generating a task plan according to the multimodal alignment text and the port operation scheduling model include:

[0100] Inputting the multimodal aligned text into the domain identification module to obtain candidate domains;

[0101] Inputting the candidate fields into the job classifier and the semantic slot filling classifier respectively to obtain the job classification probability of the semantic representation and the prediction probability of the slot filling text;

[0102] The task plan is generated according to the job classification probability and the predicted probability of the slot filling text.

[0103] Port operation dispatching plays an important role in port operation, and is responsible for the organization and coordination of ship operations, handling of abnormal situations, and effective utilization of resources and coordination of various elements. Dispatching work covers a wide range, including collecting relevant information such as shipping and automobile freight, to ensure smooth loading and unloading of materials.

[0104] Figure 4 A schematic diagram of port task planning and scheduling provided by an embodiment of the present invention is as follows: Figure 4 As shown, the embodiment of the present invention adopts a small sample learning strategy to reduce the dependence on labeled data. The port operation scheduling model mainly consists of three parts: domain identification, job classification and semantic slot filling. It is a joint prediction module of jobs and slots based on the ChatGLM3 model. By performing semantic understanding, job classification and semantic slot filling on multimodal aligned text, it can better meet the needs of port operations, coordinate and solve problems in the operation process, and ensure the smooth progress of port operations.

[0105] The port operation scheduling model is based on the ChatGLM3 pre-trained language model with two classifiers. cls , to predict the job classification probability P of the semantic representation T i , and its calculation formula is:

[0106] The calculation formula of the job classification probability is shown in (6):

[0107] P(y i |T)=softmax(W i ·H cls +b i ); (6)

[0108] i∈1…M;

[0109] Among them, P(y i |T) is the job classification probability P of the semantic representation T i , W i represents the classification weight of the i-th port operation, b i represents the classification bias of the i-th port operation, M is the total number of port operation classifications, cls represents a special word, and H cls is the feature vector corresponding to the special word, and i and M are positive integers.

[0110] The calculation formula of the predicted probability of the slot filling text is shown in (7):

[0111]

[0112] n∈1…N;

[0113] in, The predicted probability of filling the slot of the semantic representation T with text W s is the classification weight filled in the sth slot, b s is the graph bias of the sth slot filling, H n is the nth element of the feature vector H, n, N, s are positive integers, and N is the total number of word unit sequences corresponding to the special word unit.

[0114] At this time, the objective function of port operation classification and slot filling is shown as (8):

[0115]

[0116] i∈1…M, n∈1…N.

[0117] Among them, the ChatGLM3 model is a new generation of dialogue pre-training model. It adopts the Transformer architecture. With its self-attention mechanism and position encoding ability, it can effectively process long sequence information and is particularly good at understanding and generating complex contextual dialogues, so that it can better understand and generate natural and fluent texts; and it can understand and generate high-quality answers in a short time, with high dialogue coherence. In addition, due to the use of a large amount of multi-source text data in the pre-training stage, it has rich cross-domain knowledge and can cope with discussions on various topics.

[0118] As an embodiment, the method further includes:

[0119] The damage information of the container is extracted according to the multimodal data and the multimodal large model.

[0120] Then the multimodal data is input into the multimodal large model to perform container inspection and obtain the damage information of the container.

[0121] In the embodiment of the present invention, the damage information of the container includes: container information in the video / image including the container number, container type, and net weight; damage states such as mechanical loss, rust, and corrosion; and the license plate of the container truck / trailer in the recognition image and the seal number information in the photo. The specific information can be determined according to actual conditions, and the embodiment of the present invention does not make specific limitations on this.

[0122] The embodiment of the present invention achieves seamless connection of the entire process from detection to scheduling by closely combining the two key links of container detection and task scheduling, greatly shortening the response time and improving the efficiency and accuracy of the overall system.

[0123] As an implementation mode, the steps of performing port scheduling and executing the task plan according to the task plan include:

[0124] Prioritize each task in the task plan and store the priority in the task queue;

[0125] The tasks in the task queue are scheduled to be executed in sequence.

[0126] See also Figure 2As shown, after the task plan in JSON format is generated, the task plan is added to the job task queue, the priority sorting agent prioritizes all task plans in the job task queue, and the task execution agent executes each task plan in the task queue in turn.

[0127] After executing the task planning in the embodiment of the present invention, the execution result will be fed back to the task creation agent, and the task creation agent will decide whether to create a new task plan based on the result feedback. This is repeated iteratively executed. If the task creation agent decides to generate a new task, the new task is added to the task planning queue and enters the next cycle.

[0128] The embodiment of the present invention provides an accurate, efficient and robust solution, which is based on the design concept of integrated multimodal information integration, port monitoring and operation scheduling. It not only greatly improves the accuracy, stability and anti-interference of port monitoring, but also significantly improves the efficiency and accuracy of port operation scheduling.

[0129] The main advantages are as follows:

[0130] (1) Deepen the understanding of multimodal information and enhance cognitive ability: Compared with traditional single-modal models, multimodal large models can process diversified data such as text, images, videos, infrared, 3D point clouds, etc. By leveraging the complementary characteristics of each modality, the model's understanding and processing capabilities are improved, thereby more accurately identifying and understanding object features and environmental backgrounds, further improving the accuracy of target detection, text recognition, and job scheduling;

[0131] (2) Enhanced stability and anti-interference ability: The multimodal large model can utilize the complementary characteristics between the modes to effectively reduce the impact of data noise or outliers of a certain mode on the model, thereby improving the stability and anti-interference ability of the model;

[0132] (3) Achieve seamless connection from detection to dispatch: By closely combining the two key links of detection and dispatch, the whole process from detection to dispatch is seamlessly connected. When a target is detected, it can respond immediately, greatly shortening the response time and improving the efficiency and accuracy of the overall system.

[0133] Figure 5 A schematic diagram of the structure of a container detection and dispatching device provided by an embodiment of the present invention is shown in FIG. Figure 5 As shown, the container detection and scheduling device includes:

[0134] An acquisition module 510, configured to acquire multimodal data of a container, wherein the multimodal data includes at least two of text data, image data, infrared data, point cloud data, and radiographic image data of the container;

[0135] A detection module 520 is used to perform semantic alignment according to the multimodal data and the multimodal large model, and extract multimodal aligned text, wherein the multimodal large model is obtained by semantic alignment training using contrastive learning and parameter fine-tuning operations;

[0136] A generation module 530, for generating a task plan according to the multimodal alignment text and the port operation scheduling model;

[0137] The scheduling module 540 is used to perform port scheduling and execute the mission plan according to the mission plan.

[0138] This embodiment is a device embodiment corresponding to the above method embodiment, and its specific implementation process is the same as that of the above method embodiment. For details, please refer to the above method embodiment, and this device embodiment will not elaborate on it.

[0139] Based on the above container detection and scheduling method, an embodiment of the present invention further provides a computer device, the computer device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus;

[0140] Memory, used to store computer programs;

[0141] The processor is used to implement the steps of the above container detection and scheduling method when executing the program stored in the memory.

[0142] For other details about the processor in the above-mentioned computer device implementing the above-mentioned technical solution, please refer to the description of the container detection and scheduling method provided in the above-mentioned invention embodiment, which will not be repeated here.

[0143] Among them, the processor can also be called CPU (Central Processing Unit), the processor may be an integrated circuit chip with signal processing capabilities; the processor can also be a general-purpose processor, DSP (Digital Signal Process), ASIC (Application Specific Integrated Circuit), FPGA (Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, among which the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0144] The embodiment of the present invention also provides a computer-readable storage medium, on which a readable computer program is stored; wherein the computer program can be stored in the above storage medium in the form of a software product, including several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, magnetic disk or optical disk, ROM (Read-Only Memory), RAM (Random Access Memory) and other media that can store program codes, or terminal devices such as computers, servers, mobile phones, and tablets.

[0145] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0146] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0148] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0149] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, a computer, a server, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a server or a data center that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0150] The technical solution provided by the present application is introduced in detail above. The principles and implementation methods of the present application are explained by using specific examples in the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0151] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0152] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0155] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A container detection and scheduling method, characterized in that: The method comprises: Acquire multimodal data of the container, the multimodal data comprising at least two of text data, image data, infrared data, point cloud data, and radiographic image data of the container; According to the multimodal data and the multimodal large model, semantic alignment is performed to extract multimodal aligned text, wherein the multimodal large model is obtained by semantic alignment training using contrastive learning and parameter fine-tuning operations; Generate a task plan based on the multimodal alignment text and the port operation scheduling model; According to the mission plan, port scheduling is performed and the mission plan is executed.

2. The container detection and scheduling method according to claim 1, characterized in that: The multimodal large model includes a language encoder, a multimodal encoder and a multimodal joint training module. The multimodal large model is obtained by performing semantic alignment training using contrastive learning and parameter fine-tuning operations, and the steps include: Inputting a text sample into the language encoder to obtain text features; Inputting image samples, infrared samples, point cloud samples and radiographic image samples into the multimodal encoder, and performing parameter fine-tuning operations to obtain image features, infrared features, point cloud features and radiographic features; Based on the text features, semantic alignment training is performed on the image features, the infrared features, the point cloud features and the ray features to obtain a trained multimodal large model.

3. The container detection and scheduling method according to claim 1, characterized in that: The contrastive learning loss formula of the multimodal large model is as follows: Among them, L contrast represents the contrastive learning loss, L m2t and L t2m are similarity losses, K is the number of multimodal and text pairs in each batch during the training of the multimodal large model, i and j are both positive integers, and m i is the feature vector of the i-th multimodal data, t j is the feature vector of the j-th text data, T represents the amount of the multimodal data, N represents the amount of the text data, τ is the temperature parameter, and sim represents the cosine similarity.

4. The container detection and scheduling method according to claim 1, characterized in that: The port operation scheduling model includes a domain identification module, a job classifier and a semantic slot filling classifier. The steps of generating a task plan according to the multimodal alignment text and the port operation scheduling model include: Inputting the multimodal aligned text into the domain identification module to obtain candidate domains; Inputting the candidate fields into the job classifier and the semantic slot filling classifier respectively to obtain the job classification probability of the semantic representation and the prediction probability of the slot filling text; The task plan is generated according to the job classification probability and the predicted probability of the slot filling text.

5. The container detection and scheduling method according to claim 4, characterized in that: The calculation formula of the job classification probability is as follows: P(y i |T)=softmax(W i ·H cls +b i ): i∈1…M; Among them, P(y i |T) is the operation classification probability Pi of the semantic representation T, Wi represents the classification weight of the i-th port operation, b i represents the classification bias of the i-th port operation, M is the total number of port operation classifications, cls represents a special word, and H cls is the feature vector corresponding to the special word, i and M are positive integers; The calculation formula for the predicted probability of the slot filling text is as follows: n∈1…N; in, The predicted probability of filling the slot of the semantic representation T with text W s is the classification weight filled in the sth slot, b s is the graph bias of the sth slot filling, H n is the nth element of the feature vector H, n, N, s are positive integers, and N is the total number of word unit sequences corresponding to the special word unit.

6. The container detection and scheduling method according to any one of claims 1 to 5, characterized in that: The method further comprises: The damage information of the container is extracted according to the multimodal data and the multimodal large model.

7. The container detection and scheduling method according to any one of claims 1 to 5, characterized in that: The steps of performing port dispatch and executing the task plan according to the task plan include: Prioritize each task in the task plan and store the priority in the task queue; The tasks in the task queue are scheduled to be executed in sequence.

8. A container detection and dispatching device, characterized in that: The device comprises: A collection module, used to obtain multimodal data of the container, wherein the multimodal data includes at least two of text data, image data, infrared data, point cloud data and radiographic image data of the container; A detection module, used to perform semantic alignment according to the multimodal data and the multimodal large model, and extract multimodal aligned text, wherein the multimodal large model is obtained by semantic alignment training using contrastive learning and parameter fine-tuning operations; A generation module, used for generating a task plan according to the multimodal alignment text and the port operation scheduling model; The scheduling module is used to perform port scheduling and execute the task plan according to the task plan.

9. A computer device, characterized in that: The device includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; The processor is used to implement the steps of the container detection and scheduling method according to any one of claims 1 to 7 when executing the program stored in the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the container detection and scheduling method according to any one of claims 1 to 7 are implemented.

Images