Logistics transportation safety compliance multi-mode automatic detection system and method

Abstract

The invention provides a logistics transportation safety compliance multi-mode automatic detection system and method, and relates to the technical field of visual identification, and the method comprises the steps: S1, receiving a to-be-detected image uploaded by a client and associated task identification information; s2, according to the task identification information, a corresponding dynamic cue word template is matched and obtained from a rule configuration library, the dynamic cue word template defines at least two virtual roles with a strict execution sequence and task logics of the virtual roles in a text form, and the virtual roles comprise a first anti-counterfeiting verification role and a second business compliance reasoning role; s3, inputting the to-be-detected image and the cue word packaged with the first anti-counterfeiting verification role task into a multi-modal large language model; the cue word instructs the multi-modal large language model to play a first anti-counterfeiting verification role, and executes a task of identifying whether the image contains an anti-counterfeiting watermark in a preset format or not. The beneficial effects of the invention are that the method is high in data authenticity, is very low in maintenance cost, and facilitates the improvement of the accuracy.

Description

Technical Field

[0001] This invention relates to the field of visual recognition technology, and in particular to a multimodal automatic detection system and method for logistics transportation safety and compliance. Background Technology

[0002] Safety and compliance supervision in the logistics and transportation industry is a crucial link in ensuring property security and the healthy development of the industry. Currently, the industry mainly uses the following two technical solutions for logistics supervision:

[0003] The first approach is a purely manual review scheme. This scheme requires reviewers to examine each on-site photo uploaded by the driver, visually judging whether there are anti-counterfeiting watermarks such as time and location in the images, and judging whether the cargo bundling method and vehicle facilities meet the requirements based on their personal understanding of safety regulations. However, this scheme has many drawbacks: First, the review efficiency is extremely low, making it difficult to handle large-scale, high-concurrency image upload scenarios; second, the review standards are highly subjective, easily leading to misjudgments or omissions due to staff fatigue, differences in experience, etc.; finally, labor costs increase linearly with the growth of business volume, lacking economic scalability.

[0004] The second type is an automated solution based on traditional computer vision, such as the visual recognition-based hazardous chemical data processing and supervision system and method disclosed in application number CN118569762A. This method attempts to use convolutional neural networks or object detection models to identify specific violations such as not wearing a safety helmet or reflective clothing. However, this type of solution has certain technical limitations: First, it lacks deep logical reasoning capabilities. Traditional visual models can only identify what an object is, but cannot perform state judgments such as how or why, which are precisely the core requirements of compliance inspection. Second, its anti-counterfeiting and anti-interference capabilities are weak. Traditional OCR technology can only extract text information from images and cannot perceive advanced features such as the degree of fusion between the watermark and the background image, and the consistency of light and shadow. Therefore, it is difficult to effectively identify watermarks forged by post-processing techniques such as Photoshop overlay. Finally, the system is rigid and has poor generalization ability. Whenever the type of transported goods increases or safety rules are updated, a large amount of data needs to be collected, labeled, and a dedicated model needs to be trained again for the new detection targets, resulting in high system maintenance costs and slow response speed.

[0005] In other words, existing technologies cannot achieve intelligent, refined, and scalable automatic detection of logistics transportation safety and compliance while ensuring high efficiency and low cost. In particular, they lack effective means to integrate and configurable process image anti-counterfeiting verification with deep business logic judgment. Summary of the Invention

[0006] This invention overcomes the shortcomings of the prior art and provides a multimodal automatic detection system and method for logistics transportation safety and compliance, which has the advantages of high data authenticity, extremely low maintenance cost, and helps to improve accuracy.

[0007] To solve the above-mentioned technical problems, the present invention is achieved through the following technical solution:

[0008] A multimodal automatic detection method for logistics transportation safety and compliance includes the following steps:

[0009] Step S1: Receive the image to be detected and the associated task identifier information uploaded by the client;

[0010] Step S2: Based on the task identification information, match and obtain the corresponding dynamic prompt word template from the rule configuration library. The dynamic prompt word template defines at least two virtual roles with a strict execution order and their task logic in text form, including a first anti-counterfeiting verification role and a second business compliance reasoning role.

[0011] Step S3: Input the image to be detected and the prompt word encapsulating the first anti-counterfeiting verification role task into the multimodal large language model; the prompt word instructs the multimodal large language model to play the first anti-counterfeiting verification role and perform the task of identifying whether the image contains an anti-counterfeiting watermark that conforms to a preset format;

[0012] Step S4: If the task judgment of the first anti-counterfeiting verification role passes, the image to be detected and the prompt words encapsulating the task of the second business compliance reasoning role are input into the multimodal large language model; the prompt words instruct the multimodal large language model to play the role of the second business compliance reasoning role, and to perform the security compliance judgment task corresponding to the task identification information based on the visual understanding and logical reasoning of the image.

[0013] Step S5: Receive the structured judgment result output by the multimodal large language model, and generate feedback information based on the result.

[0014] Furthermore, in step S3, if the task determination of the first anti-counterfeiting verification role fails, the process is terminated and a structured result containing anti-counterfeiting verification failure information is directly output.

[0015] Furthermore, the dynamic prompt word template is a text template, in which business parameters determined by the task identification information are referenced by variable interpolation. The business parameters include at least the type of component to be detected and / or specific compliance judgment criteria.

[0016] Furthermore, the rule configuration library is used to store several dynamic prompt word templates corresponding to different business scenarios; when a new business scenario is added or compliance rules are changed, the system function is updated by updating the corresponding template text in the rule configuration library, without having to retrain the multimodal large language model.

[0017] Furthermore, in step S5, the structured judgment result is a JSON format object, which includes at least a boolean field indicating whether the overall result passes and an array field describing the detailed judgment reasons; the dynamic prompt word template includes an instruction to force the output to be in this JSON format.

[0018] Furthermore, the task of the first anti-counterfeiting verification role specifically includes: scanning a preset area of ​​the image to be detected, and finding and identifying digital watermark information that conforms to the date or time format.

[0019] A multimodal automatic detection system for logistics transportation safety and compliance, used to implement the method described in any one of the above embodiments, the system comprising:

[0020] The interface module is used to receive the image to be detected and the associated task identification information uploaded by the client;

[0021] The rule configuration library is used to store dynamic prompt word templates corresponding to several business scenarios in text format;

[0022] The rules engine module is used to match the corresponding dynamic prompt word template from the rule configuration library based on the task identification information, and assemble and generate a complete prompt word containing the role task definition;

[0023] The multimodal large language model inference module is used to receive the prompt words assembled by the rule engine module and the image to be detected, and to perform anti-counterfeiting verification and security compliance business inference in sequence according to the role definition in the prompt words;

[0024] The parsing feedback module is used to parse the structured results output by the multimodal large language model inference module and generate compliance detection feedback information.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] By leveraging the large model's understanding of global image information, logical forgeries that traditional technologies cannot identify can be identified, significantly improving data authenticity. There is no need to retrain the model; when adding components or changing rules, only adding or modifying text in the database is required, and the system takes effect immediately, resulting in extremely low maintenance costs. This solves the pain point of traditional AI's inability to judge the state, contributing to improved accuracy. Attached Figure Description

[0027] The accompanying drawings are provided to further illustrate the invention and are used together with the embodiments of the invention to explain the invention. They do not constitute a limitation of the invention. In the drawings:

[0028] Figure 1 This is a schematic diagram of the overall process.

[0029] Figure 2 It is a dual-role judgment logic diagram, that is, the core algorithm process of sequential execution and condition triggering of the two roles of watermark prosecutor and component inspector.

[0030] Figure 3 This is a diagram illustrating the dynamic assembly of Prompt. Detailed Implementation

[0031] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0032] like Figures 1 to 3 As shown, this invention claims protection for a multimodal automatic detection method for logistics transportation safety and compliance, comprising the following steps:

[0033] Step S1: Receive the image to be detected and the associated task identifier information uploaded by the client;

[0034] Step S2: Based on the task identification information, match and obtain the corresponding dynamic prompt word template from the rule configuration library. The dynamic prompt word template defines at least two virtual roles with a strict execution order and their task logic in text form, including a first anti-counterfeiting verification role and a second business compliance reasoning role.

[0035] Step S3: Input the image to be detected and the prompt word encapsulating the first anti-counterfeiting verification role task into the multimodal large language model; the prompt word instructs the multimodal large language model to play the first anti-counterfeiting verification role and perform the task of identifying whether the image contains an anti-counterfeiting watermark that conforms to a preset format;

[0036] Step S4: If the task judgment of the first anti-counterfeiting verification role passes, the image to be detected and the prompt words encapsulating the task of the second business compliance reasoning role are input into the multimodal large language model; the prompt words instruct the multimodal large language model to play the role of the second business compliance reasoning role, and to perform the security compliance judgment task corresponding to the task identification information based on the visual understanding and logical reasoning of the image.

[0037] Step S5: Receive the structured judgment result output by the multimodal large language model, and generate feedback information based on the result.

[0038] Specifically,

[0039] Step S1 refers to the process where, at the logistics transportation site, the driver or operator uses a mobile terminal such as a smartphone or tablet, or a dedicated vehicle-mounted device, to take photos of the cargo loading, vehicle status, and safety facilities, and then uploads them through a client application or web interface. When uploading, the corresponding task identifier information for this inspection must be selected or entered, and it should be noted that the identifier information is used to clearly define the specific business scenario of this inspection. For example, if it is the transportation of wind turbine blades, the inspection will focus on the compliance of the blade root support binding.

[0040] Step S2 refers to the establishment of a rule configuration library in the system backend, typically stored in YAML, JSON, or database tables, pre-defining a dynamic prompt word template for each business scenario. Each dynamic prompt word template is a structured text fragment. The core innovation lies in the explicit definition of at least two virtual roles with a strict execution order and their task logic. Typical role designs include:

[0041] First role: Anti-counterfeiting verification role, such as watermark prosecutor, responsible for verifying the authenticity and timeliness of images.

[0042] The second role is the business compliance reasoning role, such as a component inspector or security equipment inspector, who is responsible for making specific business compliance judgments based on image content.

[0043] The system searches the rule configuration library based on the received task identifier information and extracts the corresponding dynamic prompt word template.

[0044] Step S3 refers to the rule engine assembling the image to be detected and the prompt words encapsulating the first role task into a complete input, which is then submitted to the multimodal large language model. The prompt words explicitly instruct the model to play the first anti-spoofing verification role.

[0045] In this embodiment, the specific task of the character is to scan a preset area of ​​the image to be detected, and to find and identify digital watermark information that conforms to the date or time format. Watermarks are typically text information automatically superimposed by a photo-taking app, containing the time the image was taken.

[0046] The model performs visual analysis and logical judgments as follows:

[0047] If a compliant watermark is detected, meaning it contains valid time information, the task for that role is deemed successful, and the process continues. If no watermark is detected, or the watermark does not contain time information, the task for that role is deemed unsuccessful. In this case, the process will terminate immediately, and subsequent, more costly business reasoning will not be performed. The model directly outputs a structured result containing information about the anti-counterfeiting verification failure.

[0048] Step S4 specifically refers to the system activating the second role after the first role's judgment is successful. The rule engine assembles the same image to be detected with prompts that encapsulate the second role's task and submits it again to the multimodal large language model. At this time, the model plays the role of the second business compliance reasoning according to the instructions.

[0049] The second business compliance reasoning role's task is to perform specific security and compliance judgments corresponding to task identification information based on deep visual understanding and logical reasoning of images. This goes beyond the simple object recognition of traditional visual AI, requiring the model to understand object states, spatial relationships, and business rules.

[0050] For example, as mentioned earlier, in the scenario of wind turbine blades and blade root supports, the prompts from the second role would explicitly require determining whether the blade root supports are cross-linked with iron chains. Then, the multimodal large language model needs to identify the iron chains and supports in the image and analyze whether the direction of the iron chains constitutes a cross and whether the binding state is tight, rather than simply identifying that there are iron chains and supports.

[0051] Step S5 refers to the output of the results after the multimodal large language model completes inference. The key technique is to enforce a strict structured format for the model's output through the Prompt project, preventing it from freely generating natural language.

[0052] If the structured judgment result is a JSON object, the JSON object must contain at least two core fields:

[0053] Pass: Boolean, indicating whether the overall test passed.

[0054] message: A string array used to describe in detail the reason for the judgment. When pass is false, this array will contain one or more specific violation descriptions; when pass is true, this array can be empty.

[0055] The system backend parses this JSON result. If the detection fails, the parsing feedback module will extract the content of the message, generate specific rectification suggestions, and provide real-time feedback to the driver who uploaded the image through message push, APP notification, or SMS to guide them to make on-site rectifications.

[0056] In addition, the dynamic prompt word template is a text template that uses variable interpolation technology. The template contains placeholders marked with curly braces, which will be replaced by specific business parameters at runtime.

[0057] The example template structure is as follows:

[0058] The watermark inspector scans the image to check for watermarks containing date or time formatted text in the four corners. Then, the component inspector checks whether the image meets the following safety requirements: when the task identification information is wind turbine blades and blade root supports, the rule engine will query the corresponding parameter values ​​from the business database and replace them.

[0059] The rule configuration library stores templates for all business scenarios. When the business scope expands or safety regulations are updated, maintenance personnel do not need to modify the code or retrain the AI ​​model. For example, to add a new scenario involving hazardous chemical transportation and valve seal inspection, simply add a record to the rule configuration library and create a new template defining the roles of watermark inspection and seal integrity inspection. The system enables hot updates of functions through configuration, greatly reducing operation and maintenance costs and technical barriers.

[0060] The present invention also claims protection for a system for implementing the above method, comprising the following modules:

[0061] Interface module: As the system entry point, it provides a RESTful API or message queue interface, and is responsible for receiving the images and metadata to be detected uploaded from the driver client, and performing preliminary verification, traffic limiting and other processing.

[0062] Rule configuration library: Utilizing persistent storage such as a MySQL database, Redis, or file server, all dynamic prompt word templates are stored in key-value pairs or document format. The key is typically `rule_id`, and the value is the corresponding template text and associated variables.

[0063] The rules engine module's workflow is as follows:

[0064] First, the task identifier information is obtained from the interface module. Then, the original dynamic prompt word template is obtained from the configuration library based on the identifier query rules. Next, the specific business parameter values ​​are obtained from other business systems or configurations based on the task identifier. Finally, template rendering is performed, replacing variable placeholders with actual parameter values, and assembling them into the final prompt word containing complete role definitions, ready to be sent to the large model.

[0065] The multimodal large language model inference module encapsulates calls to one or more multimodal large language model services. It receives prompt words and images to be detected from the rule engine, calls the large model API such as Azure OpenAI or the locally deployed Qwen-VL API, and returns the raw output of the large model. It is primarily responsible for handling the details of interacting with the large model, such as token counting, request retries, and exception handling.

[0066] The feedback parsing module receives the structured text returned by the inference module. First, it performs rigorous JSON syntax parsing and format validation. Then, it extracts key fields and generates a standardized detection report. Finally, it sends the detection results to the user via a message gateway, push service, or by directly calling the client callback interface.

[0067] The above modules form a complete automated closed-loop system from data uploading and intelligent analysis to result feedback.

[0068] This invention utilizes a large model's understanding of global image information to identify logical forgeries that traditional technologies cannot detect, significantly improving data authenticity. No model retraining is required; when new components are added or rules are changed, only text needs to be added or modified in the database, and the system takes effect immediately, resulting in extremely low maintenance costs. It solves the pain point of traditional AI's inability to determine states, contributing to improved accuracy.

[0069] Finally, it should be noted that the above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. However, any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multimodal automatic detection method for logistics transportation safety and compliance, characterized in that, Includes the following steps: Step S1: Receive the image to be detected and the associated task identifier information uploaded by the client; Step S2: Based on the task identification information, match and obtain the corresponding dynamic prompt word template from the rule configuration library. The dynamic prompt word template defines at least two virtual roles with a strict execution order and their task logic in text form, including a first anti-counterfeiting verification role and a second business compliance reasoning role. Step S3: Input the image to be detected and the prompt word encapsulating the first anti-counterfeiting verification role task into the multimodal large language model; the prompt word instructs the multimodal large language model to play the first anti-counterfeiting verification role and perform the task of identifying whether the image contains an anti-counterfeiting watermark that conforms to a preset format; Step S4: If the task judgment of the first anti-counterfeiting verification role passes, the image to be detected and the prompt words encapsulating the task of the second business compliance reasoning role are input into the multimodal large language model; the prompt words instruct the multimodal large language model to play the role of the second business compliance reasoning role, and to perform the security compliance judgment task corresponding to the task identification information based on the visual understanding and logical reasoning of the image. Step S5: Receive the structured judgment result output by the multimodal large language model, and generate feedback information based on the result.

2. The method according to claim 1, characterized in that, In step S3, if the task determination of the first anti-counterfeiting verification role fails, the process is terminated and a structured result containing anti-counterfeiting verification failure information is directly output.

3. The method according to claim 1, characterized in that, The dynamic prompt word template is a text template in which business parameters determined by the task identification information are referenced by variable interpolation. The business parameters include at least the type of component to be detected and / or specific compliance judgment criteria.

4. The method according to claim 3, characterized in that, The rule configuration library is used to store several dynamic prompt word templates corresponding to different business scenarios. When a new business scenario is added or compliance rules are changed, the system function is updated by updating the corresponding template text in the rule configuration library, without having to retrain the multimodal large language model.

5. The method according to claim 1, characterized in that, In step S5, the structured judgment result is a JSON format object, which includes at least a boolean field indicating whether the overall result passes or fails, and an array field describing the detailed judgment reasons; the dynamic prompt word template includes an instruction to force the output to be in this JSON format.

6. The method according to claim 1, characterized in that, The specific tasks of the first anti-counterfeiting verification role include: scanning a preset area of ​​the image to be detected, and finding and identifying digital watermark information that conforms to the date or time format.

7. A multimodal automatic detection system for logistics transportation safety and compliance, characterized in that, The system for implementing the method as described in any one of claims 1 to 6 comprises: The interface module is used to receive the image to be detected and the associated task identification information uploaded by the client; The rule configuration library is used to store dynamic prompt word templates corresponding to several business scenarios in text format; The rules engine module is used to match the corresponding dynamic prompt word template from the rule configuration library based on the task identification information, and assemble and generate a complete prompt word containing the role task definition; The multimodal large language model inference module is used to receive the prompt words assembled by the rule engine module and the image to be detected, and to perform anti-counterfeiting verification and security compliance business inference in sequence according to the role definition in the prompt words; The parsing feedback module is used to parse the structured results output by the multimodal large language model inference module and generate compliance detection feedback information.

Images