Intelligent video inspection method and system based on multi-modal large model

Abstract

The invention belongs to the field of airport management, and relates to an intelligent video inspection method and system based on a multi-modal large model, and the method comprises the steps: 1, constructing a structured knowledge graph based on an unstructured text in airport data, and obtaining a high-dimensional semantic vector based on nodes of the structured knowledge graph; step 2, performing feature extraction on various data including the monitoring video to generate multi-modal joint feature representation; the multi-modal joint feature representation comprises spatio-temporal information and a rule context; step 3, based on the high-dimensional semantic vector and the multi-modal joint feature representation, judging whether an abnormal event occurs in the terminal; 4, if the abnormal event occurs, generating a structured instruction based on the abnormal event, and performing closed-loop optimization on the terminal based on the structured instruction; on the basis of the existing video monitoring environment and data of an airport, an integrated intelligent system is provided for intelligent inspection of an airport terminal, and automatic, refined and closed-loop management of inspection is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of airport management, and specifically discloses a smart video inspection method and system based on a multi-modal large model. BACKGROUND

[0002] Currently, video inspection of airport terminals mainly focuses on the following aspects: high dependence on manpower and low efficiency: the inspection work is mainly manually inspected in 23 time periods, and the coverage range and timeliness are limited by human resources, which is prone to fatigue and omission. Regional and process supervision are separated: terminal video inspection is mainly divided into regional supervision and process node supervision. Some regions use a passive waiting event reporting mode, which is difficult to actively discover and intervene in potential events. The trigger mechanism is simple and rough: existing intelligent analysis mainly uses crowd density as the main trigger point, and manually checks high-density areas, lacking fine-grained identification and correlation analysis of complex service and safety events. The existing smart inspection based on video analysis mainly has the following shortcomings: incomplete event coverage: existing video analysis technology can only analyze some common events, such as key node passenger flow statistics and forbidden zone intrusion, and has serious deficiencies in coverage and identification accuracy for complex, multi-step, and non-standardized service and management events (such as unattended service desk, passenger help-seeking behavior, and facility damage). It stays at the perception level and lacks a closed-loop management: from the perspective of airport terminal service and management, for process and regional supervision, the existing analysis only stays at the perception level (event identification and early warning) and cannot achieve full-process and automated analysis and triggering from supervision standards to disposal actions, making it difficult to form an efficient management closed loop. Single data modality, insufficient information utilization: traditional video analysis mainly relies on image / video data and cannot effectively utilize and integrate existing text information (such as regulations and standards, standard operating procedures SOP) and audio information (such as broadcasts and calls) in the airport, resulting in limited understanding and judgment ability for complex scenarios.

[0003] Therefore, the present application proposes a smart video inspection method and system based on a multi-modal large model to provide basic data fusion, advanced semantic analysis, real-time early warning, and decision support for terminal smart inspection based on the existing video monitoring environment and data of the airport, and to build an integrated intelligent system of "perception-cognition-decision-action" to realize the automation, refinement, and closed-loop management of inspection. SUMMARY

[0004] The purpose of the present application is to provide a multi-modal large model-based intelligent video inspection method and system, which solves the problem of how to assist airports in implementing inspection, supervision and disposition of various events in the terminal building through video and multi-modal large model-based analysis means, thereby improving passenger service quality, reducing safety accidents, improving inspection efficiency and fine management level; the specific scheme is as follows: A multi-modal large model-based intelligent video inspection method, comprising: step 1, constructing a structured knowledge graph based on unstructured text in airport data, and obtaining a high-dimensional semantic vector based on nodes of the structured knowledge graph; step 2, performing feature extraction on various data including monitoring videos to generate multi-modal joint feature representation; the multi-modal joint feature representation includes spatio-temporal information and rule context; step 3, judging whether an abnormal event occurs in the terminal building based on the high-dimensional semantic vector and the multi-modal joint feature representation; step 4, if an abnormal event occurs, generating a structured instruction based on the abnormal event, and performing closed-loop optimization on the terminal building based on the structured instruction.

[0005] Further, based on unstructured text in airport data, a structured knowledge graph is constructed, and a high-dimensional semantic vector is obtained based on nodes of the structured knowledge graph, comprising: step 1.1, using natural language processing technology to analyze the unstructured text and extract a core entity set and a relationship set; the unstructured text includes SOP documents, supervision standard lists and job responsibility instructions; step 1.2, constructing a structured knowledge graph according to the core entity set and the relationship set; the entities of the structured knowledge graph include events and disposition actions, and the edges include standard relationships and causal relationships; step 1.3, performing semantic vectorization on the standard nodes of the structured knowledge graph to obtain the high-dimensional semantic vector of the standard nodes; the high-dimensional semantic vector includes semantic information and context information of the standard nodes.

[0006] Further, the core entity set includes supervision items, supervision standards, disposition actions, responsible units and time constraints; the relationship set includes triggering, corresponding, responsible subject and time requirement.

[0007] Further, performing feature extraction on various data including monitoring videos to generate multi-modal joint feature representation, comprising: step 2.1, performing video spatio-temporal feature extraction on the monitoring videos to obtain a high-dimensional visual feature vector; the high-dimensional visual feature vector is used to represent micro-target features and macro-group behavior features; step 2.2, retrieving relevant standard operating procedure text segments according to the scene shown in the current video frame, and generating a context knowledge vector based on the standard operating procedure text segments; the context knowledge vector represents the semantic representation of the rules that should be followed in the current scene; step 2.3, aligning and fusing the high-dimensional visual feature vector and the context knowledge vector to obtain the multi-modal joint feature representation.

[0008] Further, the micro-target features include a unique identifier of each pedestrian in the video frame, a 3D spatial coordinate in the terminal coordinate system, and a real-time moving speed; the macro-group behavior features include a crowd density in a specified area, a queuing length of a queuing area, and a spatial distribution of a crowd gathering area.

[0009] Further, the calculation formula of the multi-modal joint feature representation is: ; wherein, is the multi-modal joint feature representation; CMFAN is the cross-modal feature attention network; is a high-dimensional visual feature vector; is a context knowledge vector.

[0010] Further, based on the high-dimensional semantic vector and the multi-modal joint feature representation, a structured instruction is generated, and the terminal is optimized based on the structured instruction, including: step 3.1, calculating the similarity of the multi-modal joint feature representation and a plurality of high-dimensional semantic vectors, respectively, and determining an activated graph node and a potential abnormal event based on a similarity threshold; step 3.2, constructing context knowledge based on neighbor node information of the activated graph node, and performing probabilistic reasoning based on the multi-modal joint feature representation and the context knowledge to obtain an event determination result and a confidence probability; step 3.3, when the confidence probability is greater than a preset confidence threshold, it is determined that the potential abnormal event occurs.

[0011] Further, the calculation formula of the similarity of the multi-modal joint feature representation and a plurality of high-dimensional semantic vectors is: ; wherein, ActivationLevel is the similarity; CosineSimilarity is a cosine similarity algorithm; is the multi-modal joint feature representation of the current video; is a high-dimensional semantic vector of the supervision standard.

[0012] Further, if an abnormal event occurs, structured instructions are generated based on the abnormal event, and the terminal is closed-loop optimized based on the structured instructions, including: step 4.1, generating structured instructions based on the abnormal event and the abnormal event association; the abnormal event association includes a handling relationship and a time limit relationship; step 4.2, distributing and executing the structured instructions, and recording the optimization time; step 4.3, obtaining the execution feedback text, and verifying the effectiveness of the structured instructions based on the execution feedback text and the optimization high-dimensional visual feature vector; step 4.4, if the structured instructions are effective, continuously adjust based on the structured instructions until the abnormal event is eliminated; step 4.5, if the structured instructions are not effective, adjust the confidence threshold and regenerate the structured instructions; step 4.6, if the abnormal event has not been eliminated when the optimization time is greater than the optimization time threshold, an alarm is given.

[0013] The application further provides a multi-modal large model-based intelligent video inspection system applied to the multi-modal large model-based intelligent video inspection method, comprising a graph processing module, a joint processing module, an abnormality judgment module and an execution module; the graph processing module is used to construct a structured knowledge graph based on unstructured text in airport data, and obtain a high-dimensional semantic vector based on nodes of the structured knowledge graph; the joint processing module is used to perform feature extraction on various data including monitoring videos, and generate multi-modal joint feature representation; the multi-modal joint feature representation includes space-time information and rule context; the abnormality judgment module is used to judge whether an abnormal event occurs in the terminal based on the high-dimensional semantic vector and the multi-modal joint feature representation; and the execution module is used to generate structured instructions based on the abnormal event if the abnormal event occurs, and perform closed-loop optimization on the terminal based on the structured instructions.

[0014] The application has the following advantages and beneficial effects: The application introduces a multi-modal large model and a knowledge graph, deeply fuses video visual information with text knowledge such as management SOP and regulations of the airport, solves the problem of insufficient recognition ability of complex and semantic events in traditional video analysis, and realizes comprehensive coverage of service and management events.

[0015] The application constructs a standardized knowledge graph of supervision standards and a similarity calculation method, so that the analysis result no longer stays at a simple perception and early warning level, but can be directly associated with the supervision standards and SOP of the airport, automatically generates and triggers full-process handling actions, and greatly improves the fine management and emergency response efficiency of the airport.

[0016] The application unifies heterogeneous data (video, text) in a high-dimensional space through a multi-modal feature fusion method, so that the large model can perform high-order semantic reasoning. Improve the automation and intelligence level of inspection: change passive waiting to active, real-time and comprehensive intelligent inspection, significantly reduce the intensity and frequency of manual inspection, reduce labor costs, and improve the prevention ability of safety accidents and passenger service quality. BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 An exemplary flowchart of a smart video inspection method based on a multi-modal large model is provided. DETAILED DESCRIPTION

[0018] To make the purposes, technical solutions and advantages of the embodiments of the application clearer, the technical solutions in the embodiments of the application will be described clearly and completely below with reference to the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all embodiments. The components of the embodiments of the application described and shown in the drawings herein can be arranged and designed in various different configurations.

[0019] The content of the application will be described in detail and logically coherently below in combination with the collected airport terminal video data, standard operating procedure (SOP) text and related supervision standards. In order to clearly show how the application converts from static rules to dynamic intelligent decision-making, as shown in Figure 1 The entire implementation process is divided into four major steps, each step including the following content: Step 1, build a structured knowledge graph. The structured knowledge graph is a structured knowledge graph based on "event-standard-disposal". This method aims to convert high-precision, process management requirements, SOP and other unstructured text in terminal flow node supervision and area supervision into structured knowledge graphs (Structured Knowledge Graph, SKG) that can be queried and reasoned by large models in real time, and establish the association between events. Including the following steps: The core task of step 1 is to convert the existing, unstructured natural language text in the airport data into structured knowledge graphs (Structured Knowledge Graph, SKG) that can be read, queried and reasoned by machine algorithms through digital means.

[0020] Step 1.1, parsing unstructured text using natural language processing techniques to extract core entity set and relationship set. First, receive the airport data provided by the airport management department, and use natural language processing (NLP) techniques to deeply analyze the unstructured text in the airport data. Unstructured text mainly includes SOP documents, supervision standard lists, and job responsibility descriptions, etc. By scanning the keywords in the text, identify the nouns and verb phrases with specific management meaning, and map them to predefined entity types and relationship types to obtain the core entity set and relationship set.

[0021] For the core entity set, the core entities and relationships in the supervision standard can be extracted. From the supervision content provided by the airport, the entity E and the relationship R are accurately extracted. Entity E can include supervision items , supervision standards , disposal actions , responsible units and time constraints , etc. Supervision items include counter check-in and garbage cans, etc.; supervision standards include whether the queue is crowded and whether the garbage is overflowing, etc.; disposal actions can be to notify the airline to add counters within 5 minutes, etc.; responsible units can include airlines; time constraints can be 5 minutes, etc. Identifying entity E can construct the core entity set : . For example, the specific object of the inspection is extracted from the document ; the determination basis of the abnormal event is extracted ; the specified rectification time is extracted .

[0022] For the relationship set, relationship R can include trigger , corresponding , responsible subject and time requirement , etc. Trigger can include event trigger standard; corresponding can include standard corresponding action; responsible subject can include action performer; time requirement can include the time when the action needs to be completed. Extract relationship R and construct relationship set : . For example, the describing the cause and effect of the event is extracted, indicating that a phenomenon will cause a standard to be activated; the describing the attribution of rights and responsibilities is extracted, indicating who is responsible for solving the problem.

[0023] Step 1.2, according to the core entity set and relationship set, according to the graph construction rules, link the discrete entities through the relationships, and batch construct structured knowledge triples with attributes, in the format of . These triples together constitute the structured knowledge graph The entities of the structured knowledge graph include events and handling actions, and the edges include standard relationships and causal relationships. may be: For example, the triples in the structured knowledge graph can include event definition triples, handling association triples, and time limit triples, etc., wherein the event definition triples can be: (check-in counter, triggered by, queue congestion and cross isolation belt), which defines what kind of situation at the check-in counter is abnormal; the handling association triples can be: (queue congestion and cross isolation belt, corresponding to, open additional counter to relieve queue), which defines the standard action to be taken after the abnormality; and the time limit triples can be: (open additional counter to relieve queue, time limit requirement, 5 minutes), which defines the time window in which the action must be completed, which is the basis for subsequent automatic supervision.

[0024] Step 1.3, semantic vectorization of the nodes of the structured knowledge graph, to obtain high-dimensional semantic vectors of the nodes. Since the computer cannot directly understand the character meaning of "queue" or "congestion", a pre-trained deep language model (such as BERT, etc.) can be used to perform Embedding coding on each standard node in the constructed knowledge graph , and convert it into a numerical vector in a high-dimensional space. A high-dimensional semantic vector of each standard node is generated . This parameter not only contains semantic information (such as literal meaning) of the text, but also implicitly contains context information (such as information of other nodes connected to the standard node), which will be directly used as a comparison reference in step 3.1 for similarity matching with real-time video features.

[0025] The structured knowledge graph construction and vectorization method based on "event-standard-handling" proposed by the present application converts the unstructured standard operating procedure (SOP) documents of the terminal into structured knowledge and semantic vectors that can be understood by machines, provides standard basis for intelligent reasoning, and realizes accurate dynamic association between supervision standards and video events.

[0026] Step 2, feature extraction of various data including monitoring videos, to generate multi-modal joint feature representation. By processing massive monitoring video streams in real time and combining the context of SOP text, multi-modal joint feature representation containing rich spatio-temporal information and semantic information is generated. The multi-modal joint feature representation includes spatio-temporal information and rule context.

[0027] Step 2.1, advanced computer vision models (such as Transformer-based Vision Model or 3D-CNN) are used to extract video spatio-temporal feature information from real-time monitoring video frame sequences from cameras in various areas of the terminal, and high-dimensional visual feature vectors The video spatio-temporal feature extraction step not only detects objects in a single frame of image, but also analyzes the temporal changes between multiple frames. The high-dimensional visual feature vector can be used to represent micro-target features and macro-group behavior features. For example, micro-target features can include the unique identification ID of each pedestrian in the video frame, 3D spatial coordinates (x, y, z) in the terminal coordinate system, and real-time moving speed v, etc.; macro-group behavior features can include the crowd density p in a specified area, the queue length L of the key queuing area, and the spatial distribution of the crowd gathering area, etc.

[0028] Step 2.2, according to the scene shown in the current video frame, retrieve the relevant standard operating procedure text segment, and generate the context knowledge vector based on the standard operating procedure text segment. For example, according to the physical location of the scene shown in the current video obtained by monitoring, the relevant SOP text segment is retrieved from the document library. The physical location of the scene can include the security check area and the check-in island, etc. The retrieved standard operating procedure text segment is input into the pre-trained knowledge-enhanced large language model for encoding to obtain the context knowledge vector . The knowledge-enhanced large language model can include ERNIE or a specially fine-tuned LLaMA model, etc. The context knowledge vector represents the semantic representation of the rules that should be followed in the current scene.

[0029] Step 2.3, after aligning the high-dimensional visual feature vector and the context knowledge vector, cross-modal feature fusion is performed to obtain a multi-modal joint feature representation. Since video features and text features are in different feature spaces, direct concatenation does not work well, so cross-modal feature attention network (CMFAN) can be used to align and fuse the two. CMFAN mechanism can make visual features focus on relevant text rules, while making text rules adapt to the current visual scene. The multi-modal joint feature representation is: ; wherein, is the multi-modal joint feature representation; CMFAN is the cross-modal feature attention network; is the high-dimensional visual feature vector; is the context knowledge vector. The multi-modal joint feature representation highly condenses the information of "what happened on site" and "what rules require".

[0030] The multi-modal feature fusion and semantic perception method fuses video spatio-temporal features and text knowledge features deeply, generates multi-modal joint feature representation containing rich scene semantics, and solves the problem of insufficient recognition ability of single modal data for complex events.

[0031] Step 3, based on the high-dimensional semantic vector and the multi-modal joint feature representation, a structured instruction is generated, and the terminal is optimized based on the structured instruction. By deeply matching the real-time multi-modal joint feature representation with the pre-defined high-dimensional semantic vector, it is determined whether an abnormal event occurs.

[0032] Step 3.1, the similarity of the multi-modal joint feature representation and the plurality of high-dimensional semantic vectors is calculated, and based on the similarity threshold, the activated graph node and the potential abnormal event are determined. For example, the cosine similarity between the multi-modal joint feature representation and the high-dimensional semantic vector can be calculated. Specifically, the cosine similarity is calculated as follows: ; Wherein, ActivationLevel is the similarity; CosineSimilarity is the cosine similarity algorithm; is the multi-modal joint feature representation of the current video; is the high-dimensional semantic vector of the supervision standard. Then a similarity threshold is set When the calculated is greater than the similarity threshold, it is determined that the scene shown in the current video hits the standard, and the corresponding graph node is activated. For example, contains the visual semantics of "no one at the counter" and "idle equipment", which is closest to the semantic vector of the "off-duty personnel" node in the graph and has a similarity greater than the similarity threshold, thereby activating the graph node of the potential abnormal event "off-duty personnel".

[0033] Step 3.2, based on the neighbor node information of the activated graph node, the context knowledge is constructed, and based on the multi-modal joint feature representation and the context knowledge, the event determination result and the confidence are obtained. The neighbor nodes can include graph nodes of types such as associated disposal actions and responsible persons. For example, the activated graph subgraph centered on the activated graph node can be extracted from the structured knowledge graph to form the context knowledge for the current event. Then, the multi-modal joint feature representation and the context knowledge are combined into a complete input sequence The input is a multimodal large model (MM-LLM). Based on its powerful reasoning capabilities, the MM-LLM performs probabilistic reasoning on the input sequence and outputs the event determination result for event type Y and the confidence probability P of the event determination result. The execution probability P is: ; in, Representation based on multimodal joint feature representation and contextual knowledge The confidence probability of determining that event Y is an anomalous event.

[0034] Step 3.3: When the confidence probability is greater than a preset confidence threshold, the potential abnormal event is determined to have occurred, and a structured instruction is generated based on the handling and timeliness relationships in the structured knowledge graph. For example, if the confidence probability P of a specific abnormal event output by the model is greater than the preset confidence threshold... At that time, the abnormal event was finally determined to have occurred, and the subsequent handling process was triggered.

[0035] Step 4: If an abnormal event occurs, structured instructions are generated based on the abnormal event, and closed-loop optimization of the terminal is performed based on these structured instructions. The system can automatically generate SOP-compliant handling instructions based on the reasoning results, monitor the execution process, and optimize the system based on feedback results, forming a closed loop.

[0036] Step 4.1, based on abnormal events By associating abnormal events with relevant relationships, structured instructions are generated. Abnormal event relationships refer to the attribute information associated with abnormal events within a structured knowledge graph. For example, abnormal event relationships can include handling relationships and time-sensitive relationships; that is, they include the actions corresponding to the relationship edges. and timeliness requirements And through corresponding actions and timeliness requirements The entities pointed to by equilateral lines are combined with anomalous events to generate structured instructions. For example, the function calling capability of MM-LLM can be used to transform unstructured disposal descriptions in the structured knowledge graph into structured instruction code that can be executed by the scheduling system. The model automatically fills in the parameters in the instructions based on the graph information to generate structured instructions. For example, the structured instruction Command can be an action chain, with a standard format of: FunctionCall(Action="Dispatch Support",Target="Security Check Area A",Resource="2 Staff",Priority="High",Deadline="5 Minutes").

[0037] Step 4.2, distribute and execute the structured instructions, and conduct time monitoring to obtain the optimization time. The generated structured instructions are sent to the integrated dispatching system of the airport through an API interface for physical execution (such as sending notifications to handheld terminals, broadcast systems, etc.). At the same time, a timer for the event is started inside the system to monitor whether the time exceeds the optimization time threshold (e.g., 5 minutes). The structured instructions flow to the dispatching system and the field staff.

[0038] Step 4.3, obtain the execution feedback text, and based on the execution feedback text and the optimized high-dimensional visual feature vector, verify the effectiveness of the structured instructions. The execution feedback text fed back by the dispatching system or the staff is re-input as text, combined with the current latest monitoring video to generate an optimized high-dimensional visual feature vector, and then input into the MM-LLM for reasoning to verify the effectiveness of the structured instructions. For example, the execution feedback text can be: "Support personnel arrived at the security checkpoint at 14:05 and are currently queueing and guiding." When the abnormal event has been alleviated, the structured instructions are considered effective. For example, when the queue length L has significantly decreased, the structured instructions at this time are considered effective. Otherwise, the structured instructions at this time are considered ineffective.

[0039] Step 4.4, if the structured instructions are effective, continue to adjust based on the structured instructions until the abnormal event is eliminated. For example, if the verification shows that the executed structured instructions are effective, the system records the successful reasoning path this time and may fine-tune to increase the weight of related features.

[0040] Step 4.5, if the structured instructions are ineffective, adjust the confidence threshold and regenerate the structured instructions. For example, if the verification shows that it is ineffective or has timed out, the system will adjust the confidence threshold for the next time .

[0041] Step 4.6, if the abnormal event has not been eliminated when the optimization time is greater than the optimization time threshold, an alarm is given.

[0042] Through the above steps, a complete automated closed-loop management from "perceiving events" to "matching standards", "executing disposal", and "verifying results" is realized, ensuring the efficiency and traceability of the inspection work.

[0043] The disposal action automatic triggering process based on structured knowledge graph matching and large model reasoning proposed by the present application realizes closed-loop management from event recognition, standard determination to action triggering through calculation of the similarity of features and standards and large model probabilistic reasoning.

[0044] ​​The application further provides a smart video inspection system based on a multi-modal large model, comprising a graph processing module, a joint processing module, an anomaly judgment module and an execution module; the graph processing module is used for constructing a structured knowledge graph based on unstructured text in airport data, and obtaining a high-dimensional semantic vector based on nodes of the structured knowledge graph; the joint processing module is used for performing feature extraction on various data including monitoring videos, to generate multi-modal joint feature representation; the joint feature comprises space-time information and rule context; the anomaly judgment module is used for judging whether an abnormal event occurs in the terminal building based on the high-dimensional semantic vector and the multi-modal joint feature representation; and the execution module is used for generating a structured instruction based on the abnormal event if the abnormal event occurs, and performing closed-loop optimization on the terminal building based on the structured instruction.

[0045] The above is only the preferred embodiment of the present application and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A smart video inspection method based on a multimodal large model, characterized in that, include: Step 1: Based on the unstructured text in the airport data, construct a structured knowledge graph, and obtain a high-dimensional semantic vector based on the nodes of the structured knowledge graph; Step 2: Extract features from various data, including surveillance video, to generate a multimodal joint feature representation; the multimodal joint feature representation includes spatiotemporal information and rule context; Step 3: Based on high-dimensional semantic vectors and multimodal joint feature representation, determine whether an abnormal event has occurred in the terminal. Step 4: If an abnormal event occurs, generate structured instructions based on the abnormal event, and perform closed-loop optimization of the terminal based on the structured instructions.

2. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, Based on unstructured text from airport data, a structured knowledge graph is constructed, and high-dimensional semantic vectors are obtained based on the nodes of the structured knowledge graph, including: Step 1.1: Use natural language processing technology to parse unstructured text and extract core entity sets and relation sets; unstructured text includes SOP documents, inspection standard lists, and job descriptions. Step 1.2: Construct a structured knowledge graph based on the core entity set and relation set; the entities in the structured knowledge graph include events and actions, and the edges include standard relations and causal relations. Step 1.3: Semantically vectorize the standard nodes of the structured knowledge graph to obtain high-dimensional semantic vectors of the standard nodes; the high-dimensional semantic vectors include the semantic information and contextual information of the standard nodes.

3. The intelligent video inspection method based on a multimodal large model according to claim 2, characterized in that, The core entity set includes the inspection items, inspection standards, handling actions, responsible units, and time constraints; The set of relationships includes the trigger, the corresponding entity, the responsible party, and the time limit requirement.

4. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, Feature extraction is performed on various data, including surveillance video, to generate multimodal joint feature representations, including: Step 2.1: Extract spatiotemporal features from the surveillance video to obtain a high-dimensional visual feature vector; the high-dimensional visual feature vector is used to characterize microscopic target features and macroscopic group behavior features. Step 2.2: Based on the scene shown in the current video frame, retrieve relevant standard operating procedure text fragments, and generate a contextual knowledge vector based on the standard operating procedure text fragments; the contextual knowledge vector represents the semantic representation of the rules that should be followed in the current scene. Step 2.3: Align and fuse the high-dimensional visual feature vector and the contextual knowledge vector to obtain a multimodal joint feature representation.

5. The intelligent video inspection method based on a multimodal large model according to claim 4, characterized in that, Microscopic target features include the unique identifier of each pedestrian in the video footage, their 3D spatial coordinates in the terminal coordinate system, and their real-time movement speed; Macro-level group behavior characteristics include population density within a designated area, queue length in queuing areas, and spatial distribution of population gathering areas.

6. The intelligent video inspection method based on a multimodal large model according to claim 4, characterized in that, The calculation formula for multimodal joint feature representation is as follows: ; in, It represents multimodal joint features; CMFAN is a cross-modal feature attention network; It is a high-dimensional visual feature vector; This is a contextual knowledge vector.

7. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, Based on high-dimensional semantic vectors and multimodal joint feature representation, structured instructions are generated, and the terminal is optimized based on these structured instructions, including: Step 3.1: Calculate the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors, and determine the activated graph nodes and potential abnormal events based on the similarity threshold; Step 3.2: Construct contextual knowledge based on the neighbor node information of the activated graph node, and perform probabilistic reasoning based on multimodal joint feature representation and contextual knowledge to obtain the event judgment result and confidence probability; Step 3.3: When the confidence probability is greater than the preset confidence threshold, it is determined that the potential abnormal event has occurred.

8. The intelligent video inspection method based on a multimodal large model according to claim 7, characterized in that, The formula for calculating the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors is as follows: ; Where ActivationLevel represents the similarity score; CosineSimilarity represents the cosine similarity algorithm. This represents the multimodal joint features of the current video. This is a high-dimensional semantic vector for the inspection standards.

9. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, If an abnormal event occurs, structured instructions are generated based on the abnormal event, and closed-loop optimization of the terminal is performed based on the structured instructions, including: Step 4.1: Generate structured instructions based on abnormal events and their relationships; the relationships between abnormal events include handling relationships and time-limited relationships. Step 4.2: Distribute and execute structured instructions, and record the optimization time; Step 4.3: Obtain the execution feedback text, and verify the effectiveness of the structured instructions based on the execution feedback text and the optimized high-dimensional visual feature vector; Step 4.4: If the structured instructions are valid, continue adjusting based on the structured instructions until the abnormal event is eliminated; Step 4.5: If the structured instruction is invalid, adjust the confidence threshold and regenerate the structured instruction. Step 4.6: If the abnormal event has not been eliminated before the optimization time exceeds the optimization time threshold, an alarm will be triggered.

10. A smart video inspection system based on a multimodal large model, applying the smart video inspection method based on a multimodal large model as described in any one of claims 1-9, characterized in that, It includes a graph processing module, a joint processing module, an anomaly detection module, and an execution module; The graph processing module is used to construct a structured knowledge graph based on unstructured text in airport data, and to obtain high-dimensional semantic vectors based on the nodes of the structured knowledge graph. The joint processing module is used to extract features from various types of data, including surveillance video, and generate multimodal joint feature representations; the multimodal joint feature representations include spatiotemporal information and rule context; The anomaly detection module is used to determine whether an anomaly has occurred in the terminal based on high-dimensional semantic vectors and multimodal joint feature representation. The execution module is used to generate structured instructions based on abnormal events if such events occur, and to perform closed-loop optimization of the terminal based on these structured instructions.

Images