A smart video inspection method and system based on a multi-modal large model

Abstract

The application belongs to the field of airport management, and relates to a smart video inspection method and system based on a multi-modal large model, which comprises the following steps: 1, based on unstructured text in airport data, a structured knowledge graph is constructed, and a high-dimensional semantic vector is obtained based on nodes of the structured knowledge graph; 2, feature extraction is performed on various data including monitoring video to generate multi-modal joint feature representation; the multi-modal joint feature representation comprises space-time information and rule context; 3, based on the high-dimensional semantic vector and the multi-modal joint feature representation, it is judged whether an abnormal event occurs in the terminal building; 4, if an abnormal event occurs, a structured instruction is generated based on the abnormal event, and the terminal building is optimized in a closed loop based on the structured instruction; based on the existing video monitoring environment and data of the airport, an integrated intelligent system is provided for intelligent inspection of the terminal building, and automation, refinement and closed-loop management of the inspection are realized.

Description

Technical Field

[0001] This invention relates to the field of airport management, and specifically discloses a smart video inspection method and system based on a multimodal large model. Background Technology

[0002] Currently, video surveillance in airport terminals mainly focuses on the following aspects: High reliance on manual labor and low efficiency: Inspection work is mainly divided into 23 time slots for manual inspection. The coverage and timeliness are limited by human resources, making it prone to fatigue and omissions. Separation of area and process supervision: Terminal video surveillance is mainly divided into two parts: area supervision and process node supervision. Some areas adopt a passive waiting mode for event reporting, making it difficult to proactively discover and intervene in potential events. Simple and crude triggering mechanisms: Existing intelligent analysis mostly uses crowd density as the main trigger point, requiring manual inspection of high-density areas, lacking fine-grained identification and correlation analysis of complex service and security events. Existing smart inspection based on video analysis has the following shortcomings: Incomplete event coverage: Existing video analysis technology can only analyze some common events, such as key node passenger flow statistics and intrusion into restricted areas. For complex, multi-step, and non-standardized service and management events (e.g., unattended service desks, passenger requests for help, facility damage, etc.), its coverage and recognition accuracy are seriously insufficient. Remaining at the perception level and lacking closed-loop management: From the perspective of airport terminal service and management, existing analysis for process and area supervision only stays at the perception level (event identification and early warning), failing to achieve full-process, automated analysis and triggering from supervision standards to handling actions, making it difficult to form an efficient management closed loop. Limited data modality and insufficient information utilization: Traditional video analysis mainly relies on image / video data, failing to effectively utilize and integrate existing multimodal data within the airport, such as text information (e.g., regulations, standard operating procedures SOPs) and audio information (e.g., broadcasts, calls), resulting in limited understanding and judgment capabilities for complex scenarios.

[0003] In view of this, the present invention proposes a smart video inspection method and system based on a multimodal large model. Based on the existing video surveillance environment and data foundation of the airport, it provides basic data fusion, advanced semantic analysis, real-time early warning and decision support for smart inspection of the terminal building, and constructs an integrated intelligent system of "perception-cognition-decision-action" to realize automated, refined and closed-loop management of inspection. Summary of the Invention

[0004] The purpose of this invention is to provide a smart video inspection method and system based on a multimodal large model, addressing the problem of how to use video analysis combined with a multimodal large model to assist airports in inspecting, supervising, and formulating response measures for various events in the terminal, thereby improving passenger service quality, reducing safety incidents, increasing inspection efficiency, and enhancing refined management. The specific solution is as follows:

[0005] A smart video inspection method based on a multimodal large model includes: Step 1, constructing a structured knowledge graph based on unstructured text in airport data, and obtaining high-dimensional semantic vectors based on the nodes of the structured knowledge graph; Step 2, extracting features from various data, including surveillance videos, to generate multimodal joint feature representations; the multimodal joint feature representations include spatiotemporal information and rule context; Step 3, determining whether an abnormal event has occurred in the terminal based on the high-dimensional semantic vectors and the multimodal joint feature representations; Step 4, if an abnormal event has occurred, generating structured instructions based on the abnormal event, and performing closed-loop optimization of the terminal based on the structured instructions.

[0006] Furthermore, based on the unstructured text in the airport data, a structured knowledge graph is constructed, and high-dimensional semantic vectors are obtained based on the nodes of the structured knowledge graph, including: Step 1.1, using natural language processing technology to parse the unstructured text and extract the core entity set and relation set; the unstructured text includes SOP documents, inspection standard lists, and job descriptions; Step 1.2, constructing a structured knowledge graph based on the core entity set and relation set; the entities in the structured knowledge graph include events and actions, and the edges include standard relations and causal relations; Step 1.3, semantically vectorizing the standard nodes of the structured knowledge graph to obtain high-dimensional semantic vectors of the standard nodes; the high-dimensional semantic vectors include the semantic information and contextual information of the standard nodes.

[0007] Furthermore, the core entity set includes the inspection project, inspection standards, handling actions, responsible units, and time constraints; the relationship set includes the trigger, the corresponding entity, the responsible subject, and time requirements.

[0008] Furthermore, feature extraction is performed on various data, including surveillance videos, to generate a multimodal joint feature representation, including: Step 2.1, extracting spatiotemporal features from the surveillance video to obtain a high-dimensional visual feature vector; the high-dimensional visual feature vector is used to represent micro-target features and macro-group behavior features; Step 2.2, retrieving relevant standard operating procedure text fragments based on the scene shown in the current video frame, and generating a contextual knowledge vector based on the standard operating procedure text fragments; the contextual knowledge vector represents the semantic representation of the rules that should be followed in the current scene; Step 2.3, aligning and fusing the high-dimensional visual feature vector and the contextual knowledge vector to obtain a multimodal joint feature representation.

[0009] Furthermore, micro-target features include the unique identifier of each pedestrian in the video footage, their 3D spatial coordinates in the terminal coordinate system, and their real-time movement speed; macro-group behavior features include the crowd density in the designated area, the queue length in the queuing area, and the spatial distribution of the crowd gathering area.

[0010] Furthermore, the calculation formula for multimodal joint feature representation is as follows:

[0011] ;

[0012] in, It represents multimodal joint features; CMFAN is a cross-modal feature attention network; It is a high-dimensional visual feature vector; This is a contextual knowledge vector.

[0013] Furthermore, based on high-dimensional semantic vectors and multimodal joint feature representations, structured instructions are generated, and the terminal is optimized based on these structured instructions, including: Step 3.1, calculating the similarity between the multimodal joint feature representations and multiple high-dimensional semantic vectors, and determining activated graph nodes and potential abnormal events based on similarity thresholds; Step 3.2, constructing contextual knowledge based on the neighbor node information of activated graph nodes, and performing probabilistic reasoning based on multimodal joint feature representations and contextual knowledge to obtain event judgment results and confidence probabilities; Step 3.3, determining that the potential abnormal event has occurred when the confidence probability is greater than a preset confidence threshold.

[0014] Furthermore, the formula for calculating the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors is as follows:

[0015] ;

[0016] Where ActivationLevel represents the similarity score; CosineSimilarity represents the cosine similarity algorithm. This represents the multimodal joint features of the current video. This is a high-dimensional semantic vector for the inspection standards.

[0017] Furthermore, if an abnormal event occurs, structured instructions are generated based on the abnormal event, and closed-loop optimization of the terminal is performed based on the structured instructions, including: Step 4.1, generating structured instructions based on the abnormal event and its correlation; the correlation includes handling relationship and timeliness relationship; Step 4.2, distributing and executing the structured instructions, and recording the optimization time; Step 4.3, obtaining the execution feedback text, and verifying the effectiveness of the structured instructions based on the execution feedback text and the optimized high-dimensional visual feature vector; Step 4.4, if the structured instructions are effective, continuous adjustments are made based on the structured instructions until the abnormal event is eliminated; Step 4.5, if the structured instructions are invalid, the confidence threshold is adjusted, and the structured instructions are regenerated; Step 4.6, if the abnormal event is not eliminated before the optimization time exceeds the optimization time threshold, an alarm is triggered.

[0018] This invention also provides a multimodal large-scale intelligent video inspection system applied to the aforementioned intelligent video inspection method based on a multimodal large-scale model. The system includes a graph processing module, a joint processing module, an anomaly detection module, and an execution module. The graph processing module constructs a structured knowledge graph based on unstructured text in airport data and obtains high-dimensional semantic vectors based on the nodes of the structured knowledge graph. The joint processing module extracts features from various data, including surveillance video, to generate multimodal joint feature representations. These multimodal joint feature representations include spatiotemporal information and rule context. The anomaly detection module determines whether an abnormal event has occurred in the terminal building based on the high-dimensional semantic vectors and the multimodal joint feature representations. The execution module generates structured instructions based on the abnormal event if one occurs and performs closed-loop optimization of the terminal building based on these instructions.

[0019] The present invention has the following advantages and beneficial effects:

[0020] This invention introduces a multimodal large model and knowledge graph to deeply integrate video visual information with textual knowledge such as airport management SOPs and regulations, solving the problem of insufficient ability of traditional video analysis to identify complex and semantic events, and achieving comprehensive coverage of service and management events.

[0021] This invention, through the construction of a knowledge graph of institutionalized inspection standards and a similarity calculation method, enables the analysis results to go beyond simple perception and early warning, and directly link to the airport's inspection standards and SOPs, automatically generating and triggering full-process handling actions, which greatly improves the airport's refined management and emergency response efficiency.

[0022] This invention utilizes a multimodal feature fusion method to unify heterogeneous data (video, text) in a high-dimensional space, enabling large models to perform high-order semantic reasoning. It improves the automation and intelligence of inspections: transforming passive waiting into proactive, real-time, and comprehensive intelligent inspections, significantly reducing the intensity and frequency of manual inspections, lowering labor costs, and simultaneously enhancing the ability to prevent safety incidents and the quality of passenger service. Attached Figure Description

[0023] Figure 1 An exemplary flowchart of a smart video inspection method based on a multimodal large model provided by the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. The components of the embodiments of the present invention described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] The following section provides a detailed and logically coherent application description of this invention, based on collected airport terminal video data, standard operating procedure (SOP) texts, and relevant inspection standards. To clearly demonstrate how this invention transforms static rules into dynamic intelligent decision-making, such as... Figure 1 As shown, the entire implementation process is divided into four main steps, each of which includes the following:

[0026] Step 1: Construct a structured knowledge graph. The structured knowledge graph is based on the "event-standard-action" framework. This method aims to transform high-precision, process-oriented management requirements, SOPs, and other unstructured text from terminal process node supervision and regional supervision into a structured knowledge graph (SKG) that can be queried and reasoned about in real time by large models, and to establish the relationships between events. This includes the following steps:

[0027] The core task of Step 1 is to transform the existing unstructured natural language text in airport data into a structured knowledge graph (SKG) that can be read, queried, and reasoned about by machine algorithms through digital means.

[0028] Step 1.1: Utilize Natural Language Processing (NLP) technology to parse unstructured text and extract the core entity set and relation set. First, receive airport data provided by the airport management department and perform deep analysis on the unstructured text within the airport data using NLP technology. The unstructured text mainly includes SOP documents, inspection standard lists, and job descriptions. By scanning keywords in the text, identify nouns and verb phrases with specific management significance, and map them to predefined entity types and relation types to obtain the core entity set and relation set.

[0029] For the core entity set, the core entities and relationships in the inspection standards can be extracted. From the inspection content provided by the airport, entity E and relationship R can be precisely extracted. Entity E may include inspection items. Inspection standards , handling actions Responsible Unit and time constraints The inspection items include check-in counters and trash cans; the inspection standards include whether queues are congested and whether trash is overflowing; the action can be to notify the airline to open more counters within 5 minutes; the responsible unit can include the airline; the time limit can be 5 minutes, etc. Identifying entity E can construct a core entity set. : For example, extracting specific objects for inspection from a document. Extract the criteria for judging abnormal events. Extract the stipulated rectification time. .

[0030] For a set of relations, relation R can include those triggered by ,correspond Responsible entities and timeliness requirements Etc. The trigger can include event triggering criteria; the corresponding action can include the action corresponding to the criteria; the responsible party can include the party performing the action; the time requirement can include the time the action needs to be completed. Extract relation R and construct a relation set. : For example, extracting descriptions of the cause and effect of events. This indicates that a certain phenomenon will cause a certain standard to be activated; extract the description of the attribution of rights and responsibilities. Clearly define who is responsible for resolving the problem.

[0031] Step 1.2: Based on the core entity set and relation set, and according to the graph construction rules, link the discrete entities together through relations to batch construct structured knowledge triples with attributes, in the following format: These triples together form a structured knowledge graph. Entities in a structured knowledge graph include events and actions, while edges include standard relations and causal relationships. (Structured knowledge graph) It can be: For example, triples in a structured knowledge graph can include event definition triples, action association triples, and time constraint triples. An event definition triple could be: (Check-in counter, triggered by, queue congestion and exceeding the isolation zone), which defines what constitutes an anomaly at the check-in counter. An action association triple could be: (Queue congestion and exceeding the isolation zone, corresponding to, opening additional counters to alleviate queues), which defines the standard action to be taken after an anomaly. A time constraint triple could be: (Open additional counters to alleviate queues, time requirement, 5 minutes), which defines the time window within which the action must be completed, serving as the basis for subsequent automatic follow-up.

[0032] Step 1.3 involves semantically vectorizing the nodes of the structured knowledge graph to obtain high-dimensional semantic vectors for each node. Since computers cannot directly understand the meaning of characters like "queue" or "congestion," pre-trained deep language models (such as BERT) can be used to semantically vectorize each standard node in the constructed knowledge graph. Embedding encoding is performed to transform it into a numerical vector in a high-dimensional space. A high-dimensional semantic vector is then generated for each standard node. This parameter not only contains semantic information of the text (e.g., literal meaning) but also implicit contextual information (e.g., information about other nodes connected to the standard node). It will be directly used as the comparison benchmark in step 3.1 for similarity matching with real-time video features.

[0033] The structured knowledge graph construction and vectorization method proposed in this invention, based on "event-standard-handling", transforms the unstructured standard operating procedure (SOP) documents of the terminal into machine-understandable structured knowledge and semantic vectors, providing a standard basis for intelligent reasoning and realizing the accurate and dynamic correlation between inspection standards and video events.

[0034] Step 2 involves feature extraction from various data sources, including surveillance video, to generate a multimodal joint feature representation. By processing massive amounts of surveillance video streams in real time and combining this with the SOP text context, a multimodal joint feature representation containing rich spatiotemporal and semantic information is generated. This multimodal joint feature representation includes spatiotemporal information and rule context.

[0035] Step 2.1: Employ advanced computer vision models (such as Transformer-based Vision Model or 3D-CNN) to extract spatiotemporal feature information from real-time monitoring video frame sequences from cameras in various areas of the terminal, thereby extracting high-dimensional visual feature vectors. The video spatiotemporal feature extraction step not only detects objects in a single frame image but also analyzes temporal changes between multiple frames. High-dimensional visual feature vectors. It can be used to characterize microscopic target features and macroscopic group behavior features. For example, microscopic target features may include the unique identifier ID of each pedestrian in the video frame, their 3D spatial coordinates (x, y, z) in the terminal coordinate system, and their real-time movement speed v; macroscopic group behavior features may include the crowd density ρ in a specified area, the queue length L in key queuing areas, and the spatial distribution of crowd gathering areas. Features such as...

[0036] Step 2.2: Based on the scene shown in the current video frame, retrieve relevant standard operating procedure (SOP) text fragments, and generate a contextual knowledge vector based on these fragments. For example, based on the physical location of the scene shown in the current video obtained from monitoring, retrieve relevant SOP text fragments from the document library. The physical location of the scene can include security checkpoints and check-in islands, etc. The retrieved standard operating procedure text fragments will be used. The input is encoded into a pre-trained knowledge-enhanced large language model to obtain a contextual knowledge vector. Knowledge-enhanced large language models can include ERNIE or specially fine-tuned LLaMA models. Contextual knowledge vectors represent the semantic representation of the rules that should be followed in the current scenario.

[0037] Step 2.3 involves aligning the high-dimensional visual feature vector and the contextual knowledge vector, followed by cross-modal feature fusion to obtain a multimodal joint feature representation. Since video features and text features reside in different feature spaces, direct concatenation yields poor results. Therefore, a Cross-Modal Feature Attention Network (CMFAN) can be used to align and fuse them. The CMFAN mechanism allows visual features to focus on relevant text rules while simultaneously adapting the text rules to the current visual scene. The multimodal joint feature representation is as follows:

[0038] ;

[0039] in, It represents multimodal joint features; CMFAN is a cross-modal feature attention network; It is a high-dimensional visual feature vector; This is a contextual knowledge vector. This multimodal joint feature representation highly condenses information about "what happened on-site" and "what the rules require".

[0040] The multimodal feature fusion and semantic perception method proposed in this invention deeply integrates video spatiotemporal features with textual knowledge features to generate multimodal joint feature representations containing rich scene semantics, thus solving the problem of insufficient ability of single-modal data to recognize complex events.

[0041] Step 3: Based on high-dimensional semantic vectors and multimodal joint feature representations, structured instructions are generated, and the terminal is optimized based on these structured instructions. By performing deep matching between real-time multimodal joint feature representations and predefined high-dimensional semantic vectors, the occurrence of abnormal events can be accurately determined.

[0042] Step 3.1: Calculate the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors, and determine the activated graph nodes and potential anomalous events based on the similarity threshold. For example, the cosine similarity between the multimodal joint feature representation and the high-dimensional semantic vectors can be calculated. Specifically:

[0043] ;

[0044] Where ActivationLevel represents the similarity score; CosineSimilarity represents the cosine similarity algorithm. This represents the multimodal joint features of the current video. This is a high-dimensional semantic vector representing the inspection standards. Then, a similarity threshold is set. When calculated At that time, it is determined that the scene shown in the current video matches the standard, and the corresponding graph node. Activated. For example, It contains the visual semantics of "unmanned counter" and "equipment idle". It is closest to the semantic vector of the "personnel not at their post" node in the graph and the similarity exceeds the similarity threshold, thus activating the graph node of the potential abnormal event "personnel not at their post".

[0045] Step 3.2 involves constructing contextual knowledge based on the neighbor node information of activated graph nodes, and performing probabilistic reasoning based on multimodal joint feature representation and contextual knowledge to obtain the event judgment result and confidence level. Neighbor nodes can include graph nodes of types such as associated actions and responsible persons. For example, this can be achieved from a structured knowledge graph. Extract the activation graph subgraph centered on the activation graph node to form contextual knowledge for the current event. Subsequently, the multimodal joint feature representation will be performed. and contextual knowledge Combined into a complete input sequence The input is a multimodal large model (MM-LLM). Based on its powerful reasoning capabilities, the MM-LLM performs probabilistic reasoning on the input sequence and outputs the event determination result for event type Y and the confidence probability P of the event determination result. The execution probability P is:

[0046] ;

[0047] in, Representation based on multimodal joint feature representation and contextual knowledge The confidence probability of determining that event Y is an anomalous event.

[0048] Step 3.3: When the confidence probability is greater than a preset confidence threshold, the potential abnormal event is determined to have occurred, and a structured instruction is generated based on the handling and timeliness relationships in the structured knowledge graph. For example, if the confidence probability P of a specific abnormal event output by the model is greater than the preset confidence threshold... At that time, the abnormal event was finally determined to have occurred, and the subsequent handling process was triggered.

[0049] Step 4: If an abnormal event occurs, structured instructions are generated based on the abnormal event, and closed-loop optimization of the terminal is performed based on these structured instructions. The system can automatically generate SOP-compliant handling instructions based on the reasoning results, monitor the execution process, and optimize the system based on feedback results, forming a closed loop.

[0050] Step 4.1, based on abnormal events By associating abnormal events with relevant relationships, structured instructions are generated. Abnormal event relationships refer to the attribute information associated with abnormal events within a structured knowledge graph. For example, abnormal event relationships can include handling relationships and time-sensitive relationships; that is, they include the actions corresponding to the relationship edges. and timeliness requirements And through corresponding actions and timeliness requirements The entities pointed to by equilateral lines are combined with anomalous events to generate structured instructions. For example, the function calling capability of MM-LLM can be used to transform unstructured disposal descriptions in the structured knowledge graph into structured instruction code that can be executed by the scheduling system. The model automatically fills in the parameters in the instructions based on the graph information to generate structured instructions. For example, the structured instruction Command can be an action chain, with a standard format of: FunctionCall(Action="Dispatch Support",Target="Security Check Area A",Resource="2 Staff",Priority="High",Deadline="5 Minutes").

[0051] Step 4.2: Distribute and execute structured instructions, and monitor their timeliness to obtain the optimized time. The generated structured instructions are sent to the airport's integrated scheduling system via API for physical execution (e.g., sending notifications to handheld terminals, broadcast systems, etc.). Simultaneously, a timer is started internally to monitor whether the time exceeds the optimized time threshold. (e.g., 5 minutes). Structured instructions flow to the scheduling system and on-site personnel.

[0052] Step 4.3: Obtain the execution feedback text and, based on the execution feedback text and optimized high-dimensional visual feature vectors, verify the effectiveness of the structured instructions. This includes the execution feedback text provided by the scheduling system or staff. The input is re-entered as text, combined with the latest surveillance video, to generate an optimized high-dimensional visual feature vector. This vector is then input again into MM-LLM for inference, validating the structured instructions' effectiveness. For example, the execution feedback text could be: "Support personnel arrived at the security checkpoint at 14:05 and are currently managing the queue." Structured instructions are considered effective when the abnormal event has subsided. For example, if the queue length L has significantly decreased, the structured instructions are considered effective. Otherwise, they are considered invalid.

[0053] Step 4.4: If the structured instructions are valid, the system continues to adjust based on them until the anomalous event is eliminated. For example, if the verification structure shows that the executed structured instructions are valid, the system records the successful inference path and may fine-tune by increasing the weight of relevant features.

[0054] Step 4.5: If the structured instruction is invalid, adjust the confidence threshold and regenerate the structured instruction. For example, if the verification shows invalidity or timeout, the system will decrease the confidence threshold for the next iteration. .

[0055] Step 4.6: If the abnormal event has not been eliminated before the optimization time exceeds the optimization time threshold, an alarm will be triggered.

[0056] Through the above steps, a complete automated closed-loop management system is achieved, from "sensing events" to "matching standards," then to "execution and handling" and "verification of results," ensuring the efficiency and traceability of inspection work.

[0057] The automatic triggering process for handling actions proposed in this invention, based on structured knowledge graph matching and large model reasoning, achieves closed-loop management from event identification and standard determination to action triggering by calculating the similarity between features and standards and using large model probabilistic reasoning.

[0058] This invention also provides a smart video inspection system based on a multimodal large model, including a graph processing module, a joint processing module, an anomaly detection module, and an execution module. The graph processing module is used to construct a structured knowledge graph based on unstructured text in airport data, and obtain high-dimensional semantic vectors based on the nodes of the structured knowledge graph. The joint processing module is used to extract features from various data, including surveillance videos, and generate multimodal joint feature representations. The joint features include spatiotemporal information and rule context. The anomaly detection module is used to determine whether an abnormal event has occurred in the terminal based on the high-dimensional semantic vectors and multimodal joint feature representations. The execution module is used to generate structured instructions based on the abnormal event if an abnormal event occurs, and perform closed-loop optimization of the terminal based on the structured instructions.

[0059] The above are merely preferred embodiments of the present invention and are not intended to limit the present invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A smart video inspection method based on a multimodal large model, characterized in that, include: Step 1: Based on the unstructured text in the airport data, construct a structured knowledge graph, and obtain a high-dimensional semantic vector based on the nodes of the structured knowledge graph; Step 2: Extract features from various data, including surveillance video, to generate a multimodal joint feature representation; Multimodal joint feature representation includes spatiotemporal information and rule context; Feature extraction is performed on various data, including surveillance video, to generate multimodal joint feature representations, including: Step 2.1: Extract spatiotemporal features from the surveillance video to obtain a high-dimensional visual feature vector. The high-dimensional visual feature vector is used to characterize micro-target features and macro-group behavior features. Micro-target features include the unique identifier of each pedestrian in the video frame, their 3D spatial coordinates in the terminal coordinate system, and their real-time movement speed. Macro-group behavior features include the crowd density in the specified area, the queue length in the queuing area, and the spatial distribution of the crowd gathering area. Step 2.2: Based on the scene shown in the current video frame, retrieve relevant standard operating procedure text fragments, and generate a contextual knowledge vector based on the standard operating procedure text fragments; the contextual knowledge vector represents the semantic representation of the rules that should be followed in the current scene. Step 2.3: Align and fuse the high-dimensional visual feature vector and the contextual knowledge vector to obtain a multimodal joint feature representation; the calculation formula for the multimodal joint feature representation is: ; in, It represents multimodal joint features; CMFAN is a cross-modal feature attention network; It is a high-dimensional visual feature vector; For contextual knowledge vectors; Step 3: Based on high-dimensional semantic vectors and multimodal joint feature representation, determine whether an abnormal event has occurred in the terminal, including: Step 3.1: Calculate the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors, and determine the activated graph nodes and potential abnormal events based on the similarity threshold; the formula for calculating the similarity between the multimodal joint feature representation and multiple high-dimensional semantic vectors is as follows: ; Where ActivationLevel represents the similarity score; CosineSimilarity represents the cosine similarity algorithm. This represents the multimodal joint features of the current video. A high-dimensional semantic vector for the inspection standards; Step 3.2: Construct contextual knowledge based on the neighbor node information of the activated graph node, and perform probabilistic reasoning based on multimodal joint feature representation and contextual knowledge to obtain the event judgment result and confidence probability; Step 3.3: When the confidence probability is greater than the preset confidence threshold, it is determined that the potential abnormal event has occurred; Step 4: If an abnormal event occurs, generate structured instructions based on the abnormal event, and perform closed-loop optimization of the terminal based on the structured instructions.

2. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, Based on unstructured text from airport data, a structured knowledge graph is constructed, and high-dimensional semantic vectors are obtained based on the nodes of the structured knowledge graph, including: Step 1.1: Use natural language processing technology to parse unstructured text and extract core entity sets and relation sets; unstructured text includes SOP documents, inspection standard lists, and job descriptions. Step 1.2: Construct a structured knowledge graph based on the core entity set and relation set; the entities in the structured knowledge graph include events and actions, and the edges include standard relations and causal relations. Step 1.3: Semantically vectorize the standard nodes of the structured knowledge graph to obtain high-dimensional semantic vectors of the standard nodes; the high-dimensional semantic vectors include the semantic information and contextual information of the standard nodes.

3. The intelligent video inspection method based on a multimodal large model according to claim 2, characterized in that, The core entity set includes the inspection items, inspection standards, handling actions, responsible units, and time constraints; The set of relationships includes the trigger, the corresponding entity, the responsible party, and the time limit requirement.

4. The intelligent video inspection method based on a multimodal large model according to claim 1, characterized in that, If an abnormal event occurs, structured instructions are generated based on the abnormal event, and closed-loop optimization of the terminal is performed based on the structured instructions, including: Step 4.1: Generate structured instructions based on abnormal events and their relationships; the relationships between abnormal events include handling relationships and time-limited relationships. Step 4.2: Distribute and execute structured instructions, and record the optimization time; Step 4.3: Obtain the execution feedback text, and verify the effectiveness of the structured instructions based on the execution feedback text and the optimized high-dimensional visual feature vector; Step 4.4: If the structured instructions are valid, continue adjusting based on the structured instructions until the abnormal event is eliminated; Step 4.5: If the structured instruction is invalid, adjust the confidence threshold and regenerate the structured instruction. Step 4.6: If the abnormal event has not been eliminated before the optimization time exceeds the optimization time threshold, an alarm will be triggered.

5. A smart video inspection system based on a multimodal large model, applying the smart video inspection method based on a multimodal large model as described in any one of claims 1-4, characterized in that, It includes a graph processing module, a joint processing module, an anomaly detection module, and an execution module; The graph processing module is used to construct a structured knowledge graph based on unstructured text in airport data, and to obtain high-dimensional semantic vectors based on the nodes of the structured knowledge graph. The joint processing module is used to extract features from various types of data, including surveillance video, and generate multimodal joint feature representations; the multimodal joint feature representations include spatiotemporal information and rule context; The anomaly detection module is used to determine whether an anomaly has occurred in the terminal based on high-dimensional semantic vectors and multimodal joint feature representation. The execution module is used to generate structured instructions based on abnormal events if such events occur, and to perform closed-loop optimization of the terminal based on these structured instructions.

Images