Safety inspection method and system based on multi-model cooperation

Abstract

The application discloses a kind of based on the safety inspection method and system of multi-model cooperation, which comprises: the multi-dimensional detection data of target area is preprocessed, and detection image and accompanying metadata are obtained;Detection image and accompanying metadata are input to lightweight target detection model, and target area image and structured metadata are obtained;Detection image and accompanying metadata, target area image and structured metadata are respectively input to multi-modal large model, and global semantic information and local semantic information are obtained;Global semantic information is searched in external professional knowledge base, vector matching and cross-modal adapter retrieval are carried out, and safety knowledge item is obtained;Local semantic information, global semantic information and safety knowledge item are input into multi-modal large model, and safety hazard area, safety hazard content, improvement suggestion and safety regulation are obtained.The application has high positioning accuracy, high output reliability, and has professional field knowledge guarantee.

Description

Technical Field

[0001] This invention relates to the field of safety production data processing and monitoring technology, and in particular to a safety inspection method and system based on multi-model collaboration. Background Technology

[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.

[0003] With the rapid development of computer vision and artificial intelligence technologies, automated security inspection technologies based on images or videos have become a research hotspot in fields such as industrial safety, smart construction sites, and public security. Traditional security inspection methods mainly rely on manual inspections, which are inefficient, costly, and prone to missed inspections and misjudgments due to the subjective experience and fatigue of the inspectors.

[0004] To overcome the drawbacks of manual inspection, a series of automated solutions based on deep learning have emerged in existing technologies, such as large models. However, large models suffer from problems such as insufficient positioning accuracy, uncontrollable output, and lack of professional domain knowledge.

[0005] Therefore, there is currently no solution to the above problems. Summary of the Invention

[0006] This invention provides a security inspection method based on multi-model collaboration, which has high positioning accuracy, high output reliability, and is backed by professional domain knowledge, including:

[0007] The multidimensional detection data of the target area is preprocessed to obtain the detection image and accompanying metadata, the accompanying metadata including scene labels;

[0008] The detected image and accompanying metadata are input into a lightweight object detection model to obtain a target region image and structured metadata. The lightweight object detection model is used to perform object detection on the detected image according to the confidence threshold corresponding to the scene label to obtain object detection data. The detected target is then cropped according to the object detection data to obtain the target region image.

[0009] The detected image and its accompanying metadata, the target region image and the structured metadata are respectively input into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model.

[0010] Global semantic information is retrieved by keyword retrieval, vector matching, and cross-modal adapter retrieval in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios.

[0011] By inputting local semantic information, global semantic information, and safety knowledge items into the multimodal large model, we can obtain areas of safety hazards, content of safety hazards, improvement suggestions, and safety regulations.

[0012] This invention provides a security inspection system based on multi-model collaboration, which features high positioning accuracy, high output reliability, and is backed by expertise in the relevant domain. The system includes:

[0013] The preprocessing module is used to preprocess the multidimensional detection data of the target area to obtain the detection image and accompanying metadata, wherein the accompanying metadata includes scene labels;

[0014] The small model detection module is used to input the detection image and accompanying metadata into the lightweight object detection model to obtain the target region image and structured metadata; the lightweight object detection model is used to perform object detection on the detection image according to the confidence threshold corresponding to the scene label to obtain object detection data, and to crop the detected target according to the object detection data to obtain the target region image;

[0015] The large model analysis module is used to input the detection image and its accompanying metadata, the target region image and its structured metadata into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model.

[0016] The retrieval and matching module is used to perform keyword retrieval, vector matching, and cross-modal adapter retrieval of global semantic information in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios.

[0017] The comprehensive analysis module is used to input local semantic information, global semantic information, and security knowledge items into the multimodal large model to obtain security hazard areas, security hazard content, improvement suggestions, and security regulations.

[0018] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the above-described security inspection method based on multi-model collaboration.

[0019] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned security inspection method based on multi-model collaboration.

[0020] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned security inspection method based on multi-model collaboration.

[0021] In this embodiment of the invention, a lightweight target detection model is used for multi-scale feature extraction and dynamic confidence screening. Combined with scene labels to adaptively adjust the detection threshold, potential hazard areas are accurately located and normalized coordinates are output. Then, source information from structured metadata is used to back-map the original image, ensuring pixel-level positioning accuracy of the hazard areas. Compared to the knowledge fixation problem of traditional single-target detection models and the fuzzy positioning defects of large models, this method achieves an organic unity of rapid scanning and precise positioning, significantly reducing positioning deviations caused by missed detections and false detections. Reliability is improved through collaborative verification of large and small models and cross-validation of multi-source information: the lightweight model provides precise region positioning, the multimodal large model deeply analyzes local and global semantics, and external professional knowledge bases provide authoritative evidence. Finally, the results are generated by integrating the multimodal large model, effectively solving the problem of mismatch between hazard areas and content. Simultaneously, the cross-modal adapter eliminates the gap between colloquial semantic descriptions and rigorous safety regulations, ensuring that the output content is highly consistent with the actual scene and professional standards, avoiding the hallucination-like output that may occur from single-model reasoning, and improving the credibility of the results. An external professional knowledge base integrates historical cases and safety regulations from different scenarios. Through a hybrid retrieval mechanism combining keyword search and vector matching, it accurately matches each hazard with multi-level safety clauses and similar cases. The final improvement suggestions are generated based on historical case rectification experience and safety regulation requirements. The safety regulations provide clear basis, solving the pain point of traditional detection methods that only identify hazards without professional support. It provides managers with full-process decision support from hazard identification to basis to rectification plan, improving the professionalism and implementation efficiency of safety inspections. The lightweight target detection model is adapted to edge device deployment and can process multi-dimensional detection data (images, video streams, drone aerial data) in real time and quickly, meeting the high-efficiency detection needs of dynamic and dense scenarios. The multi-modal large model realizes deep semantic understanding and global risk assessment. Combined with a scenario label weight adjustment mechanism, it adapts to the safety inspection needs of different scenarios (such as smart construction sites and factory workshops). The collaboration of the two ensures detection efficiency and realizes the leap from single target identification to scenario-based hazard diagnosis, with broad application scenario adaptability. Attached Figure Description

[0022] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:

[0023] Figure 1 This is a flowchart of the security inspection method based on multi-model collaboration in an embodiment of the present invention;

[0024] Figure 2 This is a schematic diagram of the lightweight target detection model in an embodiment of the present invention;

[0025] Figure 3 This is a schematic diagram of the structure of the multimodal large model in an embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of a security inspection system based on multi-model collaboration in an embodiment of the present invention;

[0027] Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation

[0028] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.

[0029] The applicant argues that the core of the single-object detection model-based approach is to directly analyze the input image using a highly efficient object detection model trained on a large amount of data, locating potential safety hazards such as people not wearing helmets, improperly installed guardrails, and fireworks. Single-object detection models often employ a single deep learning model (a small, lightweight model). While lightweight, these models struggle to handle complex semantics and cross-scene detection requirements. Single-object detection models rely on large amounts of data, resulting in high success rates for specific object recognition. However, they lack versatility, requiring a new model to be trained for each new object, and suffer from a lack of semantic understanding, poor interpretability, and knowledge rigidity. Essentially, a single-object detection model is a visual pattern recognition tool that can only identify the presence and location of predefined categories of objects, but cannot understand the contextual relationships between objects and their surroundings, or between objects themselves, making it difficult to solve complex semantic understanding scenarios. Furthermore, single-object detection models only output a bounding box and a confidence score, providing only the scene location coordinates of a specific object. Safety hazard rules need to be manually constructed, as the target itself cannot explain the hazard. Various rule bases need to be manually built to adapt to the detected targets. The detection capability of a single-target detection model depends entirely on the training data. When new types of hazards or complex combinations of hazards appear that have not appeared in the training set, the model will completely fail, lacking the ability to acquire new knowledge from external sources.

[0030] With the emergence of multimodal large models, solutions have emerged that directly leverage the powerful generalization and natural language understanding capabilities of large visual language models for image analysis. Users can provide the model with an image and a text command, and the model will directly output the analysis results described in text. Although large visual language models (referred to as large models) can describe hazard areas in natural language, their pixel-level or bounding box-level localization accuracy is usually far lower than that of dedicated object detection models, making it difficult to accurately indicate the location of hazards in complex scenes. The output of large models is random and uncontrollable, and may generate fictitious, hallucinatory safety hazard descriptions unrelated to the image content, which is unacceptable in practical industrial applications. The knowledge base of large models comes from general internet data and lacks in-depth integration with industry-specific professional safety regulations and standards, resulting in recommendations that may not be professional enough or lack authoritative support.

[0031] Large and small models exhibit a clear disconnect and conflict in terms of detection efficiency and accuracy, and semantic understanding and knowledge application. Small models are fast but not intelligent, while large models are intelligent but slow and uncontrollable. Neither can guarantee efficient and accurate localization while providing interpretable, evidence-based, and professionally in-depth semantic analysis reports. Therefore, this invention proposes an innovative technical solution that organically integrates efficient target detection, deep semantic understanding, and external professional knowledge bases to address the aforementioned technical challenges.

[0032] Figure 1 This is a flowchart of a security inspection method based on multi-model collaboration in an embodiment of the present invention. The method includes:

[0033] Step 101: Preprocess the multidimensional detection data of the target area to obtain the detection image and accompanying metadata, wherein the accompanying metadata includes scene labels;

[0034] Step 102: Input the detected image and accompanying metadata into the lightweight object detection model to obtain the target region image and structured metadata; the lightweight object detection model is used to perform object detection on the detected image according to the confidence threshold corresponding to the scene label to obtain object detection data, and to crop the detected target according to the object detection data to obtain the target region image;

[0035] Step 103: Input the detected image and its accompanying metadata, the target region image and the structured metadata into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model.

[0036] Step 104: Perform keyword retrieval, vector matching, and cross-modal adapter retrieval on the global semantic information in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios.

[0037] Step 105: Input local semantic information, global semantic information, and safety knowledge items into the multimodal large model to obtain the safety hazard area, safety hazard content, improvement suggestions, and safety regulations.

[0038] Each step is described in detail below.

[0039] In step 101, the multidimensional detection data of the target area is preprocessed to obtain the detection image and accompanying metadata, wherein the accompanying metadata includes scene labels;

[0040] In this embodiment of the invention, the multidimensional detection data includes images, video streams (real-time / offline), drone aerial photography data, etc. It can uniformly decode multidimensional detection data of different resolutions and formats. Through an adaptive frame extraction algorithm, the frame extraction frequency is dynamically adjusted according to the complexity of the video scene (minimum 1 FPS for static scenes, maximum 15 FPS for dynamic, dense scenes). Frames are extracted from the multidimensional detection data according to the frame extraction frequency to obtain the first processed data. The first processed data is then jointly processed by automatic white balance, histogram equalization, and dehazing and noise reduction to obtain the processed detection image, thereby improving image quality in low-light and strong interference environments. The accompanying metadata includes image ID, timestamp, and scene tag. The detection image format requirements include, but are not limited to, dynamic resolution adaptation, uint8 data type, and 3 channels (RGB). In this embodiment of the invention, the multidimensional detection data may also include, but is not limited to, image and text information such as temperature, geographical location, and humidity.

[0041] In step 102, the detected image and accompanying metadata are input into a lightweight object detection model to obtain a target region image and structured metadata. The lightweight object detection model is used to perform object detection on the detected image according to the confidence threshold corresponding to the scene label to obtain target detection data. The detected target is then cropped according to the target detection data to obtain the target region image.

[0042] Lightweight target detection models serve as the perception front end, undertaking the crucial task of rapid initial scanning. Based on a highly optimized neural network architecture, they can perform real-time image analysis with extremely low computational latency, quickly locating potential risk targets in the image, such as people not wearing safety helmets or incorrectly stacked materials, and accurately outputting their position coordinates in the image, providing a target list for subsequent detailed analysis.

[0043] Figure 2This is a schematic diagram of the structure of a lightweight target detection model in an embodiment of the present invention. In one embodiment, the lightweight target detection model includes a target detection module 201 and a target cropping module 202.

[0044] The target detection module 201 includes:

[0045] The input adaptation module 2011 is used to process the detected image into image tensor data and obtain the processing metadata;

[0046] The preprocessing enhancement module 2012 is used to denoise and enhance image tensor data through a conditional diffusion model to obtain enhanced tensor data.

[0047] The multi-scale feature extraction module 2013 is used to extract multi-scale features from augmented tensor data and obtain a fused feature map of multiple scales.

[0048] The original detection module 2014 is used to perform target detection based on the fused feature map of multiple scales to obtain the original detection result, which includes the bounding box, category and confidence of the detected target;

[0049] The post-processing module 2015 is used to filter the original detection results according to the confidence threshold corresponding to the scene label in the processing metadata, and to perform coordinate inverse mapping on the filtered results according to the scaling ratio and padding parameters in the processing metadata, so as to map the bounding box of the detected target to the normalized coordinates of the detection image to obtain the target detection data. The target detection data includes the bounding box, category, confidence and processing metadata of the detected target.

[0050] The target cropping module 202 is used to crop the detected target according to the target detection data to obtain the target region image and structured metadata, wherein the structured metadata includes source information.

[0051] In practice, the input adaptation module performs data format conversion and validity verification on the detected images to provide input data of a unified specification for subsequent modules. The input adaptation module can also check whether the resolution, number of channels, and data type of the detected images meet the requirements, discarding damaged or abnormally formatted images. Then, based on the model input specifications (default 640×640 pixels), it uses a bilinear interpolation algorithm to scale the image proportionally to ensure that the target features are not distorted. If the image aspect ratio differs significantly from the model input ratio, black borders are used to fill in the gaps and avoid stretching the target. Pixel values ​​are mapped from [0,255] to the [0,1] range to reduce the model's computational load. Simultaneously, a batch dimension (BatchSize is configurable, default 1 at the edge) is added to generate image tensor data. The scaling ratio and padding parameters are bound to the image tensor data as metadata. Generally, a standardized image tensor can be represented as (shape: [BatchSize, 640, 640, 3]).

[0052] The core function of the preprocessing enhancement module is to enhance the features of image tensor data. Below is an example of constructing a conditional diffusion model, building a noise prediction network based on the U-Net architecture, including: a downsampling encoder: extracting multi-scale features through convolutional layers and attention mechanisms; an intermediate bottleneck layer: processing deep feature representations; an upsampling decoder: progressively restoring spatial resolution; a conditional fusion module: integrating conditional vectors into the network through cross-attention or feature concatenation; and a temporal embedding module: encoding the diffusion step number t into a temporal embedding vector. The training objective of the conditional diffusion model is to train the network to predict the noise added to the image. The loss function uses mean squared error loss, comparing the difference between the predicted noise and the actual added noise. During training, for each batch, time step t is randomly sampled, the noisy image is calculated, and the network is trained to predict the noise. When applied, the conditional diffusion model iteratively denoises from t to 1, obtaining the enhanced image tensor.

[0053] The core function of the multi-scale feature extraction module is to quickly extract multi-scale features of images based on a lightweight neural network architecture, covering the feature information of small targets (such as small fire sources), medium targets (such as people not wearing safety helmets), and large targets (such as piles of materials), providing core feature support for target detection.

[0054] The following is an example structure of a multi-scale feature extraction module. Other structures of multi-scale feature extraction modules that can implement the method proposed in the embodiments of this invention are all within the protection scope of this invention, and will not be described in detail here.

[0055] In a specific example, the multi-scale feature extraction module includes a backbone network and a lightweight feature pyramid network.

[0056] Backbone network feature extraction: A lightweight target detection architecture is adopted (redundant convolutional layers are removed, and the core CSPDarknet structure is retained). The enhanced tensor data is reduced from 640×640 to 20×20 through 5 downsampling steps, generating feature maps of 3 scales (80×80, 40×40, 20×20), which correspond to small, medium and large target features, respectively.

[0057] Feature fusion: Cross-scale feature fusion is achieved by using a lightweight feature pyramid network (FPN) to fuse feature maps at three scales. The high-level semantic features (20×20) and low-level detail features (80×80, 40×40) are weighted and summed to enhance the feature representation of small targets and obtain an initial fused feature map at three scales.

[0058] Feature compression: The initial fused feature maps at the three scales are compressed using a 1×1 convolutional layer (reducing the number of channels from 256 / 512 to 128 / 256) to reduce computational cost without losing key features. Then, the feature maps at each scale are normalized to accelerate model convergence and improve feature stability, resulting in fused feature maps at the three scales (shapes of [BatchSize,128,80,80], [BatchSize,256,40,40], and [BatchSize,256,20,20]) + metadata processing.

[0059] The core function of the original detection module is to realize the bounding box regression, category and confidence calculation of the target based on the fusion feature map of multiple scales, and accurately identify targets with potential security risks.

[0060] Prediction head decoding: Each scale of the fused feature map corresponds to a lightweight prediction head (composed of 2 convolutional layers), which outputs the bounding box offset (x / y / w / h), class probability distribution, and confidence score respectively;

[0061] Bounding box regression: Based on preset anchor boxes (9 anchor boxes optimized for the type of safety hazard target, such as [10,13], [16,30], etc.), the coordinates of the target on the feature map are obtained by decoding the offset, and then mapped to the 640×640 image coordinate system by scaling ratio;

[0062] Category: The Softmax function is used to normalize the category probability distribution, and the output is the probability that the target belongs to each preset category (such as not wearing a safety helmet, illegally stacking materials, etc., 20+ categories);

[0063] Confidence score calculation: The final confidence score is obtained by multiplying the confidence score of the target existence by the class probability, which measures the reliability of the detection results.

[0064] The core function of the post-processing module is to filter, optimize, and map coordinates of the original detection results, remove invalid results, and output accurate target coordinates and related information that meet the system requirements.

[0065] Non-maximum suppression (NMS): For multiple detection boxes of the same target, an IOU (Intersection over Union) threshold (dynamically adjusted, default 0.45) is used to filter them, retaining the unique detection box with the highest confidence and avoiding duplicate detection;

[0066] Dynamic confidence filtering: Adjusting the confidence threshold based on the scene label of the input image (from metadata):

[0067] In densely populated environments (such as smart construction sites): a threshold of 0.5 is used to reduce the risk of missed detections.

[0068] For static equipment scenarios (such as data center monitoring): a threshold of 0.7 is used to reduce the risk of false detections.

[0069] The original detection results with confidence levels below the dynamic threshold are removed to obtain the screening results;

[0070] Inverse coordinate mapping: Combining the scaling ratio and padding parameters in the metadata, the bounding box coordinates in the 640×640 image coordinate system of the filtering results are inversely mapped to the normalized coordinates in the detection image resolution (x1 / y1 / x2 / y2 are mapped to the [0,1] interval).

[0071] Abnormal result labeling: For detected targets with confidence levels in the fuzzy range (0.4-0.6), they are marked as areas to be reviewed and scene feature labels (such as severe occlusion or insufficient lighting) are attached.

[0072] Results are structured: The results are organized according to the format of target ID, normalized coordinates, category ID, category name, confidence score, and label status to generate structured output data, thus obtaining target detection data.

[0073] In one embodiment, the target cropping module 202 includes:

[0074] The coordinate mapping module 2021 is used to map normalized coordinates to pixel coordinates of the detected image;

[0075] The cropping expansion ratio adjustment module 2022 is used to adjust the cropping expansion ratio according to the category and confidence level of the detected target;

[0076] The region cropping module 2023 is used to crop the detected target within the pixel coordinates of the detected image according to the cropping expansion ratio, so as to obtain the detected target image.

[0077] The pruning and expansion ratio is dynamically adjusted based on the target category and confidence level to avoid losing context due to over-pruning or missing key features due to under-pruning.

[0078] High-confidence targets (confidence ≥ 0.7): Expand by 5%, retaining only the core context surrounding the target;

[0079] Medium confidence target (0.5 ≤ confidence < 0.7): expansion ratio 8%, balancing target focus and context preservation;

[0080] Low confidence / targets to be verified (confidence <0.5 or marked as to be verified): expansion ratio of 10% to maximize the preservation of surrounding environmental information and assist the semantic judgment of the large model.

[0081] The source information includes the detected image ID, source pixel coordinates, and processing metadata, which facilitates subsequent large model analysis to trace back to the position in the original image.

[0082] In step 103, the detected image and its accompanying metadata, the target region image and the structured metadata are respectively input into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model.

[0083] The multimodal large model is the central hub for intelligent analysis and reasoning. It receives high-resolution local images from lightweight object detection models. With its powerful cross-modal understanding capabilities, it can not only accurately identify targets, but also deeply interpret the scene context, behavioral intentions, and potential risks in the images.

[0084] Figure 3 This is a schematic diagram of the structure of a multimodal large model in an embodiment of the present invention. In one embodiment, the multimodal large model includes:

[0085] The cross-modal feature extraction module 301 is used to divide the image to be processed into multiple image patches and generate image patch embedding features. When the image to be processed is a target region image, the multi-head attention mechanism is used to focus on the target detail features in the image patch embedding features to obtain local features. After adding quality level attention weights, the features are bound to the structured metadata of the target region image to form a local feature group. When the image to be processed is a detection image, the global attention mechanism is used to capture the overall scene features in the image patch embedding features to obtain global features. After adding scene label attention weights, the features are bound to the attached metadata of the detection image to form a global semantic vector group.

[0086] The semantic parsing module 302 is used to generate local semantic information based on local feature groups and to generate global semantic information based on global feature groups.

[0087] In this embodiment of the invention, a multi-head attention mechanism is used to focus on the detailed features of the target, such as the outline of a safety helmet and the stacking shape of materials; a global attention mechanism is used to capture the overall features of the scene, such as the site layout, equipment distribution, and personnel density; quality-level attention weights are added to local features (1.0 for high-quality images, 0.8 for medium-quality images, and 0.6 for low-quality images) to enhance the expression of effective features; scene label attention weights are added to global features (e.g., 1.2 for densely populated scenes and 0.9 for static equipment scenes) to adapt to the semantic extraction needs of different scenes; local features are bound to the corresponding target IDs, and global features are bound to image IDs to establish an index association between local and global features. The aforementioned local and global features can be in vector form or other forms such as text, without limitation.

[0088] In this embodiment of the invention, the core function of the semantic parsing module is to analyze the local feature group in the local feature group and the global feature group in the global feature group to obtain local semantic information and global semantic information, both of which include scene type, core features and risk level, and generate a structured description list.

[0089] The semantic parsing module analyzes the local feature group in the local feature group and the global feature group in the global feature group through a fully connected layer and a Softmax function, and outputs the core hazard type of the target (such as not wearing a safety helmet and 20+ core categories) and the category confidence (∈[0,1]).

[0090] When extracting core features, an attention pooling mechanism is used to extract key detail features of the target from local features, such as: not wearing a safety helmet → extracting core features such as no hard protective equipment on the head and exposed hair; stacked materials → extracting core features such as stacking height exceeding 1.5 meters and blocking fire lanes.

[0091] A global attention pooling mechanism is adopted to extract key macro-risk features of the scenario from global features, such as: construction site scenario: based on smart construction site - high-altitude operation area → extract core features such as insufficient high-altitude operation protection, partial blockage of fire lanes, and messy temporary power lines; warehousing scenario: based on factory workshop - storage area → extract core features such as excessive material stacking, expired fire extinguishers, and poor ventilation.

[0092] Risk level assessment: Based on core features and scenario classification confidence, the fuzzy comprehensive evaluation method is used to calculate the risk level (low / medium / high / extremely high):

[0093] Extremely high risk level: The scene classification confidence level is ≥0.8 and multiple high-risk core features are superimposed (such as insufficient protection for high-altitude operations + dense population + slippery ground).

[0094] High-risk level: The scenario classification confidence level is ≥0.7 and the core features contain critical security vulnerabilities (such as completely blocked fire exits + flammable materials piled up).

[0095] Medium risk level: The scenario classification confidence level is ≥0.6 and the core features contain local safety hazards (such as materials being stacked too high but not blocking the passage).

[0096] Low risk level: The confidence level of the scenario classification is ≥0.5 and the core features contain minor non-compliant behaviors (such as blurred but not missing safety labels);

[0097] For local information, the structured description format is: target ID + source pixel coordinates + core features + risk level + quality description. Local information is described using natural language text (e.g., based on target ID: T001, source coordinates: (300,0)-(600,180), scene type: high-altitude work area, core features: no hard protective equipment on the head, hair exposed, risk level: high, quality description: clear image, clear features).

[0098] For global information, the structured description format is as follows: image ID + scene type + core features + risk level + quality description, generating natural language description text (e.g., image ID: I001, scene type: smart construction site - high-altitude work area, core features: insufficient high-altitude work protection, partially blocked fire lanes, messy temporary power lines, risk level: extremely high, environmental influencing factors: rainy weather, slippery ground, quality description: clear global image, complete scene features).

[0099] The structured descriptions described above are typically output in JSON format or other dictionary-like formats. Structured output allows for the rapid retrieval of necessary information via program scripts.

[0100] In one embodiment, the local semantic information includes at least the target ID, source pixel coordinates, scene type, core features, and risk level;

[0101] Global semantic information includes at least image ID, scene type, core features, and risk level.

[0102] In step 104, global semantic information is retrieved by keyword search, vector matching and cross-modal adapter search in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios.

[0103] The external professional knowledge base is a structured, massive knowledge repository that systematically stores and links various professional safety information, including standard safety operating procedures, detailed historical cases, safety regulations, and best practice examples. Its content is carefully labeled and indexed, providing comprehensive data support for accurate retrieval.

[0104] In addition to using global semantic information to search in external professional databases, embodiments of the present invention also support searching through global semantic features.

[0105] In one embodiment, global semantic information is retrieved through keyword retrieval, vector matching, and cross-modal adapter retrieval in an external professional knowledge base to obtain security knowledge entries, including:

[0106] Based on global semantic information, keyword groups are constructed, and weights are assigned to each keyword in the keyword group; the keyword group includes scenario keywords, core risk keywords, and risk level keywords corresponding to scenario type, core features, and risk level, respectively.

[0107] Using keyword groups as query conditions, and based on dual indexes of scenario type and core features, as well as a text retrieval index, a set of first security knowledge entries with a keyword matching degree greater than a first matching degree threshold is retrieved from an external professional knowledge base. The text retrieval index is formed according to scenario type, core features, and risk level.

[0108] The vector encoded by keyword groups is compared with the vector in the external professional knowledge base by cosine similarity calculation, and the second set of security knowledge items with a cosine similarity greater than the first similarity threshold is selected.

[0109] A cross-modal adapter is used to semantically align keyword groups with security knowledge entries in an external professional knowledge base to obtain a third set of security knowledge entries.

[0110] The first set of safety knowledge items, the second set of safety knowledge items, and the third set of safety knowledge items are merged and deduplicated to obtain a candidate set of safety knowledge items;

[0111] For each security knowledge entry in the candidate security knowledge entry set, security clauses at different levels are associated based on a hierarchical association index.

[0112] Reorder all security knowledge entries in the candidate security knowledge entry set to obtain the first number of security knowledge entries after sorting.

[0113] In this embodiment of the invention, the weights of scenario keywords, core risk keywords, and risk level keywords can be set to 0.3, 0.4, and 0.3, respectively, depending on the actual situation.

[0114] External expertise bases include databases, knowledge graphs, and image similarity vulnerability databases.

[0115] The database is structured according to the following attributes: knowledge type, core semantics, applicable scenario, risk level, associated environment, authority level, and hierarchy.

[0116] The basic fields include knowledge type (legal provisions / industry standards / local regulations / corporate procedures / historical cases), core semantics, applicable scenarios, risk level, and associated environment;

[0117] The extended fields include hierarchical attributes, associated case IDs, and rectification requirements, laying the foundation for multi-level associations.

[0118] The external professional knowledge base index includes a basic index and an extended index. The basic index includes a dual index of scene type and core features, as well as a text retrieval index, which supports keyword and vector matching. The extended index includes a hierarchical association index, which adapts to the needs of multi-level rule association.

[0119] The knowledge graph is built based on a large amount of security knowledge, and the construction process includes:

[0120] Define the core entity types of the graph, such as knowledge items (including attributes: knowledge type, core semantics, authority level, etc.), scenarios, risk factors, risk levels, environmental factors, security clause levels, etc.

[0121] Define the types of relationships between entities, such as applicable scenarios (knowledge item - scenario), associated risk factors (knowledge item - risk factor), belonging level (knowledge item - security clause level), associated environment (knowledge item - environmental factors), etc.

[0122] Entity and Relationship Extraction: Extract the aforementioned entities from the knowledge entries and establish relationships between entities based on the association information of the knowledge entries.

[0123] The image similarity vulnerability database adopts a four-layer architecture of "image metadata layer - feature vector layer - associated knowledge layer - index layer", and is also linked to an external text-based security knowledge base to achieve the linkage of "image-text-rules".

[0124] The image metadata layer can be used to store basic information and structured descriptions of potential hazard images; the feature vector layer can be used to store deep feature vectors of images; the associated knowledge layer can be used to associate text-based safety knowledge items; the index layer can be used to build multi-dimensional retrieval indexes; and the application of the image similarity hazard database needs to embed the aforementioned knowledge retrieval process driven by global semantic information, focusing on solving the problems of "similar matching of image-based hazards" and "cross-modal linkage of image-text knowledge".

[0125] External professional knowledge bases can load multi-dimensional indexes into the search engine, supporting mixed searches and multi-level related queries. When new content is added to the knowledge base, incremental index updates are automatically triggered to ensure the timeliness of knowledge and search adaptability. The search engine simultaneously searches the database, knowledge graph, and image similarity vulnerability database during execution.

[0126] In this embodiment of the invention, the first matching degree threshold can be determined according to the actual situation, for example, 70%, based on the dual index of scene type and core feature and text retrieval index, to recall security knowledge items with keyword matching degree greater than the first matching degree threshold;

[0127] Vector matching (semantic precision filtering): Convert keyword groups into high-dimensional vectors and perform cosine similarity calculation with vectors in an external professional knowledge base; filter security knowledge entries with a cosine similarity greater than the first similarity threshold (e.g., 0.75).

[0128] Cross-modal adapter (semantic alignment optimization): Invoke a specially trained description-clause cross-modal adapter to semantically align keyword groups with security knowledge entries in an external knowledge base to obtain a third set of security knowledge entries.

[0129] The candidate security knowledge item set is reordered by a cross encoder, taking into account semantic similarity, applicability to scenarios, and risk level relevance. The first number of most relevant knowledge base items are output, and each item includes the original text of the clause, an explanation of the applicable scenario, a case link, and rectification requirements.

[0130] In step 105, local semantic information, global semantic information, and safety knowledge items are input into the multimodal large model to obtain the safety hazard area, safety hazard content, improvement suggestions, and safety regulations;

[0131] In one embodiment, local semantic information, global semantic information, and security knowledge entries are input into a multimodal large model to obtain security hazard areas, security hazard content, improvement suggestions, and security regulations, including:

[0132] Local semantic information, global semantic information, and security knowledge items are bound together according to the scenario-objective-risk-knowledge link;

[0133] Based on the source pixel coordinates of local semantic information, combined with the metadata processing of a lightweight target detection model, the target region image is reverse-mapped to the precise pixel coordinates of the detection image, generating a security hazard region with bounding box annotations.

[0134] By integrating the core features of local semantics with the basis of security knowledge entries, content on security risks is generated;

[0135] Based on the aforementioned link, improvement suggestions and security regulations corresponding to historical cases in the security knowledge entries are obtained.

[0136] In this embodiment of the invention, the source pixel coordinates based on local semantic information are combined with the processing metadata (scaling ratio, padding parameters) of the lightweight target detection model and back-mapped to the precise pixel coordinates of the original detection image to generate a security hazard area with bounding box annotation; for the area to be reviewed, scene feature labels (such as severe occlusion, based on insufficient lighting) and uncertainty descriptions are superimposed.

[0137] By integrating the core features of local semantics with the basis of safety knowledge entries, safety hazard content is generated, which includes the manifestation of the hazard and a description of the violation.

[0138] In one embodiment, the method further includes:

[0139] The areas and content of safety hazards are input into a multimodal large model for spatial consistency verification, temporal continuity verification, and semantic rationality verification to obtain secondary verification results;

[0140] If the secondary verification result indicates a problem, the area with security risks will be marked as an area pending review.

[0141] In this embodiment of the invention, spatial consistency verification refers to checking whether the spatial distribution of the security hazard area conforms to physical constraints and scene logic. Temporal continuity verification refers to analyzing the temporal stability of all the above detection results in a video stream formed by multiple detection images. Semantic rationality verification refers to judging the semantic rationality of the detection object in combination with the scene context. When the secondary verification result differs significantly from the initial security hazard area and security hazard content, the sample is automatically marked as a region to be reviewed and included in the difference sample library.

[0142] Afterwards, the reviewers can quickly view the raw data, model output, secondary validation results, and contextual information to make a final judgment.

[0143] Furthermore, although automated systems can identify most anomalies, relying entirely on machine decision-making under current technological conditions still carries risks. Therefore, embodiments of the present invention also incorporate a tiered human intervention mechanism.

[0144] Automatic push for low-confidence samples: Images with a confidence level below a threshold are automatically assigned to trained human reviewers.

[0145] Random sampling review: A certain percentage of samples with high confidence are randomly selected periodically for manual review to prevent blind spots in the algorithm.

[0146] Enhanced review for key scenarios: Conduct full or high-percentage review of the test results for key business scenarios.

[0147] Meanwhile, the data annotation platform built using the aforementioned information allows reviewers to quickly view raw data, model output, and contextual information to make a final judgment. This assists manual reviewers in quickly locating and judging errors. The review results not only include binary labels for correct / incorrect errors but also detailed error type classifications, correction suggestions, and scenario notes.

[0148] The accumulated review results form a high-quality training dataset, triggering a model update process monthly or quarterly. The capabilities of the small models are continuously improved, while the degree of manual review and intervention is gradually reduced. The process fully transitions to a parallel approach of secondary validation results from multimodal large models and detection by small models.

[0149] This invention also proposes a security inspection system based on multi-model collaboration, the principle of which is similar to the security inspection method based on multi-model collaboration, and will not be described in detail here.

[0150] Figure 4 This is a schematic diagram of a security inspection system based on multi-model collaboration in an embodiment of the present invention, including:

[0151] Preprocessing module 401 is used to preprocess the multidimensional detection data of the target area to obtain the detection image and accompanying metadata, wherein the accompanying metadata includes scene labels;

[0152] The small model detection module 402 is used to input the detection image and accompanying metadata into a lightweight target detection model to obtain a target region image and structured metadata; the lightweight target detection model is used to perform target detection on the detection image according to the confidence threshold corresponding to the scene label to obtain target detection data, and to crop the detected target according to the target detection data to obtain a target region image;

[0153] The large model analysis module 403 is used to input the detection image and attached metadata, the target region image and structured metadata into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model.

[0154] The retrieval and matching module 404 is used to perform keyword retrieval, vector matching and cross-modal adapter retrieval of global semantic information in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios.

[0155] The comprehensive analysis module 405 is used to input local semantic information, global semantic information, and safety knowledge items into the multimodal large model to obtain safety hazard areas, safety hazard content, improvement suggestions, and safety regulations.

[0156] In one embodiment, the lightweight object detection model includes an object detection module and an object cropping module;

[0157] The target detection module includes:

[0158] The input adaptation module is used to process the detected image into image tensor data and obtain the processing metadata;

[0159] The preprocessing enhancement module is used to denoise and enhance image tensor data using a conditional diffusion model to obtain enhanced tensor data.

[0160] The multi-scale feature extraction module is used to extract multi-scale features from augmented tensor data and obtain a fused feature map of multiple scales.

[0161] The original detection module is used to perform target detection based on the fused feature map of multiple scales and obtain the original detection result, which includes the bounding box, category and confidence of the detected target;

[0162] The post-processing module is used to filter the original detection results according to the confidence threshold corresponding to the scene label in the processing metadata, and to perform coordinate inverse mapping on the filtered results according to the scaling ratio and padding parameters in the processing metadata, so as to map the bounding box of the detected target to the normalized coordinates of the detection image to obtain the target detection data. The target detection data includes the bounding box, category, confidence and processing metadata of the detected target.

[0163] The target cropping module is used to crop the detected target according to the target detection data to obtain the target region image and structured metadata, wherein the structured metadata includes source information.

[0164] In one embodiment, the target cropping module includes:

[0165] The coordinate mapping module is used to map normalized coordinates to pixel coordinates in the detected image;

[0166] The cropping expansion ratio adjustment module is used to adjust the cropping expansion ratio according to the category and confidence level of the detected target;

[0167] The region cropping module is used to crop the detected target within the pixel coordinates of the detected image according to the cropping expansion ratio, so as to obtain the detected target image.

[0168] In one embodiment, the multimodal large model includes:

[0169] The cross-modal feature extraction module is used to divide the image to be processed into multiple image patches and generate image patch embedding features. When the image to be processed is a target region image, the multi-head attention mechanism focuses on the target detail features in the image patch embedding features to obtain local features. After adding quality level attention weights, these features are bound to the structured metadata of the target region image to form a local feature group. When the image to be processed is a detection image, the global attention mechanism captures the overall scene features in the image patch embedding features to obtain global features. After adding scene label attention weights, these features are bound to the accompanying metadata of the detection image to form a global semantic vector group.

[0170] The semantic parsing module is used to generate local semantic information based on local feature groups and global semantic information based on global feature groups.

[0171] In one embodiment, the local semantic information includes at least the target ID, source pixel coordinates, scene type, core features, and risk level;

[0172] Global semantic information includes at least image ID, scene type, core features, and risk level.

[0173] In one embodiment, the retrieval matching module is used for:

[0174] Based on global semantic information, keyword groups are constructed, and weights are assigned to each keyword in the keyword group; the keyword group includes scenario keywords, core risk keywords, and risk level keywords corresponding to scenario type, core features, and risk level, respectively.

[0175] Using keyword groups as query conditions, and based on dual indexes of scenario type and core features, as well as a text retrieval index, a set of first security knowledge entries with a keyword matching degree greater than a first matching degree threshold is retrieved from an external professional knowledge base. The text retrieval index is formed according to scenario type, core features, and risk level.

[0176] The vector encoded by keyword groups is compared with the vector in the external professional knowledge base by cosine similarity calculation, and the second set of security knowledge items with a cosine similarity greater than the first similarity threshold is selected.

[0177] A cross-modal adapter is used to semantically align keyword groups with security knowledge entries in an external professional knowledge base to obtain a third set of security knowledge entries.

[0178] The first set of safety knowledge items, the second set of safety knowledge items, and the third set of safety knowledge items are merged and deduplicated to obtain a candidate set of safety knowledge items;

[0179] For each security knowledge entry in the candidate security knowledge entry set, security clauses at different levels are associated based on a hierarchical association index.

[0180] Reorder all security knowledge entries in the candidate security knowledge entry set to obtain the first number of security knowledge entries after sorting.

[0181] In one embodiment, the comprehensive analysis module is used for:

[0182] Local semantic information, global semantic information, and security knowledge items are bound together according to the scenario-objective-risk-knowledge link;

[0183] Based on the source pixel coordinates of local semantic information, combined with the metadata processing of a lightweight target detection model, the target region image is reverse-mapped to the precise pixel coordinates of the detection image, generating a security hazard region with bounding box annotations.

[0184] By integrating the core features of local semantics with the basis of security knowledge entries, content on security risks is generated;

[0185] Based on the aforementioned link, improvement suggestions and security regulations corresponding to historical cases in the security knowledge entries are obtained.

[0186] In one embodiment, the system further includes a secondary verification module, used for:

[0187] The areas and content of safety hazards are input into a multimodal large model for spatial consistency verification, temporal continuity verification, and semantic rationality verification to obtain secondary verification results;

[0188] If the secondary verification result indicates a problem, the area with security risks will be marked as an area pending review.

[0189] The system proposed in this invention involves the setup of hardware and the basic environment during deployment. Deployment on computing nodes equipped with high-performance GPU servers is recommended to ensure efficient operation of the large multimodal model; meanwhile, the lightweight object detection model can be deployed on edge computing devices. All models are encapsulated as independent Docker containers, communicating and exchanging data through a microservice architecture, thereby ensuring the system's scalability and maintainability.

[0190] The image cropping module is implemented based on the normalized coordinates output by the first model. This module preserves the original high-resolution image and uses a bilinear interpolation algorithm to accurately crop the region of interest based on the coordinates. These cropped image patches are then rescaled to a standard size, along with metadata from the original location, and combined into a batch tensor before being fed into the downstream model.

[0191] The large-scale visual language model module is a large-scale visual language model. The implementation of the external professional knowledge base is based on a search engine that excels at full-text search and vector similarity calculation. Each knowledge entry in the database is converted into a dual representation containing both text content and dense vectors. During construction, industry standards are structured and organized, and multiple semantic tags are added to each piece of content to form a knowledge graph that can be retrieved by machines.

[0192] The core of the search and matching module is a hybrid retrieval system. It first receives text descriptions generated by a multimodal large model. On one hand, it uses traditional keywords for rapid retrieval in the knowledge base; on the other hand, it converts the query text into vectors and performs semantic similarity matching through a vector database. Finally, a cross-encoder re-ranks the preliminary results, filtering out the knowledge entries most relevant to the current potential hazard, ensuring the accuracy and authority of the returned results.

[0193] In summary, the core advantage of the method and system proposed in this invention lies in the successful construction of an efficient, accurate, and deeply knowledge-supported intelligent analysis system. By combining a lightweight detection model with a multimodal large model, a clever balance is struck between detection efficiency and accuracy. This allows for both real-time, rapid global scanning using the lightweight model and deep semantic understanding and scene reasoning of potential risk areas using the multimodal large model, thereby achieving accurate identification of complex security risks and surpassing the limitations of traditional single-target detection methods.

[0194] Building upon this foundation, an innovative approach incorporates external professional knowledge databases and an intelligent search and matching mechanism, injecting authoritative domain knowledge into the analysis and decision-making process. The semantic descriptions generated by the multimodal large model can be intelligently correlated and matched with security regulations, operating procedures, and historical cases in the database. This ensures that the final detection results are not only based on visual analysis but also have solid legal basis and case support, significantly enhancing the professionalism and credibility of the decisions.

[0195] Furthermore, the solution exhibits excellent evolvability; its external professional knowledge base can continuously expand with regulatory updates and case accumulation, adapting to new security requirements without reconstructing the core model. Finally, the mechanism of secondary verification of output results through a multimodal large model effectively ensures a high degree of consistency between the hazard area and the described content, further enhancing the reliability and robustness of the entire system's output. Ultimately, this forms a complete, closed-loop intelligent security hazard detection solution from perception and analysis to verification.

[0196] Compared with existing technologies, this invention combines the semantic understanding of large models with the real-time reasoning advantages of small models, significantly improving the accuracy of hidden danger detection; through multi-model collaboration and knowledge distillation, it achieves lightweight and efficient deployment of models; it is applicable to security monitoring tasks in multiple scenarios and has good scalability and generalization ability.

[0197] This invention also provides a computer device. Figure 5 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 500 includes a memory 510, a processor 520, and a computer program 530 stored in the memory 510 and executable on the processor 520. When the processor 520 executes the computer program 530, it implements the above-mentioned security inspection method based on multi-model collaboration.

[0198] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned security inspection method based on multi-model collaboration.

[0199] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the aforementioned security inspection method based on multi-model collaboration.

[0200] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0201] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0202] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0203] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0204] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A security inspection method based on multi-model collaboration, characterized in that, include: The multidimensional detection data of the target area is preprocessed to obtain the detection image and accompanying metadata, the accompanying metadata including scene labels; The detected image and accompanying metadata are input into a lightweight object detection model to obtain the target region image and structured metadata; The lightweight object detection model is used to perform object detection on the detection image according to the confidence threshold corresponding to the scene label, obtain object detection data, and crop the detected object according to the object detection data to obtain the target region image; The detected image and its accompanying metadata, the target region image and the structured metadata are respectively input into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model. Global semantic information is retrieved by keyword retrieval, vector matching, and cross-modal adapter retrieval in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios. The cross-modal adapter retrieval method employs a cross-modal adapter to semantically align keyword groups with security knowledge entries in an external professional knowledge base. The keyword groups are constructed based on global semantic information. By inputting local semantic information, global semantic information, and safety knowledge items into the multimodal large model, we can obtain areas of safety hazards, content of safety hazards, improvement suggestions, and safety regulations. Multimodal large models include: The cross-modal feature extraction module is used to divide the image to be processed into multiple image patches and generate image patch embedding features; When the image to be processed is a target region image, a multi-head attention mechanism is used to focus on the target detail features in the image patch embedding features to obtain local features. After adding quality level attention weights, these features are bound to the structured metadata of the target region image to form a local feature group. When the image to be processed is a detection image, a global attention mechanism is used to capture the overall scene features in the image patch embedding features to obtain global features. After adding scene label attention weights, these features are bound to the accompanying metadata of the detection image to form a global semantic vector group. The quality level attention weights are the weights corresponding to the quality level of the target region image; the scene label attention weights are the weights corresponding to the scene in which the detection image is located. The semantic parsing module is used to generate local semantic information based on local feature groups and global semantic information based on global feature groups.

2. The method as described in claim 1, characterized in that, The lightweight object detection model includes an object detection module and an object cropping module; The target detection module includes: The input adaptation module is used to process the detected image into image tensor data and obtain the processing metadata; The preprocessing enhancement module is used to denoise and enhance image tensor data using a conditional diffusion model to obtain enhanced tensor data. The multi-scale feature extraction module is used to extract multi-scale features from augmented tensor data and obtain a fused feature map of multiple scales. The original detection module is used to perform target detection based on the fused feature map of multiple scales and obtain the original detection result, which includes the bounding box, category and confidence of the detected target; The post-processing module is used to filter the original detection results according to the confidence threshold corresponding to the scene label in the processing metadata, and to perform coordinate inverse mapping on the filtered results according to the scaling ratio and padding parameters in the processing metadata, so as to map the bounding box of the detected target to the normalized coordinates of the detection image to obtain the target detection data. The target detection data includes the bounding box, category, confidence and processing metadata of the detected target. The target cropping module is used to crop the detected target according to the target detection data to obtain the target region image and structured metadata, wherein the structured metadata includes source information.

3. The method as described in claim 2, characterized in that, The target cropping module includes: The coordinate mapping module is used to map normalized coordinates to pixel coordinates in the detected image; The cropping expansion ratio adjustment module is used to adjust the cropping expansion ratio according to the category and confidence level of the detected target; The region cropping module is used to crop the detected target within the pixel coordinates of the detected image according to the cropping expansion ratio, so as to obtain the detected target image.

4. The method as described in claim 1, characterized in that, Local semantic information includes at least the target ID, source pixel coordinates, scene type, core features, and risk level; Global semantic information includes at least image ID, scene type, core features, and risk level.

5. The method as described in claim 1, characterized in that, Global semantic information is retrieved through keyword retrieval, vector matching, and cross-modal adapter retrieval in an external professional knowledge base to obtain security knowledge entries, including: Based on global semantic information, keyword groups are constructed, and weights are assigned to each keyword in the keyword group; the keyword group includes scenario keywords, core risk keywords, and risk level keywords corresponding to scenario type, core features, and risk level, respectively. Using keyword groups as query conditions, and based on dual indexes of scenario type and core features, as well as a text retrieval index, a set of first security knowledge entries with a keyword matching degree greater than a first matching degree threshold is retrieved from an external professional knowledge base. The text retrieval index is formed according to scenario type, core features, and risk level. The vector encoded by keyword groups is compared with the vector in the external professional knowledge base by cosine similarity calculation, and the second set of security knowledge items with a cosine similarity greater than the first similarity threshold is selected. A cross-modal adapter is used to semantically align keyword groups with security knowledge entries in an external professional knowledge base to obtain a third set of security knowledge entries. The first set of safety knowledge items, the second set of safety knowledge items, and the third set of safety knowledge items are merged and deduplicated to obtain a candidate set of safety knowledge items; For each security knowledge entry in the candidate security knowledge entry set, security clauses at different levels are associated based on a hierarchical association index. Reorder all security knowledge entries in the candidate security knowledge entry set to obtain the first number of security knowledge entries after sorting.

6. The method as described in claim 1, characterized in that, Local semantic information, global semantic information, and security knowledge items are input into a multimodal large model to obtain security hazard areas, security hazard content, improvement suggestions, and security regulations, including: Local semantic information, global semantic information, and security knowledge items are bound together according to the scenario-objective-risk-knowledge link; Based on the source pixel coordinates of local semantic information, combined with the metadata processing of a lightweight target detection model, the target region image is reverse-mapped to the precise pixel coordinates of the detection image, generating a security hazard region with bounding box annotations. By integrating the core features of local semantics with the basis of security knowledge entries, content on security risks is generated; Based on the aforementioned link, improvement suggestions and security regulations corresponding to historical cases in the security knowledge entries are obtained.

7. The method as described in claim 1, characterized in that, Also includes: The areas and content of safety hazards are input into a multimodal large model for spatial consistency verification, temporal continuity verification, and semantic rationality verification to obtain secondary verification results; If the secondary verification result indicates a problem, the area with security risks will be marked as an area pending review.

8. A security inspection system based on multi-model collaboration, characterized in that, include: The preprocessing module is used to preprocess the multidimensional detection data of the target area to obtain the detection image and accompanying metadata, wherein the accompanying metadata includes scene labels; The small model detection module is used to input the detection image and accompanying metadata into the lightweight object detection model to obtain the target region image and structured metadata; the lightweight object detection model is used to perform object detection on the detection image according to the confidence threshold corresponding to the scene label to obtain object detection data, and to crop the detected target according to the object detection data to obtain the target region image; The large model analysis module is used to input the detection image and its accompanying metadata, the target region image and its structured metadata into the multimodal large model to obtain global semantic information and local semantic information. The multimodal large model is a large visual language model. The retrieval and matching module is used to perform keyword retrieval, vector matching, and cross-modal adapter retrieval of global semantic information in an external professional knowledge base to obtain security knowledge entries. The external professional knowledge base is used to store historical cases and security regulations for different scenarios. The cross-modal adapter retrieval method employs a cross-modal adapter to semantically align keyword groups with security knowledge entries in an external professional knowledge base. The keyword groups are constructed based on global semantic information. The comprehensive analysis module is used to input local semantic information, global semantic information, and security knowledge items into the multimodal large model to obtain security hazard areas, security hazard content, improvement suggestions, and security regulations. Multimodal large models include: The cross-modal feature extraction module is used to divide the image to be processed into multiple image patches and generate image patch embedding features; When the image to be processed is a target region image, a multi-head attention mechanism is used to focus on the target detail features in the image patch embedding features to obtain local features. After adding quality level attention weights, these features are bound to the structured metadata of the target region image to form a local feature group. When the image to be processed is a detection image, a global attention mechanism is used to capture the overall scene features in the image patch embedding features to obtain global features. After adding scene label attention weights, these features are bound to the accompanying metadata of the detection image to form a global semantic vector group. The quality level attention weights are the weights corresponding to the quality level of the target region image; the scene label attention weights are the weights corresponding to the scene in which the detection image is located. The semantic parsing module is used to generate local semantic information based on local feature groups and global semantic information based on global feature groups.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.

Images