A security scene risk identification method, device and equipment

Abstract

This application discloses a method, apparatus, and device for risk identification in security scenarios. The method includes: acquiring target detection data of at least one modality in a target security scenario; wherein each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras; inputting each target detection data into a preprocessing model corresponding to each modality to detect at least one type of preset target object, obtaining the detection results and confidence scores corresponding to each modality, as well as video context information of the video detection results; responding to a video confidence score corresponding to a video modality being less than a preset threshold, inputting the video detection results and video context information into a preset visual language large model to obtain event semantic description information; and identifying the risk of the target object based on the detection results and event semantic description information corresponding to each modality. This application's solution improves the accuracy of identifying security risks in a park.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a risk identification method, apparatus and equipment for security scenarios. Background Technology

[0002] Currently, the intelligent analysis methods for park security monitoring are relatively simple. The monitoring system can only perform basic identification of objects such as people and vehicles within the frame, and the output monitoring information is relatively superficial, generating only simple alarm prompts. With the widespread application of artificial intelligence technology in the security field, park security systems are gradually evolving from traditional manual monitoring modes to intelligent monitoring modes based on computer vision.

[0003] Current mainstream improvements to campus security systems typically employ algorithms based on visual artificial intelligence models to analyze surveillance videos, enabling personnel detection, vehicle recognition, and behavior identification, thereby analyzing potential security risks. How to further improve the accuracy of security risk identification has become a key research focus for professionals in related fields. Summary of the Invention

[0004] This application provides a risk identification method, apparatus, and device for security scenarios to improve the accuracy of security risk identification.

[0005] According to one aspect of this application, a risk identification method for security scenarios is provided, comprising: Acquire target detection data of at least one modality in a target security scenario; wherein each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras; Input the detection data of each target into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the detection results and confidence scores corresponding to each modality, as well as the video context information of the video detection results; When the video confidence corresponding to a video modality is less than a preset threshold, the video detection results and video context information are input into a preset visual language large model to obtain event semantic description information. Based on the detection results and event semantic description information corresponding to each modality, the risk of the target object is identified.

[0006] According to another aspect of this application, a risk identification device for security scenarios is provided, comprising: The detection data acquisition module is used to acquire target detection data of at least one modality in the target security scene; wherein, each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras; The detection result determination module is used to input the detection data of each target into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the detection results and confidence scores corresponding to each modality, as well as the video context information of the video detection results. The description information determination module is used to input the video detection results and video context information into the preset visual language large model in response to the video confidence corresponding to the video modality being less than a preset threshold, so as to obtain the event semantic description information. The object risk identification module is used to identify the risk of a target object based on the detection results and event semantic description information corresponding to each modality.

[0007] According to another aspect of this application, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the risk identification method for security scenarios described in any embodiment of this application.

[0008] According to another aspect of this application, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the risk identification method for security scenarios described in any embodiment of this application.

[0009] According to another aspect of this application, a computer program product is provided, the computer program product including a computer program that, when executed by a processor, implements the risk identification method for security scenarios according to any embodiment of this application.

[0010] In the technical solution of this application embodiment, multi-view video multimodal detection data is collected simultaneously, and target detection is completed based on a preprocessing model. For targets with insufficient recognition credibility, a large visual language model is invoked to supplement semantic information, and the two types of data are integrated to comprehensively identify risks. This application solution takes into account both basic target recognition and complex event semantic analysis, breaks through the limitations of single monitoring analysis, and improves the comprehensiveness and accuracy of target risk identification in security scenarios.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1A This is a flowchart of a risk identification method for a security scenario provided in Embodiment 1 of this application; Figure 1B This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 1C This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 1D This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 1E This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 1F This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 1G This is a schematic diagram of the risk identification method provided according to Embodiment 1 of this application; Figure 2 This is a schematic diagram of the system architecture for multi-level visual model collaborative reasoning and risk classification in the security scenario corresponding to Embodiment 2 of this application; Figure 3 This is a schematic diagram of the structure of a risk identification device for a security scenario according to Embodiment 3 of this application; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the risk identification method for security scenarios according to embodiments of this application. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1A This application provides a flowchart of a risk identification method for security scenarios in Embodiment 1. This embodiment is applicable to situations involving monitoring and risk identification of objects (such as people and vehicles) in security scenarios. The method can be executed by a risk identification device for security scenarios, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1A As shown, the method includes: S110. Acquire target detection data of at least one modality in the target security scene; wherein each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras.

[0017] The target security scenario can be any area requiring security control. In this application embodiment, it can refer to a full-area security control scenario for a park, such as dividing control areas into park entrances and exits, perimeter walls, fire lanes, equipment rooms, and underground parking garages. The modality can be the type of signal acquisition used to identify target objects, such as, but not limited to, video modality, infrared modality, and radar modality. This application embodiment and its implementations focus on video modality. Target detection data can be raw acquisition data that can be parsed by a preprocessing model and used to identify target objects such as people and vehicles. Furthermore, the park-wide network of high-definition surveillance cameras, such as fixed cameras and intelligent PTZ cameras, has 24 / 7 video acquisition capabilities, forming a combination of multiple cameras capturing multiple different perspectives. These different perspectives can be formed by the different deployment heights, shooting angles, and shooting directions of each camera, enabling full-coverage video acquisition of the park.

[0018] For example, multiple surveillance cameras deployed within the park continuously collect real-time video streams. Relying on the park's intranet, the raw video from different perspectives is transmitted to the nearest edge computing node. The edge computing node then aggregates the multiple video streams, performs noise reduction and frame slicing preprocessing, and obtains target detection data for the target security scenario in video modalities. It should be noted that various sensors (such as infrared sensors and thermal imagers) pre-installed within the park can be used to acquire target detection data in multiple modalities.

[0019] S120. Input the detection data of each target into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the detection results and confidence scores corresponding to each modality, as well as the video context information of the video detection results.

[0020] The preprocessing model can be a lightweight AI detection model adapted to the corresponding modality and deployed on edge computing nodes. For example, the preprocessing model for the video modality can use a lightweight visual detection model from the You Only LookOnce (YOLO) series of single-stage object detection algorithms. The target object can be an object to be identified and controlled under park security regulations, including but not limited to personnel, motor vehicles, and non-motorized vehicles. The detection result can be structured data such as the target location, category, and image slices output by the model inference. The confidence score can be a reliable score indicating the model's judgment of the target's authenticity and category accuracy, with a value between 0 and 1. The video context information can be spatiotemporal behavioral supplementary data bound to a single target, used to assist in verification and judgment, and generated solely based on the video modality detection results. For example, it can be generated based on the detection of video data accompanying the YOLO model.

[0021] For example, the edge computing node calls the locally deployed YOLO lightweight visual mini-model, accesses multi-view video data, performs target recognition and detection frame by frame, determines targets such as people and vehicles in the scene, calculates and determines the confidence value simultaneously, and binds each recognized target to generate exclusive video context information to provide data support for subsequent hierarchical inference.

[0022] S130. In response to the video confidence level corresponding to the video modality being less than a preset threshold, the video detection result and video context information are input into the preset visual language large model to obtain event semantic description information.

[0023] The video confidence score can be the single-objective comprehensive confidence score calculated by the YOLO mini-model of the video modality. Correspondingly, the preset threshold can be a critical value for confidence judgment of the video confidence score; for example, a common threshold for park security could be 0.7. A score higher than the threshold indicates a reliable identification, while a score lower than the threshold indicates a questionable identification. The Vision-Language Model (VLM) can be a pre-trained, finely tuned cross-modal semantic reasoning model adapted to security scenarios, possessing the ability to jointly understand text and images semantically. The event semantic description information can be the natural language structured event content output by this Vision-Language Model, which may include, but is not limited to, information such as objective behavior, scene attributes, and risk assessment.

[0024] For example, the overall confidence score of a single-target video is compared in real time with a preset threshold of 0.7. If the confidence score is less than 0.7, it means that it cannot be confirmed whether the identified target object matches several preset object types. For example, the target may have occlusion, backlighting, or be blurred at a distance. The YOLO small model's recognition result is not reliable enough. Therefore, the target video detection result and video context information are packaged and pushed to a pre-deployed and debugged visual language large model. The large model then performs deep semantic analysis and outputs corresponding event semantic description information.

[0025] S140. Based on the detection results and event semantic description information corresponding to each modality, identify the risk of the target object.

[0026] The detection results for each modality can be structured data such as target category, location, and trajectory output by all modal models from multiple cameras and multiple perspectives. Risk identification can be a process of judging whether the target behavior is compliant and whether there are security risks such as park intrusion, unauthorized loitering, or boundary crossing by combining model analysis results and park management rules.

[0027] For example, by integrating the detection data corresponding to the target location, category, and trajectory output by the YOLO lightweight small model, and overlaying the behavioral intent and scene semantic description information output by the visual language large model, and combining the park's overall security control rules, the system can comprehensively determine whether the current behavior of personnel, vehicles, and other target objects is compliant, and complete the risk qualitative identification of the target objects.

[0028] For example, the system aggregates raw video data from multiple cameras in the park from different perspectives to complete video modal data access; then, a lightweight YOLO preprocessing model at the edge performs full-domain target detection, outputting target detection results, recognition confidence, and the exclusive spatiotemporal context information of the bound target; subsequently, the system makes branch judgments based on the confidence threshold, and for targets with low confidence and suspicion, it calls the VLM visual language model to complete the image and text semantic review and generate event semantic description content; finally, it integrates the concrete detection data of the small model and the semantic judgment data of the large model, combined with the park's security rules, to complete the full-domain risk identification of personnel and vehicle targets in the park. The entire process relies on a collaborative architecture of small and large models, taking into account both the real-time performance of edge detection and the accuracy of judgment in complex scenarios.

[0029] In one optional embodiment, the detection result may include infrared thermal detection result and audio waveform detection result; wherein, the infrared thermal detection result may be the detection result obtained by processing the information of the target object collected by the infrared thermal sensor using an infrared preprocessing model; similarly, the audio waveform detection result may be the detection result obtained by processing the information of the target object collected by the sound sensor using an audio preprocessing model. Correspondingly, such as Figure 1B As shown, step S140, which identifies the risk of a target object based on the detection results and event semantic description information corresponding to each modality, may include: S141. Based on the infrared thermal detection results, determine the target heat source characteristics of the target object.

[0030] The target heat source features can be structured features that characterize the heat source attributes of the target object, extracted from infrared thermal detection results. These features may include, but are not limited to, peak heat source temperature, heat source coverage area, heat source movement speed, heat source duration, and spatial coordinates. This information is then converted into target heat source features using a pre-prepared vectorized encoder.

[0031] S142. Based on the audio waveform detection results, determine the target sound wave characteristics of the target object.

[0032] The target acoustic wave features can be structured features extracted from the audio waveform detection results that can characterize the sound emitted by the target object in the security scenario of this application. For example, they may include, but are not limited to, the decibel amplitude of the sound wave, the abnormal frequency range, the location of the sound source, the duration of the abnormality, etc. This information is converted into target acoustic wave features through a pre-prepared vector encoder.

[0033] S143. Based on the preset vector embedding model, the event semantic description information is transformed into event semantic features.

[0034] The vector embedding model used in this application embodiment can be a text encoding model that has been fine-tuned and trained in advance based on a security dataset. It can realize the mapping and transformation of natural language semantics to high-dimensional feature vectors, and is suitable for security event semantic parsing scenarios.

[0035] S144. The target heat source features, the target sound wave features, and the event semantic features are fused into a vector according to the confidence level and the pre-assigned weights to obtain a fused feature vector; wherein the event semantic weight is greater than the heat source weight, and the heat source weight is greater than the sound wave weight.

[0036] The fusion method can be to perform vector weighted summation of the confidence scores and weights corresponding to different features. It is understandable that, due to the higher reliability of the video modality, the event semantic weight is the largest, while the sound modality is most susceptible to environmental interference, thus having the lowest sound wave weight.

[0037] S145. Input the fused feature vector into the preset security risk fully connected mapping layer and output the risk score.

[0038] The security risk fully connected mapping layer can be a neural network mapping layer pre-trained with security intrusion samples, which can quantify high-dimensional fusion features into standardized risk values. The integrated global fusion feature vector is imported into the pre-set and trained security risk fully connected mapping layer. Through feature parsing and quantization operations of the network layer, the multi-dimensional fusion features are transformed into quantifiable standardized risk scores, realizing the conversion of security risk from feature information to numerical indicators.

[0039] S146. Determine the risk level of the target object based on the risk score and the preset risk threshold range.

[0040] The risk threshold range can be a pre-defined standardized score range corresponding to different security risk levels, providing a judgment standard for risk classification. This range can be set by technical personnel in the relevant field based on extensive testing; this application does not specifically limit this. For example, based on the output quantified risk score, a preset multi-level risk threshold range is matched. By comparing the score with the threshold, the standardized security risk level corresponding to the target object is finally determined and output, completing the entire process of intelligent risk identification and judgment.

[0041] In the above implementation, three modal features—infrared heat source, sound wave, and event semantics—are extracted and weighted and fused. Event semantics serves as the core judgment criterion, supplemented by infrared and sound wave sensing information, taking into account both behavioral intent and objective environmental perception. Modal confidence-corrected feature weights are introduced to effectively suppress false detections caused by interference from light, noise, and temperature from a single sensor. Risk scores are quantified and output through a fully connected layer, and combined with graded judgments within fixed threshold ranges to achieve multi-layered information complementarity verification. Compared to traditional solutions relying solely on video recognition, this significantly reduces the probability of false alarms and missed alarms in obstructed, nighttime, and noisy environments. The risk grading results are interpretable, providing accurate and stable quantitative decision-making basis for differentiated early warning and response in park security.

[0042] In another alternative implementation, such as Figure 1C As shown, after identifying the risk of the target object based on the detection results and event semantic description information corresponding to each modality in S140, the method may further include: S150. Based on the target identification result of the target object for risk identification, query the pre-stored correspondence between risk level and risk identification result to obtain the target risk level corresponding to the target identification result.

[0043] The target identification result can be the final conclusion of S140 regarding whether the target object's behavior is compliant. The risk level correspondence can be a pre-stored mapping form in the backend, matching different risk levels. For example, it can be divided into four risk standards: Level 1 - Normal behavior, Level 2 - Minor abnormality, Level 3 - Violation, and Level 4 - High-risk intrusion. Different risk level standards correspond to various identification results, and the corresponding risk level can be queried or matched according to the identification result. Correspondingly, the target risk level can be the hierarchical label obtained by matching the target identification result in the correspondence.

[0044] S160. Based on the pre-set handling strategies corresponding to the risk levels, determine the target risk strategy corresponding to the target risk level.

[0045] The processing strategy can be a pre-defined hierarchical linkage handling rule corresponding to different risk levels, such as Level 1 recording, Level 2 pop-up reminder, Level 3 security patrol, and Level 4 full-domain alarm linkage; Target risk strategy: an automated handling plan to be executed after matching the level.

[0046] For example, after completing the qualitative identification of the target object's risk, the system can retrieve the pre-set multi-level risk mapping rule library in the background, match the risk identification conclusion for this instance, and label the specific target risk level; then, it can retrieve the automated security handling strategy bound to that level to determine the final coordinated handling plan to be executed for this event, thus completing the entire closed loop from target detection, semantic analysis, risk identification to graded handling.

[0047] In the above implementation, after risk identification is completed, a preset risk level and corresponding handling strategy are matched to achieve automatic hierarchical management of incidents. This provides security personnel with clear handling priorities, avoids indiscriminate alarms that cause confusion in handling, realizes hierarchical closed-loop handling of security incidents, and improves the efficiency of security response in the park.

[0048] In the technical solution of this application embodiment, multi-view video multimodal detection data is collected simultaneously, and target detection is completed based on a preprocessing model. For targets with insufficient recognition credibility, a large visual language model is invoked to supplement semantic information, and the two types of data are integrated to comprehensively identify risks. This application solution takes into account both basic target recognition and complex event semantic analysis, breaks through the limitations of single monitoring analysis, and improves the comprehensiveness and accuracy of target risk identification in security scenarios.

[0049] In one optional implementation, the detection result includes detection boxes corresponding to each target object; such as Figure 1D As shown, step S120, which involves inputting the detection data of each target into the preprocessing model corresponding to each modality, detecting at least one preset class of target objects, and obtaining the detection results and confidence levels corresponding to each modality, may include: S121. Input the detection data of each target into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the target existence score of the detection box corresponding to each target object and the category probability group corresponding to each detection box.

[0050] The detection box can be a pixel coordinate frame determined by the preprocessing model based on the target detection data. For example, it can be composed of four sets of pixel values ​​(Xmin, Ymin, Xmax, Ymax) to accurately delineate the pixel range of the target object within the image. The target presence score can be a score given by the preprocessing model to the target object. For example, the value can be in the range of 0-1, used to characterize the correlation between the target object within the detection box and the preset object category. Correspondingly, the category probability set can be the sum of the probabilities of all preset target categories corresponding to a single detection box, with the sum of the probabilities of all categories within the same detection box being 1. For example, the category probability set corresponding to a certain detection box can be "Personnel: 0.98; Motor Vehicles: 0.01; Non-Motor Vehicles: 0.01". It can be seen that in this example, the target object selected by the detection box is highly likely to be a person.

[0051] For example, the YOLO preprocessing model parses multiple video frames one by one, and generates a dedicated pixel detection box for each suspected target in the frame. The model assigns a target presence score to each detection box, and calculates the probability that the detection box belongs to a category such as people or vehicles. These are then aggregated into category probability groups, and finally output three sets of associated data: a one-to-one detection box, a target presence score, and a category probability group.

[0052] S122. Determine the confidence level based on the target existence score and the maximum class probability in the class probability group.

[0053] Among them, the maximum class probability can be the probability value with the largest value in the class probability group of a single detection box, representing the highest probability that the model determines that the target belongs to a certain class; Confidence: The comprehensive effective confidence calculated in this step is the only judgment value for subsequent threshold comparison. The calculation formula can be: Confidence = Target Existence Score × Maximum Class Probability, with the value normalized to the 0-1 range.

[0054] For example, the edge-side YOLO preprocessing model first detects targets in the image, generates target pixel detection boxes, and simultaneously outputs the target presence score and the probability group of all categories corresponding to the detection boxes. Then, the system selects the optimal probability of each category, performs a weighted calculation based on the target presence confidence, and calculates a comprehensive confidence level that is compliant and comparable. This completes the standardization of the model detection score and provides accurate numerical basis for subsequent confidence level classification. This process is fully compatible with the confidence level calculation logic of small models in the technical solution.

[0055] In the above implementation, the confidence level is calculated by multiplying the target presence score by the maximum probability of the category, thus quantifying the reliability of the model's recognition. Unlike a single scoring method, this confidence level considers both the presence of a real target in the image and the accuracy of the target category determination. The quantification standard is unified and objective, providing a precise and stable basis for determining whether to initiate large-scale model inference.

[0056] In one alternative implementation, such as Figure 1E As shown, the step S130, which involves inputting the video detection results and video context information into a preset visual language model to obtain event semantic description information, may include: S131. Convert the video detection results according to the preset format to generate an image data packet.

[0057] The preset format can be a unified input format adapted to a large visual language model, such as image tensor format and standardized text format. The image data package can be a single-source encapsulated data package that integrates visual images and detects structured text, containing only the detection-related data of the aforementioned preprocessing model (e.g., YOLO).

[0058] In further alternative implementations, such as Figure 1F The step S131, which involves converting the video detection results according to a preset format to generate an image data packet, may include: S1311. Convert the image frames in the video detection results into input image frames according to the image tensor format supported by the visual language large model.

[0059] The image frame can be the original target image captured by a camera, cropped from a small YOLO model, or a partially cropped target image. The image tensor format can be a pixel matrix format specific to the large model; for example, a uniform resolution of 384×384 with pixel values ​​normalized to the 0-1 floating-point range. The input image frame can be a standardized image matrix that has been modified and can be directly read by the image encoder.

[0060] S1312. Convert the detection box coordinates, target category and confidence score obtained from the video detection results into text description information according to the preset standardized text format.

[0061] The standardized text format can be a pre-defined standard format used to convert the video detection results into text. It can include fixed fields such as pixel coordinates, predicted categories, and overall confidence levels. The text description information can be the result of text standardization, derived from the detection values, and is readable and parsable textual analysis content.

[0062] S1313. Pack the input image frame and text description information into an image data packet.

[0063] For example, the target image captured by the preprocessing model is first modified in terms of size, pixels, and encoding, and then converted into an input image frame that can be recognized by the video language large model tensor. Then, the target detection data is extracted and converted into standardized detection text according to the standardized text format. Finally, the modified image and standardized detection text are paired, bound, encrypted and encapsulated to generate a compliant image data package. The preprocessing model detection data adaptation and modification is completed from both image and text dimensions to ensure that there are no data format conflicts in subsequent multimodal fusion.

[0064] In the above implementation, image frames are converted into tensor formats supported by the model, and detection values ​​are converted into standard text and packaged into image data packets. This standardizes the visual and detection text data formats, eliminates the problem of messy original detection data formats, ensures data compatibility in subsequent multimodal fusion stages, and avoids errors in image and detection information parsing.

[0065] S132. Perform text structuring on the video context information to obtain a behavior data packet; wherein, the context information includes the time information corresponding to the video, the location information of the shooting location, and the historical behavior information of the target object.

[0066] Here, text structuring can refer to organizing fragmented and unstructured supplementary information into standard text with fixed fields and fixed sentence structures; correspondingly, the behavior data packet can be in the form of a data packet after text structuring. Time information can be the standard millisecond time of target detection, control period attributes, and cumulative dwell time of the target, etc.; location information can be the physical location of the camera, the area control level, and the pixel distance of the target relative to the fence boundary; historical behavior information can be the movement trajectory of the same target recently cached by edge nodes, past recognition behavior, and area traversal records, etc. The embodiments in this application are only examples and are not exhaustive.

[0067] S133. The image data packet and behavior data packet are fused to obtain a multimodal input dataset.

[0068] Among them, the multimodal input dataset can be a compliant dataset that combines image visual data and behavioral text data corresponding to spatiotemporal behavior, and can be directly adapted to large model interfaces.

[0069] S134. Input the multimodal input dataset into the visual language large model to obtain event semantic description information.

[0070] Among them, the semantic description information of the event can be standardized natural language text containing objective facts, risk predictions, and disposal suggestions.

[0071] For example, the format of the native video detection data output by YOLO is first modified to adapt to the image and text input requirements of the large model, and then image data packets are generated. Then, the fragmented native context information of the target time, location, and historical behavior is decomposed, standardized into standard text, and then encapsulated into behavior data packets. The two are then integrated to generate a multimodal input dataset that integrates images and text. Finally, the dataset is encrypted and transmitted to a pre-prepared visual language large model, which performs cross-modal reasoning and finally outputs complete event semantic description information. The entire data flow process is clearly layered and fully matches the low-confidence target large model review process of the technical solution.

[0072] In another alternative implementation, such as Figure 1G As shown, the process described in S134, which involves inputting the multimodal input dataset into a large visual language model to obtain event semantic description information, may include: S1341. Fill the multimodal input dataset into the pre-set prompt word engineering template to generate image-text input pairs that the visual language large model can recognize.

[0073] The prompt word engineering template can be a pre-customized prompt word template adapted to park security, including prompt word rules and other content. The multimodal input dataset is populated into this template to generate the corresponding prompt word engineering, which is used as input to the visual language large model. Correspondingly, the image-text input pair can be a prompt word engineering that includes both images and text. For example, it can be a combination of a one-to-one pair of tensor images and instruction prompt text, serving as the smallest inference unit of the video language large model.

[0074] S1342. The image encoder in the visual language large model parses the image in the image-text input pair and extracts the visual feature vectors corresponding to the outline of the target object, the scene environment, and the spatial position.

[0075] The image encoder can be a built-in visual encoding module of the VLM. The visual feature vector can be a digital image feature matrix, which can contain digital features such as the target shape, the environment, and the target's spatial orientation, and is used to characterize the features of the image in the text-image input pair.

[0076] S1343, the text encoder in the visual language large model performs word embedding on the text in the image-text input pair and extracts the text semantic feature vectors corresponding to time, region and behavior.

[0077] The text encoder can be a built-in text encoding module of VLM (such as Transformer); word embedding refers to the encoding algorithm that converts natural language into digital semantic vectors. The text semantic feature vector can be a digital text feature matrix, which can include digital features of text meaning corresponding to time period, controlled area, and historical behavior, and is used to characterize the features of Chinese characters in image-text input pairs.

[0078] S1344. Spatiotemporally align the visual feature vector and the text semantic feature vector to obtain the fused global features.

[0079] Spatiotemporal alignment can refer to the coordinated matching of attention modules in a large video language model, mapping visual features, spatiotemporal behavioral text features, and temporal and spatial dimensions one-to-one. Global features can be a feature matrix that fuses visual feature vectors and textual semantic feature vectors, which can include information across all dimensions such as image, time, location, and behavior. The fusion method can be direct concatenation or dynamic fusion through an attention mechanism; this application does not limit the specific method used.

[0080] S1345. The decoder in the visual language big model processes global features to obtain event semantic description information.

[0081] The decoder can be a built-in text decoding and generation module of VLM, used to decode global features and output event semantic description information.

[0082] For example, firstly, the integrated multimodal dataset is embedded into a security-specific prompt word engineering template to generate a text-image pairing inference unit (i.e., the aforementioned text-image input pair) that can be directly read by the video language big model; then, the image visual vector and the spatiotemporal behavioral text semantic vector are extracted by the image encoder and text encoder respectively; then, the spatiotemporal dimension alignment and fusion of text-image features are completed by the big model attention module to generate a global feature that is integrated across the entire domain; finally, the decoder combines the park security rule base to deduce and analyze the global feature, and decodes and outputs complete event semantic description information.

[0083] For example, firstly, the preprocessed standardized multimodal dataset is embedded into a security-specific customized prompt word engineering template. This template can be optimized based on a security scenario knowledge base, integrating multiple types of prompt modules such as scenario classification, target definition, behavior judgment, and spatiotemporal constraints. Through structured layered arrangement technology, it uniformly adapts and transforms multi-source heterogeneous data such as video images, scene text, behavior annotations, and environmental logs, generating standardized image-text input pairs that can be directly read and parsed by the video language large model. This achieves efficient transformation of unstructured security data into model-computable structured data, avoiding recognition failure problems caused by data heterogeneity and semantic bias from the source.

[0084] Subsequently, a dual-branch heterogeneous coding architecture can be adopted to complete the image and text feature decomposition and extraction. The image encoder can use an improved visual backbone network, combined with techniques such as inter-frame difference and optical flow tracking, to accurately extract the static appearance features and dynamic trajectory, posture changes, and other visual vectors of the park targets; the text encoder relies on a security-specific pre-trained model, and through semantic word segmentation and entity association parsing algorithms, mines semantic vectors such as target attributes, behavior types, and spatiotemporal locations in the data, to achieve decoupled and refined extraction of visual and text features, ensuring the integrity and effectiveness of feature information.

[0085] Based on the dual-branch extraction results, a large-model dynamic multi-head attention module is used to achieve spatiotemporal fusion of image and text features. This module abandons the fixed-weight fusion method and adopts a spatiotemporal adaptive matching algorithm to dynamically optimize feature association weights, achieving precise alignment between image spatial coordinates, frame temporal changes, and text semantics and spatiotemporal descriptions. Simultaneously, through feature denoising and information complementarity enhancement techniques, it compensates for the deficiencies of single-modal features, aggregating them to form a comprehensive, unified global feature set covering scene, target, behavior, and spatiotemporal associations.

[0086] Finally, a lightweight, dedicated decoder, in conjunction with an iterative park security rule base, performs layered analysis, logical verification, and intelligent deduction of global features. Based on event judgment criteria and scene association analysis algorithms, it achieves precise matching between features and security rules, filters invalid information, avoids misjudgments, deeply mines scene anomaly risks, and ultimately decodes and outputs complete structured semantic information containing event subject, location, time period, behavior process, and risk level, providing accurate and interpretable data support for park security early warning, event tracing, and intelligent management.

[0087] In the above implementation, the input content is standardized using prompt word templates. Semantic information is generated by decoding the extracted image and text features through dual encoders, spatiotemporally aligned and fused. This standardized large-scale model inference process enables cross-modal correlation analysis between visual images and spatiotemporal behavioral information, ensuring that event descriptions accurately reflect the actual situation and reducing the probability of semantic misjudgment.

[0088] In the above implementation, the video detection data and contextual information are standardized and encapsulated separately, then fused into a multimodal dataset before being input into the large model. This fully links image visual information with time, location, and historical behavioral textual information. This ensures that the input data to the large model has a unified format and complete information, enabling the large model to output a complete semantic description of the event based on all-dimensional information, thus improving the completeness of semantic analysis.

[0089] In one alternative implementation, the method may further include: In response to a video confidence level greater than or equal to a preset threshold, the risk of the target object is identified based on each detection result.

[0090] When the video confidence level is greater than or equal to the preset threshold, the YOLO edge small model detection results can be determined to be accurate and reliable. The detection results of the detection box, target category, and movement trajectory are directly adopted, and the risk identification is completed by directly matching the basic management and control rules of the park. It does not require triggering the VLM large model inference process, realizing the dual-link diversion judgment of high and low confidence levels, and taking into account both the real-time performance and accuracy of the system.

[0091] Example 2 Figure 2 This is a schematic diagram of a system architecture for multi-level visual model collaborative reasoning and risk classification in a security scenario, provided in Embodiment 2 of this application. This embodiment is a specific example provided based on the foregoing embodiments and implementation methods.

[0092] like Figure 2As shown, the system is divided into six layers from bottom to top: data acquisition layer, edge computing layer (lightweight processing and preliminary recognition), cloud-based large model layer (deep semantic understanding and reasoning), multi-model collaborative fusion and risk classification layer, application and linkage layer, and a support layer that runs horizontally throughout the entire system. Each layer contains several functional modules, which together constitute a complete intelligent video analysis and linkage response system.

[0093] The data acquisition layer is responsible for acquiring video or data streams. The edge computing layer comprises five parts: edge access, a lightweight visual model module, a result generation module, a confidence assessment module, and an edge output module. The lightweight visual model module integrates object detection models (such as YOLOv8), behavior recognition models (such as SlowFast), and attribute recognition models (such as wearable detection), outputting bounding boxes or categories, behavior types, and attribute information, respectively. The result generation module generates preliminary event results (i.e., initial results), while the confidence assessment module performs confidence calculation, threshold comparison, and dynamic threshold adjustment, outputting the confidence score and, based on the comparison results, providing low-risk or high-confidence events (with feature summaries, time or location information, and feature vectors) or trigger decisions (when the confidence score falls below the threshold). These preliminary processing results are ultimately output by the edge output module.

[0094] The cloud-based large model layer comprises a data receiving and preprocessing module, a large model inference module, a semantic understanding and inference module, and an inference result output module. The data receiving and preprocessing module receives detection results from the edge and performs feature alignment and enhancement, while constructing contextual information using information such as time, location, and historical trajectories. Subsequently, the large model inference module (containing a Visual Language Model (VLM) or a multimodal large model) passes the data to the semantic understanding and inference module, which performs semantic description generation, behavioral intent analysis, event causal inference, and risk clue mining. Finally, the inference result output module generates an event semantic description, risk assessment suggestions, and key evidence extraction, and returns the inference results.

[0095] The multi-model collaborative fusion and risk grading layer consists of a multi-model collaborative fusion module, a risk grading decision module, and an event management module. The multi-model collaborative fusion module is responsible for multi-source data access, result alignment, conflict resolution, and confidence-weighted fusion. The risk grading decision module performs information fusion, risk feature extraction, risk grading calculation, and risk rule matching, outputting the risk level. The event management module stores, retrieves, traces, and statistically analyzes event information and correlates it with event information. The application and linkage layer integrates access control systems, visualization platforms, perimeter radar, alarm centers, linkage control, alarm hosts, other IoT (Internet of Things) devices, mobile push notifications, and other business systems to execute final linkage control and alarm push notifications. The support layer includes a model management module (supporting model updates and version management), a data management module (covering data storage and data governance), a rule management module (with a built-in risk rule base), a user and permission management module, and a system operation and maintenance monitoring module, providing basic operation and maintenance and governance capabilities for the entire system.

[0096] In terms of data flow, each camera in the acquisition layer sends video or data streams down to the edge access module of the edge computing layer. The edge access module then passes these streams to the lightweight vision model module, where various models output bounding boxes or categories, behavior types, and attribute information. These results are processed in two ways: first, by the result generation module to generate preliminary event results; and second, by the confidence assessment module, which calculates confidence, compares thresholds, and adjusts dynamic thresholds to output the confidence level and corresponding decision results (high confidence includes feature summaries, time or location information, and feature vectors; low confidence triggers a decision). These preliminary event results and trigger decisions are then aggregated in the edge output module, which pushes the detection results up to the data receiving and preprocessing module of the cloud-based large model layer.

[0097] On the cloud, the data receiving and preprocessing module performs feature alignment and enhancement on the detection results, and constructs context information based on time, location, historical trajectories, etc., which then sequentially passes through the large model reasoning module and the semantic understanding and reasoning module to complete semantic description generation, behavior intention analysis, event causal reasoning and risk clue mining. The reasoning result output module finally generates event semantic descriptions, risk judgment suggestions and key evidence extraction, and transmits the reasoning results back to the lower layer. These reasoning results enter the multi-model collaborative fusion module of the multi-model collaborative fusion and risk grading layer, which executes multi-source data access, result alignment, conflict resolution and confidence weighted fusion; the fused data is sent to the risk grading decision module, which sequentially performs information fusion, risk feature extraction, risk grading calculation and risk rule matching to output a risk level. At the same time, relevant event information is also synchronized to the event management module for storage, retrieval, traceability and statistical analysis. Finally, the risk level output by the risk grading decision module and the event information in the event management module are transmitted downward to the application and linkage layer, triggering linkage responses of the access control system, visualization platform, perimeter radar, alarm center, linkage control, alarm host, other IoT devices, mobile terminal push and other business systems. Throughout the whole process, each management module in the support layer (model management, data management, rule management, user and permission management, system operation and maintenance monitoring) provides full-chain operation and maintenance, governance and rule support for data circulation and processing.

[0098] Third Embodiment Figure 3 is a schematic structural diagram of a risk identification apparatus for a security scenario provided in the third embodiment of the present application. As shown in Figure 3 , the apparatus 300 includes: A detection data acquisition module 310, configured to acquire target detection data of at least one modality in a target security scenario; wherein each modality at least includes a video modality, and correspondingly, the target detection data at least includes videos of different perspectives collected by a plurality of cameras; A detection result determination module 320, configured to input each target detection data into a preprocessing model corresponding to each modality, detect at least one preset type of target object, obtain a detection result and a confidence level corresponding to each modality, and video context information of the video detection result; A description information determination module 330, configured to, in response to the video confidence level corresponding to the video modality being less than a preset threshold, input the video detection result and the video context information into a preset vision-language large model to obtain event semantic description information; An object risk identification module 340, configured to identify the risk of the target object according to the detection result corresponding to each modality and the event semantic description information.

[0099] In the technical solution of this application embodiment, multi-view video multimodal detection data is collected simultaneously, and target detection is completed based on a preprocessing model. For targets with insufficient recognition credibility, a large visual language model is invoked to supplement semantic information, and the two types of data are integrated to comprehensively identify risks. This application solution takes into account both basic target recognition and complex event semantic analysis, breaks through the limitations of single monitoring analysis, and improves the comprehensiveness and accuracy of target risk identification in security scenarios.

[0100] In one optional implementation, the detection result includes a detection box corresponding to each target object; The detection result determination module 320 may include: The target object detection unit is used to input the target detection data into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the target existence score of the detection box corresponding to each target object and the category probability group corresponding to each detection box. The confidence determination unit is used to determine the confidence level based on the target existence score and the maximum class probability in the class probability group.

[0101] In one optional implementation, the description information determination module 330 may include: The image packet generation unit is used to convert the video detection results according to a preset format and generate image data packets; The behavior packet generation unit is used to perform text structuring processing on the video context information to obtain behavior data packets; wherein, the context information includes the time information corresponding to the video, the location information of the shooting location, and the historical behavior information of the target object; The dataset fusion unit is used to fuse image data packets and behavior data packets to obtain a multimodal input dataset; The event description determination unit is used to input multimodal input datasets into the visual language large model to obtain event semantic description information.

[0102] In one optional implementation, the image packet generation unit may include: The input image frame determination subunit is used to convert the image frames in the video detection results into input image frames according to the image tensor format supported by the large visual language model; The text description determination subunit is used to convert the detection box coordinates, target category and confidence score in the video detection results into text description information according to a preset standardized text format; The image data packet packaging subunit is used to package the input image frame and text description information into an image data packet.

[0103] In one optional implementation, the event description determination unit may include: The image-text input pair determination subunit is used to populate the multimodal input dataset into a pre-defined prompt word engineering template to generate image-text input pairs that the visual language large model can recognize. The visual vector determination subunit is used in the image encoder of the visual language large model to parse the image in the image-text input pair and extract the visual feature vectors corresponding to the outline of the target object, the scene environment and the spatial position. The text vector determination subunit is used in the text encoder of the visual language large model to perform word embedding on the text in the image-text input pair and extract the text semantic feature vectors corresponding to time, region and behavior. The global feature determination subunit is used to align the visual feature vector and the text semantic feature vector in a spatiotemporal manner to obtain the fused global features. The event semantic determination subunit is used by the decoder in the large visual language model to process global features and obtain event semantic description information.

[0104] In one optional implementation, the detection results corresponding to each mode include infrared thermal detection results and audio waveform detection results; Accordingly, the object risk identification module 340 may include: A heat source feature determination unit is used to determine the target heat source features of the target object based on the infrared thermal detection results. The acoustic feature determination unit is used to determine the target acoustic features of the target object based on the audio waveform detection results. The event semantic feature determination unit is used to convert the event semantic description information into event semantic features according to a preset vector embedding model; The weighted fusion unit is used to perform vector fusion of the target heat source features, the target acoustic wave features, and the event semantic features according to the confidence level and the pre-assigned weights to obtain a fused feature vector; wherein, the event semantic weight is greater than the heat source weight, and the heat source weight is greater than the acoustic wave weight; The risk score output unit is used to input the fused feature vector into a preset security risk fully connected mapping layer and output the risk score; The risk level matching unit is used to determine the risk level of the target object based on the risk score and a preset risk threshold range.

[0105] In one alternative embodiment, the device 300 may further include: The video reliability risk identification module is used to identify the risk of a target object based on the detection results when the video confidence level is greater than or equal to a preset threshold.

[0106] In one alternative embodiment, the device 300 may further include: The risk level determination unit is used to query the pre-stored correspondence between risk levels and risk identification results based on the target object's risk identification result, and obtain the target risk level corresponding to the target identification result. The risk strategy determination unit is used to determine the target risk strategy corresponding to the target risk level based on the pre-set processing strategy corresponding to the risk level.

[0107] The risk identification device for security scenarios provided in this application can execute the risk identification method for security scenarios provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects for executing the risk identification method for each security scenario.

[0108] Example 4 Figure 4 A schematic diagram of an electronic device 10, which can be used to implement embodiments of this application, is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0109] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0110] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0111] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as risk identification methods in security scenarios.

[0112] In some embodiments, the risk identification method for a security scenario can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the risk identification method for a security scenario described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the risk identification method for a security scenario by any other suitable means (e.g., by means of firmware).

[0113] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0114] Computer programs used to implement the methods of this application may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0115] In the context of this application, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium can be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0116] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0117] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0118] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0119] This application also discloses a computer program product, which includes a computer program that, when executed by a processor, implements the risk identification method for security scenarios provided in any embodiment of this application. This program product shares the same inventive concept as the risk identification methods for security scenarios disclosed in the embodiments of this application, and therefore will not be described in detail here.

[0120] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this application can be achieved, and this is not limited herein.

[0121] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A risk identification method for security scenarios, characterized in that, include: Acquire target detection data of at least one modality in a target security scene; wherein, each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras; The target detection data of each of the above targets are input into the preprocessing model corresponding to each of the above modalities, and at least one type of target object is detected to obtain the detection results and confidence scores corresponding to each of the above modalities, as well as the video context information of the video detection results. In response to the video confidence score corresponding to the video modality being less than a preset threshold, the video detection result and the video context information are input into a preset visual language large model to obtain event semantic description information; The risk of the target object is identified based on the detection results corresponding to each modality and the event semantic description information.

2. The method according to claim 1, characterized in that, The detection results include the detection boxes corresponding to each of the target objects; The step of inputting the target detection data into the preprocessing model corresponding to each modality, detecting at least one type of target object, and obtaining the detection results and confidence levels corresponding to each modality includes: The target detection data of each target is input into the preprocessing model corresponding to each modality, and at least one type of target object is detected to obtain the target presence score of the detection box corresponding to each target object and the category probability group corresponding to each detection box. The confidence level is determined based on the target existence score and the maximum class probability in the class probability group.

3. The method according to claim 1, characterized in that, The step of inputting the video detection results and the video context information into a preset visual language model to obtain event semantic description information includes: The video detection results are converted according to a preset format to generate an image data packet; The video context information is processed into a text structure to obtain a behavior data packet; wherein, the context information includes the time information corresponding to the video, the location information of the shooting location, and the historical behavior information of the target object; The image data packet and the behavior data packet are fused to obtain a multimodal input dataset; The multimodal input dataset is input into the visual language large model to obtain the event semantic description information.

4. The method according to claim 3, characterized in that, The step of converting the video detection results according to a preset format to generate an image data packet includes: The image frames in the video detection results are converted into input image frames according to the image tensor format supported by the large visual language model; The detection box coordinates, target category, and confidence score obtained from the video detection results are converted into text description information according to a preset standardized text format. The input image frame and the text description information are packaged into the image data packet.

5. The method according to claim 3, characterized in that, The step of inputting the multimodal input dataset into the visual language large model to obtain the event semantic description information includes: The multimodal input dataset is populated into a pre-defined prompt word engineering template to generate image-text input pairs that the visual language large model can recognize; The image encoder in the large visual language model parses the image in the image-text input pair and extracts the visual feature vectors corresponding to the outline of the target object, the scene environment, and the spatial position. The text encoder in the visual language large model performs word embedding on the text in the image-text input pair and extracts text semantic feature vectors corresponding to time, region and behavior. The visual feature vector and the text semantic feature vector are spatiotemporally aligned to obtain the fused global features; The decoder in the large visual language model processes the global features to obtain the event semantic description information.

6. The method according to claim 1, characterized in that, The detection results corresponding to each mode include infrared thermal detection results and audio waveform detection results; Accordingly, the step of identifying the risk of the target object based on the detection results corresponding to each modality and the event semantic description information includes: Based on the infrared thermal detection results, the target heat source characteristics of the target object are determined; Based on the audio waveform detection results, the target acoustic wave characteristics of the target object are determined; Based on a preset vector embedding model, the event semantic description information is transformed into event semantic features; The target heat source features, the target acoustic wave features, and the event semantic features are fused into a vector according to the confidence level and pre-assigned weights to obtain a fused feature vector; wherein the event semantic weight is greater than the heat source weight, and the heat source weight is greater than the acoustic wave weight. The fused feature vector is input into a preset security risk fully connected mapping layer, and a risk score is output. The risk level of the target object is determined based on the risk score and the preset risk threshold range.

7. The method according to claim 1, characterized in that, The method further includes: In response to the video confidence level being greater than or equal to the preset threshold, the risk of the target object is identified based on each of the detection results.

8. The method according to claim 1, characterized in that, After identifying the risk of the target object based on the detection results and the event semantic description information, the method further includes: Based on the target object's target identification result for risk identification, a query is performed in the pre-stored correspondence between risk levels and risk identification results to obtain the target risk level corresponding to the target identification result; Based on the pre-defined processing strategies corresponding to the risk levels, the target risk strategy corresponding to the target risk level is determined.

9. A risk identification device for security scenarios, characterized in that, include: The detection data acquisition module is used to acquire target detection data of at least one modality in the target security scene; wherein, each modality includes at least a video modality, and correspondingly, the target detection data includes at least videos from different perspectives captured by multiple cameras; The detection result determination module is used to input the detection data of each target into the preprocessing model corresponding to each modality, detect at least one type of target object, and obtain the detection result and confidence level corresponding to each modality, as well as the video context information of the video detection result; The description information determination module is used to input the video detection result and the video context information into a preset visual language large model in response to the video confidence corresponding to the video modality being less than a preset threshold, so as to obtain event semantic description information. The object risk identification module is used to identify the risk of the target object based on the detection results and the event semantic description information.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the risk identification method for security scenarios according to any one of claims 1-8.

Images