Intelligent safety early warning system and method based on multi-mode sensing fusion

Abstract

The invention discloses an intelligent safety early warning system and method based on multi-modal perception fusion. The system outputs a multi-target category, a target bounding box, confidence and a timestamp through a video intelligent identification subsystem; calculating an overlapping degree through a track space-time illegal behavior judgment subsystem, if the overlapping degree is greater than a first threshold value, counting the number of track points located outside the forbidden area in a preset time period, if the number of the track points is greater than or equal to a second threshold value, judging that the target category has an illegal behavior, and determining that the target category has the illegal behavior according to the confidence coefficient corresponding to each target category. Determining a target confidence degree corresponding to the illegal behavior; calculating a time-space confidence coefficient through a multi-modal fusion early warning subsystem, and calculating a multi-modal conflict index between different modal data; and if the multi-modal conflict index is lower than a preset threshold value, carrying out comprehensive risk score calculation, and determining an early warning level according to a comprehensive risk score calculation result. According to the invention, the accuracy of safety early warning can be improved, and real-time continuous safety early warning can be realized.

Description

Technical Field

[0001] This application relates to the field of industrial safety early warning technology, and in particular to an intelligent safety early warning system and method based on multimodal perception fusion. Background Technology

[0002] With the expansion of industrial production scale and the increasing complexity of working environments, construction sites, mining operations, and industrial parks are placing higher demands on the real-time performance and accuracy of safety monitoring and risk early warning. Existing safety monitoring systems still have the following shortcomings in practical applications: (1) High dependence on computing and communication conditions. Some security monitoring systems need to transmit video data to remote computing nodes for analysis and processing, which is easily affected by network bandwidth, latency and stability, making it difficult to achieve timely and continuous security warnings under complex network conditions. (2) Insufficient multi-source sensing data collaboration capability. In existing systems, multiple sensing means such as video surveillance, environmental sensors and positioning devices often operate independently, lacking a unified spatiotemporal correlation and fusion analysis mechanism, making it difficult to comprehensively evaluate personnel behavior and environmental status, thus affecting the accuracy of overall security situation judgment. (3) The reliability of violation behavior judgment needs to be improved. Traditional behavior detection methods based on regional rules usually judge based on only a single frame or instantaneous state, which easily misjudges targets that are initially located in the monitoring area as violations; at the same time, there is a lack of effective trajectory history analysis and state management mechanism for violations with temporal continuity, resulting in a high false alarm rate and making it difficult to achieve graded and continuous security warnings. Summary of the Invention

[0003] This application aims to propose an intelligent security early warning system and method based on multimodal perception fusion, which can improve the accuracy of security early warning and realize real-time and continuous security early warning.

[0004] In a first aspect, embodiments of this application provide an intelligent security early warning system based on multimodal perception fusion, the system comprising: The video intelligent recognition subsystem is used to decode the input video stream data into raw image frames, preprocess the raw image frames to obtain preprocessed images, perform neural network inference and post-processing on the preprocessed images, and output multiple target categories, target bounding boxes corresponding to each of the multiple target categories, confidence scores corresponding to each of the multiple target categories, and timestamps corresponding to each of the multiple target categories. A trajectory spatiotemporal violation determination subsystem is used to perform multi-target tracking on the multi-target categories. Based on the timestamp, a trajectory history queue arranged in chronological order is constructed for each target category. The trajectory history queue includes the trajectory point coordinates of each target category within a preset time period. Based on the target bounding box and prohibited area corresponding to each target category, the overlap is calculated. If the overlap is greater than a first threshold, the number of trajectory points located outside the prohibited area within the preset time period is counted based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to a second threshold, the target category is determined to have a violation. Based on the confidence level corresponding to each target category, the target confidence level corresponding to the violation is determined. A multimodal fusion early warning subsystem is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data, when determining that the target category has violated regulations. The visual structured data includes the violation and the target confidence level corresponding to the violation. Based on the visual structured data, the environmental sensor data, and the positioning data, a spatiotemporal confidence level is calculated. Based on the spatiotemporal confidence level, a multimodal conflict index between the different modal data is calculated. If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score is calculated based on the target confidence level, and an early warning level is determined based on the comprehensive risk score calculation result.

[0005] Compared with the prior art, the first aspect of this application has the following beneficial effects: This method achieves joint analysis of target behavior and environmental status by collaborating among a video intelligent recognition subsystem, a trajectory spatiotemporal violation determination subsystem, and a multimodal fusion early warning subsystem. It outputs a security situation assessment result and corresponding early warning response without requiring video stream data to be transmitted to a remote computing node for analysis, enabling real-time and continuous security early warning. By comprehensively considering the target bounding box, restricted area, and trajectory point coordinates for each target category, the trajectory spatiotemporal violation determination subsystem accurately determines whether a target category exhibits violation behavior. By fusing visual structured data, environmental sensor data, and positioning data, spatiotemporal confidence is calculated. Based on this confidence, a multimodal conflict index between different modalities is calculated, followed by a comprehensive risk score calculation. The early warning level is then determined based on the comprehensive risk score, improving the accuracy of security early warning and enabling the multimodal fusion early warning subsystem to provide accurate and continuous real-time security warnings.

[0006] In some embodiments, the video intelligent recognition subsystem includes: The video decoding unit is used to decode the video stream data through a hardware decoder to obtain the original image frames; An image preprocessing unit is used to perform preprocessing operations on the original image frame to obtain a preprocessed image; The neural network inference unit is used to input the preprocessed image into the neural network model to obtain the target detection result, which includes multiple category results, the confidence score and bounding box corresponding to each category result; The post-detection processing unit is used to perform non-maximum suppression and confidence filtering on the target detection results to remove redundant bounding boxes and retain high-confidence categories, thereby obtaining multiple target categories, the confidence scores corresponding to each of the multiple target categories, and the target bounding boxes corresponding to each of the multiple target categories. An adaptive adjustment unit is used to adjust the sampling interval of the original image frame by the image preprocessing unit; The result output unit is used to output the multi-object category, the target bounding box corresponding to each of the multi-object categories, the confidence level corresponding to each of the multi-object categories, and the timestamp corresponding to each of the multi-object categories to the trajectory spatiotemporal violation determination subsystem.

[0007] In some implementations, the trajectory spatiotemporal violation determination subsystem includes: A multi-target tracking module is used to perform cross-frame tracking of the multiple target categories and assign a unique tracking identifier to each target category; The trajectory history maintenance module is used to construct a trajectory history queue arranged in chronological order for each target category based on the timestamp. The trajectory history queue is used to record the trajectory point coordinates, timestamps, and attribute status of each target category in a preset time period. The attribute status includes the unique tracking identifier of each target category and whether it is in a restricted area. The regional intrusion detection module is used to calculate the overlap between the current target bounding box of the target category and the preset restricted area. The overlap is the ratio of the intersection area of ​​the target bounding box and the restricted area to the total area of ​​the target bounding box. The real intrusion determination module is used to: if the overlap is greater than the first threshold, count the number of trajectory points located outside the restricted area within the preset time period based on the trajectory point coordinates; if the number of trajectory points is greater than or equal to the second threshold, determine that the target category has violated regulations, and determine the target confidence level corresponding to the violation based on the confidence level corresponding to each target category; if the number of trajectory points is less than the second threshold, determine that the target category does not violate regulations and does not trigger an alarm. The violation status management module is used to manage the status of the violation based on a finite state machine. The violation includes area intrusion, failure to wear security protective equipment, and abnormal gathering of people. The status includes normal status, first violation status, and continuous violation status. If the target category is judged to be in violation for the first time, an alarm is triggered and the target category enters the first violation status. If the target category continues to violate the rules and the time since the last alarm exceeds a preset reminder interval, an alarm is triggered again and the continuous violation status is maintained. If the target category leaves the restricted area or the trajectory of the target category has not been updated for a long time, the status is reset to normal status.

[0008] In some embodiments, the multimodal fusion early warning subsystem includes: The data integration module is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data. The spatiotemporal confidence calculation module is used to calculate the spatiotemporal confidence based on the visual structured data, the environmental sensor data, and the positioning data. The multimodal consistency analysis module is used to calculate the multimodal conflict index between the different modal data based on the spatiotemporal confidence level. The comprehensive risk score calculation module is used to calculate a comprehensive risk score based on the target confidence level if the multimodal conflict index is lower than a preset threshold. The early warning response module is used to determine the early warning level based on the comprehensive risk score calculation results, and to trigger the corresponding early warning response based on the early warning level.

[0009] In some implementations, calculating spatiotemporal confidence based on the visual structured data, the environmental sensor data, and the positioning data includes: Based on a time sliding window, the historical baselines corresponding to the visual structured data, the environmental sensor data, and the positioning data are calculated respectively. Initial confidence weights are assigned to the visual structured data, the environmental sensor data, and the positioning data, respectively. Based on the historical benchmark and the initial confidence weight, the spatiotemporal confidence scores corresponding to the visual structured data, the environmental sensor data, and the positioning data are calculated respectively.

[0010] In some implementations, calculating the multimodal conflict index between the different modal data based on the spatiotemporal confidence includes: ; in, This indicates the number of modal data types involved in the fusion. Indicates the multimodal conflict index. Indicates the first Spatiotemporal confidence of modal data at the current moment Indicates that within the same spatiotemporal window, the first The strength of evidence value for class modal data that negates or weakens the violation determined by the visual modal data.

[0011] In some implementations, the step of calculating a comprehensive risk score based on the target confidence level includes: ; in, This represents the overall risk score. This indicates the number of modal data types involved in the fusion. Indicates the first The spatiotemporal confidence of modal data at the current moment. Indicates the first The evidence strength value of the modal data at the current moment, wherein the evidence strength value is equal to the target confidence level.

[0012] Secondly, embodiments of this application also provide a multimodal perception fusion-based intelligent security early warning method, the method being applied to the multimodal perception fusion-based intelligent security early warning system, comprising: The video stream data is decoded into raw image frames, and the raw image frames are preprocessed to obtain a preprocessed image. The preprocessed image is input into a neural network model to obtain target detection results. The target detection results are then post-processed to obtain multiple target categories, target bounding boxes corresponding to each of the multiple target categories, confidence scores corresponding to each of the multiple target categories, and timestamps corresponding to each of the multiple target categories. Based on the timestamp, a trajectory history queue arranged in chronological order is constructed for each target category. The trajectory history queue includes the coordinates of trajectory points for each target category within a preset time period. Based on the target bounding box and restricted area corresponding to each target category, the overlap is calculated. If the overlap is greater than a first threshold, the number of trajectory points located outside the restricted area within the preset time period is counted based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to a second threshold, the target category is determined to have a violation. Based on the confidence level corresponding to each target category, the target confidence level corresponding to the violation is determined. When determining that the target category has a violation, different modal data including visual structured data, environmental sensor data and positioning data are acquired. The visual structured data includes the violation and the target confidence level corresponding to the violation. Based on the visual structured data, the environmental sensor data, and the positioning data, calculate the spatiotemporal confidence score; Based on the spatiotemporal confidence level, calculate the multimodal conflict index between the different modal data; If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score is calculated based on the target confidence level, and the warning level is determined based on the comprehensive risk score calculation result.

[0013] Thirdly, embodiments of this application also provide an electronic device, including at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, the instructions being executed by the at least one control processor to enable the at least one control processor to perform a multimodal perception fusion intelligent security early warning method as described above.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium storing computer-executable instructions for causing a computer to execute a multimodal perception fusion intelligent security early warning method as described above.

[0015] It is understood that the beneficial effects of the second to fourth aspects compared with the related technologies are the same as the beneficial effects of the first aspect compared with the related technologies. Please refer to the relevant description in the first aspect above, which will not be repeated here. Attached Figure Description

[0016] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the description of the embodiments taken in conjunction with the following drawings, in which: Figure 1 This is a schematic diagram of the structure of an embodiment of the intelligent safety early warning system based on multimodal perception fusion provided in this application; Figure 2 This is a schematic diagram of the structure of the intelligent security early warning system based on visual recognition and multimodal perception fusion in the preferred embodiment of the multimodal perception fusion intelligent security early warning system provided in this application; Figure 3 This is a schematic diagram of the structure of the multimodal fusion early warning subsystem in the preferred embodiment of the intelligent security early warning system based on multimodal perception fusion provided in this application; Figure 4 This is a flowchart illustrating an embodiment of the intelligent security early warning method based on multimodal perception fusion provided in this application; Figure 5 This is a schematic diagram of the structure of an embodiment of the electronic device provided in this application. Detailed Implementation

[0017] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application.

[0018] In the description of this application, the use of terms such as "first," "second," etc., is for the purpose of distinguishing technical features only and should not be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated or the order of the technical features indicated.

[0019] In the description of this application, it should be understood that the orientation descriptions, such as up, down, etc., are based on the orientation or positional relationship shown in the accompanying drawings, and are only for the convenience of describing this application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0020] In the description of this application, it should be noted that, unless otherwise explicitly defined, terms such as "setup," "installation," and "connection" should be interpreted broadly, and those skilled in the art can reasonably determine the specific meaning of the above terms in this application in conjunction with the specific content of the technical solution.

[0021] To address the issues of insufficient real-time performance, fragmented multi-source sensing data, high misjudgment rate of violations, and incomplete safety situation assessment in existing safety monitoring systems, and to achieve comprehensive perception, accurate assessment, and graded early warning of personnel behavior and environmental risks in complex work scenarios, this application proposes an intelligent safety early warning system and method based on multimodal perception fusion.

[0022] Reference Figure 1 This application provides a multimodal perception fusion intelligent safety early warning system, which includes a video intelligent recognition subsystem 101, a trajectory spatiotemporal violation determination subsystem 102, and a multimodal fusion early warning subsystem 103, wherein: The video intelligent recognition subsystem 101 is used to decode the input video stream data into raw image frames, preprocess the raw image frames to obtain preprocessed images, perform neural network inference and postprocessing on the preprocessed images, and output multiple target categories, target bounding boxes corresponding to each of the multiple target categories, confidence scores corresponding to each of the multiple target categories, and timestamps corresponding to each of the multiple target categories. The trajectory spatiotemporal violation determination subsystem 102 is used for multi-target tracking of multiple target categories. Based on the timestamp, it constructs a trajectory history queue arranged in chronological order for each target category. The trajectory history queue includes the trajectory point coordinates of each target category within a preset time period. Based on the target bounding box and prohibited area corresponding to each target category, it calculates the overlap. If the overlap is greater than a first threshold, it counts the number of trajectory points located outside the prohibited area within the preset time period based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to a second threshold, it determines that the target category has a violation. Based on the confidence level corresponding to each target category, it determines the target confidence level corresponding to the violation. The multimodal fusion early warning subsystem 103 is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data, when determining that a target category has violated regulations. The visual structured data includes the violation and the target confidence level corresponding to the violation. Based on the visual structured data, environmental sensor data, and positioning data, the system calculates the spatiotemporal confidence level. Based on the spatiotemporal confidence level, the system calculates the multimodal conflict index between different modal data. If the multimodal conflict index is lower than a preset threshold, the system calculates a comprehensive risk score based on the target confidence level and determines the early warning level based on the comprehensive risk score calculation result.

[0023] In this embodiment, the video intelligent recognition subsystem decodes the input video stream data into raw image frames, preprocesses the raw image frames to obtain preprocessed images, and performs neural network inference and post-processing on the preprocessed images to output multiple target categories, the target bounding boxes corresponding to each target category, the confidence scores corresponding to each target category, and the timestamps corresponding to each target category. The trajectory spatiotemporal violation determination subsystem performs multi-target tracking for the multiple target categories. Based on the timestamps, a trajectory history queue arranged in chronological order is constructed for each target category. The trajectory history queue includes the trajectory point coordinates of each target category within a preset time period. The overlap is calculated based on the target bounding boxes and prohibited areas corresponding to each target category. If the overlap is greater than a first threshold, the position of each target within the preset time period is statistically analyzed based on the trajectory point coordinates. If the number of trajectory points outside the restricted area is greater than or equal to a second threshold, the target category is determined to have violated regulations. Based on the confidence level corresponding to each target category, the target confidence level corresponding to the violation is determined. When determining that a target category has violated regulations, the multimodal fusion early warning subsystem acquires different modal data, including visual structured data, environmental sensor data, and positioning data. The visual structured data includes the violation and the corresponding target confidence level. Based on the visual structured data, environmental sensor data, and positioning data, a spatiotemporal confidence level is calculated. Based on the spatiotemporal confidence level, a multimodal conflict index between different modal data is calculated. If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score is calculated based on the target confidence level, and the early warning level is determined based on the comprehensive risk score calculation result.

[0024] Thus, through the collaborative work of the video intelligent recognition subsystem, the trajectory spatiotemporal violation determination subsystem, and the multimodal fusion early warning subsystem, a joint analysis of target behavior and environmental status is achieved, outputting security situation assessment results and corresponding early warning responses. Furthermore, video stream data does not need to be transmitted to remote computing nodes for analysis and processing, enabling real-time and continuous security early warnings. By comprehensively considering the target bounding box, restricted area, and trajectory point coordinates corresponding to each target category, the trajectory spatiotemporal violation determination subsystem can accurately determine whether a target category has violated regulations. By fusing visual structured data, environmental sensor data, and positioning data, spatiotemporal confidence is calculated. Based on the spatiotemporal confidence, a multimodal conflict index between different modalities is calculated, followed by a comprehensive risk score calculation. Based on the comprehensive risk score calculation result, the early warning level is determined, improving the accuracy of security early warnings. This allows the multimodal fusion early warning subsystem to provide accurate and real-time continuous security early warnings.

[0025] The above-mentioned decoding of the input video stream data into raw image frames can be achieved using a hardware decoder. The aforementioned hardware decoder is a type of decoder known to those skilled in the art, and will not be specifically described in this embodiment.

[0026] The preprocessing of the original image frame described above can include image scaling, color format conversion, and pixel normalization.

[0027] The aforementioned neural network inference and post-processing of the preprocessed image can be performed using a neural network model to infer the target detection result, and then non-maximum suppression and confidence filtering are applied to the target detection result to remove redundant detection boxes and retain high-confidence target categories. The neural network model used above is a neural network model known to those skilled in the art, and this embodiment does not provide a specific description or limitation. The aforementioned non-maximum suppression and confidence filtering both employ techniques known to those skilled in the art, and this embodiment does not provide a specific description.

[0028] The aforementioned multi-objective category can include multiple different operators.

[0029] The above method of counting the number of trajectory points located outside the restricted area within a preset time period based on the coordinates of the trajectory points can be used to count the number of coordinates of the trajectory points located outside the restricted area within the preset time period, and then using the number of coordinates of the trajectory points as the number of trajectory points.

[0030] The above method of determining the target confidence level corresponding to the violation based on the confidence level corresponding to each target category can be achieved by using the confidence level of the last frame image within a preset time period corresponding to each target category as the target confidence level corresponding to the violation.

[0031] In some implementations, the video intelligent recognition subsystem includes: The video decoding unit is used to decode the video stream data through a hardware decoder to obtain the original image frames; The image preprocessing unit is used to perform preprocessing operations on the original image frame to obtain the preprocessed image. The neural network inference unit is used to input the preprocessed image into the neural network model to obtain the target detection result, which includes multiple category results, the confidence score and bounding box for each category result; The post-detection processing unit is used to perform non-maximum suppression and confidence filtering on the target detection results to remove redundant bounding boxes and retain high-confidence categories, thereby obtaining multiple target categories, the confidence scores of each target category, and the target bounding boxes of each target category. An adaptive adjustment unit is used to adjust the sampling interval of the image preprocessing unit for the original image frames; The result output unit is used to output the multi-object category, the target bounding box corresponding to each multi-object category, the confidence level corresponding to each multi-object category, and the timestamp corresponding to each multi-object category to the trajectory spatiotemporal violation determination subsystem.

[0032] In this embodiment, by adjusting the sampling interval of the original image frame by the image preprocessing unit, the problem of limited computing power of edge computing devices can be solved, and there is no need to transmit the video stream data to a remote computing node for analysis and processing, thus enabling real-time and continuous security warnings.

[0033] The timestamps above represent the time points at which the target category is output.

[0034] In some implementations, the trajectory spatiotemporal violation determination subsystem includes: The multi-target tracking module is used to track multiple target categories across frames and assign a unique tracking identifier to each target category; The trajectory history maintenance module is used to build a trajectory history queue arranged in chronological order for each target category based on the timestamp. The trajectory history queue is used to record the trajectory point coordinates, timestamps and attribute status of each target category in a preset time period. The attribute status includes the unique tracking identifier of each target category and whether it is in a restricted area. The region intrusion detection module is used to calculate the overlap between the current target bounding box of the target category and the preset restricted area. The overlap is the ratio of the intersection area of ​​the target bounding box and the restricted area to the total area of ​​the target bounding box. The real intrusion determination module is used to count the number of trajectory points located outside the restricted area within a preset time period based on the trajectory point coordinates if the overlap is greater than the first threshold; if the number of trajectory points is greater than or equal to the second threshold, the target category is determined to have violated the rules, and the target confidence level corresponding to the violation is determined based on the confidence level corresponding to each target category; if the number of trajectory points is less than the second threshold, the target category is determined not to have violated the rules, and no alarm is triggered. The violation status management module is used to manage the status of violations based on a finite state machine. Violations include area intrusion, failure to wear security equipment, and abnormal gathering of people. The status includes normal status, first violation status, and continuous violation status. If a target category is judged to be in violation for the first time, an alarm is triggered and the target category enters the first violation status. If the target category continues to violate the rules and the time since the last alarm exceeds the preset reminder interval, an alarm is triggered again and the continuous violation status is maintained. If the target category leaves the restricted area or the target category's trajectory has not been updated for a long time, the status is reset to normal status.

[0035] In this embodiment, the overlap is calculated by the regional intrusion detection module, and then the real intrusion determination module determines the real violation behavior based on the overlap and determines the target confidence level corresponding to the violation behavior. This can lay a good data foundation for the multimodal fusion early warning subsystem and enable the trajectory spatiotemporal violation behavior determination subsystem to accurately determine whether the target category has a violation behavior.

[0036] The aforementioned unique tracking identifier is used to accurately identify and track target categories.

[0037] The aforementioned finite state machine can be an abstract mathematical model used to describe the behavior of a system switching between a finite number of states according to rules.

[0038] In some implementations, the multimodal fusion early warning subsystem includes: The data integration module is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data. The spatiotemporal confidence calculation module is used to calculate spatiotemporal confidence based on visual structured data, environmental sensor data, and positioning data. The multimodal consistency analysis module is used to calculate the multimodal conflict index between different modal data based on spatiotemporal confidence. The comprehensive risk score calculation module is used to calculate a comprehensive risk score based on the target confidence level if the multimodal conflict index is lower than a preset threshold. The early warning response module is used to determine the early warning level based on the comprehensive risk score calculation results, and to trigger the corresponding early warning response based on the early warning level.

[0039] In this embodiment, spatiotemporal confidence is calculated by fusing visual structured data, environmental sensor data, and positioning data. Based on the spatiotemporal confidence, a multimodal conflict index between different modalities is calculated. Then, a comprehensive risk score is calculated based on the target confidence, and the warning level is determined according to the comprehensive risk score, triggering the corresponding warning response. This improves the accuracy of safety warnings, enabling the multimodal fusion warning subsystem to provide accurate and continuous real-time safety warnings.

[0040] The environmental sensor data mentioned above can be collected by integrating multiple sensors, including parameters such as carbon monoxide, methane, hydrogen sulfide, oxygen, PM2.5 concentration, and ambient temperature and humidity.

[0041] The aforementioned positioning data can be location data collected by Global Navigation Satellite System (GNSS) positioning equipment.

[0042] The above-mentioned determination of the warning level based on the comprehensive risk score calculation results can be achieved by pre-setting multiple risk judgment thresholds and dividing the risk judgment thresholds into at least two risk intervals. If the comprehensive risk score falls into the corresponding risk interval, the warning level corresponding to the comprehensive risk score is determined based on the risk interval into which the comprehensive risk score falls.

[0043] In some implementations, spatiotemporal confidence is calculated based on visual structured data, environmental sensor data, and positioning data, including: Based on a time sliding window, the historical benchmarks corresponding to the computer vision structured data, environmental sensor data, and positioning data are respectively used. Initial confidence weights are assigned to visual structured data, environmental sensor data, and positioning data respectively; Based on historical benchmarks and initial confidence weights, the spatiotemporal confidence scores of visual structured data, environmental sensor data, and positioning data are calculated respectively.

[0044] In this embodiment, spatiotemporal confidence is calculated based on visual structured data, environmental sensor data, and positioning data. When a certain modality data source experiences abnormal fluctuations, instability, or reduced confidence, its influence weight in subsequent fusion decisions will be automatically weakened, thereby improving the reliability and robustness of the overall security risk assessment results.

[0045] In some implementations, a multimodal conflict index is calculated based on spatiotemporal confidence, including: ; in, This indicates the number of modal data types involved in the fusion. Indicates the multimodal conflict index. Indicates the first Spatiotemporal confidence of modal data at the current moment Indicates that within the same spatiotemporal window, the first The strength of evidence value for class modal data that negates or weakens the violation determined by the visual modal data.

[0046] In this embodiment, by calculating the multimodal conflict index between different modal data based on spatiotemporal confidence, a solid data foundation can be laid for subsequent comprehensive risk score calculation. If the multimodal conflict index is greater than or equal to a preset threshold, it indicates that the current modal data is unreliable, and it is necessary to reacquire or remove the current modal data. If the multimodal conflict index is less than the preset threshold, it indicates that the current modal data is reliable, and comprehensive risk score calculation can be performed to improve the reliability and accuracy of the comprehensive risk score calculation.

[0047] In some implementations, a comprehensive risk score is calculated based on the target confidence level, including: ; in, This represents the overall risk score. This indicates the number of modal data types involved in the fusion. Indicates the first The spatiotemporal confidence of modal data at the current moment. Indicates the first The evidence strength value of the modal data at the current moment is equal to the target confidence level.

[0048] In this embodiment, the first Evidence strength value corresponding to modal data This indicates the degree to which the modal data supports the existence of a security risk at the current moment, and is used in the calculation of the comprehensive risk score; Conflict evidence strength value corresponding to modal data This indicates the degree to which the modal data negates or weakens the violation determined by the visual structured data within the same spatiotemporal window, and is used for multimodal conflict consistency analysis. Both can be obtained from the same modal data through different evaluation functions, but they act on different data fusion stages and are independent in function and technical effect, thus avoiding misleading the comprehensive risk assessment results due to anomalies in a single modal data.

[0049] In this embodiment, calculating an accurate comprehensive risk score can lay a good data foundation for determining the accurate early warning level and triggering the accurate early warning response in the later stage, thereby improving the accuracy of safety early warning.

[0050] To facilitate understanding by those skilled in the art, a set of preferred embodiments is provided below: To address the issues of insufficient real-time performance, fragmented multi-source sensing data, high false positive rates for violations, and incomplete security situation assessments in existing security monitoring systems, this paper refers to... Figure 2 This embodiment proposes an intelligent security early warning system based on the fusion of visual recognition and multimodal perception. This system can be deployed on edge computing devices and comprises the following subsystems: a video intelligent recognition subsystem, a trajectory spatiotemporal violation determination subsystem, and a multimodal fusion early warning subsystem. These subsystems work collaboratively to perform joint analysis of target behavior and environmental conditions, and output security situation assessment results and corresponding early warning responses.

[0051] Specifically, this intelligent safety early warning system is configured to be deployed on edge computing devices (such as embedded industrial computers equipped with GPUs / NPUs) and connected to front-end sensing devices via a local area network. The system can complete the entire closed-loop process without relying on cloud servers. The system mainly includes three core subsystems: a video intelligent recognition subsystem, a trajectory spatiotemporal violation determination subsystem, and a multimodal fusion early warning subsystem.

[0052] System hardware environment: In this embodiment, the edge computing device connects to the following peripherals via an Ethernet interface or an industrial bus: Visual sensing equipment: High-definition network cameras (IPCs) or PTZ cameras, used to acquire video stream data in real time.

[0053] Environmental sensing equipment: Integrated gas sensor array (monitoring) , , as well as ), temperature and humidity sensors, and PM2.5 dust sensors.

[0054] Positioning device: GNSS receiver, used to provide the device's own absolute geographic coordinates.

[0055] Detailed descriptions of each subsystem include: 1. A video intelligent recognition subsystem is used to parse and process the input video stream data and output structured visual information related to target detection. To address the limitation of computing power on edge computing devices, this subsystem designs an end-to-end parallel processing pipeline. The video intelligent recognition subsystem specifically includes: The video decoding unit is used to efficiently decode the input video stream using a hardware decoder to obtain the original image frames.

[0056] The image preprocessing unit is used to perform preprocessing operations on the original image frame using the hardware acceleration unit. The preprocessing operations include image scaling, color format conversion, and pixel normalization. The hardware acceleration unit can be an acceleration unit known to those skilled in the art, and this embodiment does not specifically limit or describe it.

[0057] The neural network inference unit is used to input the preprocessed image into the neural network inference engine (i.e., the neural network model) to obtain the object detection result. The neural network inference unit also employs a pipelined parallel processing architecture, performing preprocessing on the next frame's image while executing object detection inference for the current frame, thereby improving processing throughput. The pipelined parallel processing architecture is a highly efficient computational model that achieves parallel processing by decomposing complex computational processes into multiple independent sub-processes, thus improving computational efficiency and resource utilization.

[0058] The post-detection processing unit performs nonmaximum suppression and confidence filtering on the inference results (i.e., target detection results) to remove redundant detection boxes and retain high-confidence targets.

[0059] The adaptive adjustment unit is used to dynamically adjust the sampling interval of the image preprocessing unit for video frames (i.e., original image frames) according to the real-time load status of the system, so as to balance processing efficiency under the condition of limited computing resources.

[0060] The result output unit is used to output the structured detection results to the trajectory spatiotemporal violation judgment subsystem. The structured detection results include the target bounding box, target category, timestamp, and confidence level.

[0061] 2. The trajectory spatiotemporal violation determination subsystem receives the structured detection results output by the video intelligent recognition subsystem, performs multi-target tracking and maintains corresponding trajectory history information, and analyzes target behavior based on preset spatiotemporal rules to determine whether any violations exist. This subsystem solves the problems of high false alarm rates associated with traditional "area intrusion" detection and failure to wear security equipment. Specifically, the trajectory spatiotemporal violation determination subsystem includes: The multi-target tracking module is used to track target categories across frames in the structured detection results and assign a unique tracking identifier to each target category.

[0062] The trajectory history maintenance module is used to maintain a trajectory history queue arranged chronologically for each target category. It records the spatial location (i.e., trajectory point coordinates, obtained using existing technology, which will not be specifically described in this embodiment), timestamp, and attribute status of the target category within consecutive frames over a preset time period. The attribute status includes a unique tracking identifier for each target category and whether it is within a restricted area. This trajectory history queue includes the spatial location of each tracked target within the preset time period.

[0063] The region intrusion detection module is used to calculate the overlap between the current target bounding box and the preset restricted area for the target category. The overlap is the ratio of the intersection area of ​​the target bounding box and the restricted area to the total area of ​​the target bounding box.

[0064] The real intrusion determination module is used to traverse the trajectory history queue of the target category when the overlap exceeds a first threshold, count the number of trajectory points located outside the restricted area, and obtain the number of trajectory points by counting the coordinates of trajectory points outside the restricted area. If the number of external trajectory points is not less than a second threshold, it is determined as a real intrusion behavior (i.e., violation) from the outside in; otherwise, it is considered as initially located within the area (not a violation), and no alarm is triggered. It should be noted that the first threshold and the second threshold in this embodiment can be set based on historical experience, and this embodiment does not impose specific limitations.

[0065] The violation status management module is used to manage the status of violations based on a finite state machine. The statuses include normal, initial violation, and persistent violation. When a target category is first identified as violating a rule, an alarm is triggered and the system enters the initial violation status. If the target category continues to violate the rule and the time since the last alarm exceeds a preset reminder interval, an alarm is triggered again, and the persistent violation status is maintained. When the target category leaves the restricted area or its trajectory has not been updated for an extended period, the status is reset to normal. Violations include: area intrusion, failure to wear security equipment, and abnormal gathering of people.

[0066] Based on the historical trajectory queue corresponding to the violation, the confidence level of the last frame image within a preset time period (i.e., the confidence level output by the video intelligent recognition subsystem) is obtained, and this confidence level is used as the target confidence level for the violation. Taking the absence of protective equipment as an example, the determination of a genuine intrusion differs from that of not wearing protective equipment. The determination of not wearing protective equipment is based on the recognition result of a single image frame, so the video intelligent recognition subsystem has one confidence level for each frame. Intrusion, however, is the recognition result within the historical trajectory queue for that specific time period. While there may be multiple recognitions of a person within this time period, there is only one recognition result for an intrusion event. Therefore, when a genuine intrusion event is determined, the confidence level of the last frame image within the preset time period is used as the target confidence level for the violation. This target confidence level is used in the multimodal fusion early warning subsystem to calculate the multimodal conflict index and comprehensive risk score.

[0067] The trajectory cleaning module is used to remove trajectory data that has exceeded a preset lifespan since the last update at the end of each frame processing cycle. 3. The multimodal fusion early warning subsystem receives violation information from the trajectory spatiotemporal violation determination subsystem, multimodal data from environmental sensors and positioning devices, performs fusion analysis on the multimodal data, generates a safety risk assessment result, and triggers a corresponding early warning response. The multimodal fusion early warning subsystem addresses the limitations of single visual perception, achieving data fusion and improving the accuracy of tiered early warnings. (Refer to...) Figure 3 The multimodal fusion early warning subsystem specifically includes: The data integration module is used to receive and aggregate multi-source information from different subsystems or sensing modules. The multimodal data includes: (1) visual structured data, which is behavior recognition and violation event information from the trajectory spatiotemporal violation judgment subsystem (i.e., violations such as area intrusion, failure to wear safety protective equipment, and abnormal gathering of people, as well as the confidence levels corresponding to each violation); (2) environmental sensor data, including carbon monoxide (CO), methane (CO), and so on. ), hydrogen sulfide ( ),oxygen( (3) GNSS positioning data: provided by the Global Navigation Satellite System (GNSS) positioning equipment, used to obtain the geographic coordinate information of the sensing equipment or deployment node, and output structured spatiotemporal location data.

[0068] The spatiotemporal confidence calculation module is used to assign initial confidence weights to different types of modal data. It also dynamically adjusts the spatiotemporal confidence level of the data based on the stability of the data over time and space. The spatiotemporal confidence decay model is expressed as: ; in, To determine the outlier of the current sensor data compared to historical benchmarks, the variance or mean of historical data within a statistical sliding window is used as the historical benchmark. For the first The decay coefficient corresponding to the modality data is used to characterize the rate at which the credibility of this type of modality data decreases as the outlier degree increases.

[0069] It should be noted that although this embodiment only describes three types of modal data, including visual structured data, environmental sensor data, and GNSS positioning data, multimodal data can be modified, added, or reduced according to the actual situation. This embodiment does not specifically limit or describe this.

[0070] By using the above spatiotemporal confidence calculation method, when a certain modality of data exhibits abnormal fluctuations, instability, or reduced credibility, its influence weight in subsequent fusion decisions will be automatically weakened, thereby improving the reliability and robustness of the overall security risk assessment results.

[0071] The multimodal consistency analysis module is used to evaluate the degree of consistency between data from different modalities to determine whether there are conflicts between multi-source sensing results. At the current moment... The visually structured data that triggers the determination of violations is used as the first... Modal data, using modal data other than visual structured data as the first modality. Modal data, in which Multimodal conflict index The calculation formula is: ; in, Indicates the number of modal data types participating in the fusion; Indicates the first The spatiotemporal confidence of the modal data at the current moment, and ; Indicates that within the same spatiotemporal window, the first Modal data pairs Modal data determination results form conflict evidence strength values ​​that negate or weaken the effect, and When the multimodal conflict index is greater than or equal to a preset threshold, it is determined that the modal data at the current moment has insufficient consistency due to conflict, and the corresponding modal data is either removed or reacquired.

[0072] In this embodiment, the conflict evidence strength value Can be derived from the first Confidence level of modal data regarding the non-existence of violations Obtained via a confidence inverse mapping function, for example through... The method of obtaining, among which The range of values ​​is .

[0073] By calculating the multimodal conflict index, the degree of consistency or conflict between different modal data can be quantified, providing a basis for subsequent risk fusion and decision-making. If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score can be calculated, indicating that the different modal data have achieved consistency and the data reliability is relatively high. It should be noted that the preset threshold can be a value set based on historical experience, and this embodiment does not impose a specific limitation.

[0074] The comprehensive risk score calculation module is used to generate a comprehensive risk score by weighting and fusing multimodal data based on the spatiotemporal confidence and strength of evidence values ​​of each modality, after completing multimodal consistency analysis and confirming the reliability of the modal data. The calculation method is as follows: ; In this embodiment, the first Evidence strength value corresponding to modal data This indicates the degree to which the modal data supports the existence of security risks at the current moment. Indicates the first The spatiotemporal confidence of modal data at the current moment. .

[0075] Specifically, for visually structured data, the strength of evidence value The target confidence level output by the trajectory spatiotemporal violation determination subsystem can be directly used. ,Right now For environmental sensor data, the strength of evidence value It is then calculated based on the standardized deviation between the sensor's measured values ​​and the environmental safety limits.

[0076] The early warning response module is used to trigger a corresponding tiered early warning response mechanism based on the severity of the safety risk assessment results. The module includes an early warning level determination unit, an early warning information generation unit, and an early warning signal output unit. The early warning level determination unit generates a corresponding safety risk assessment result when the comprehensive risk score reaches or exceeds a preset risk assessment threshold, and determines the corresponding early warning level based on the result. Specifically, the system pre-sets multiple risk assessment thresholds, dividing them into at least two risk ranges; when a risk falls into a different risk range, a corresponding level of early warning response is triggered, thus achieving tiered early warning control from low to high. The early warning information generation unit generates early warning information matching the early warning level; the early warning signal output unit outputs early warning signals via sound, light, network, or interface methods to provide timely reminders to relevant personnel or systems. The early warning level determination function is expressed as: ; in, As the risk assessment threshold, This represents the current warning level; a higher value indicates a higher risk level.

[0077] This embodiment integrates visual perception and multi-source environmental information to achieve comprehensive assessment and graded early warning of safety risks in complex scenarios, thereby improving the accuracy and reliability of safety monitoring and realizing real-time continuous safety early warning.

[0078] Reference Figure 4 This application provides a flowchart illustrating an intelligent security early warning method based on multimodal perception fusion. This method is applied to electronic devices, such as servers or mobile terminals. Figure 4 As shown, the intelligent security early warning method based on multimodal perception fusion may include the following steps: Step S401: Decode the video stream data into raw image frames, and preprocess the raw image frames to obtain the preprocessed image; Step S402: Input the preprocessed image into the neural network model to obtain the target detection result, and perform post-processing on the target detection result to obtain multiple target categories, target bounding boxes corresponding to each multiple target category, confidence scores corresponding to each multiple target category, and timestamps corresponding to each multiple target category. Step S403: Based on the timestamp, construct a trajectory history queue arranged in chronological order for each target category. The trajectory history queue includes the coordinates of trajectory points for each target category within a preset time period. Step S404: Calculate the overlap degree based on the target bounding box and restricted area corresponding to each target category. If the overlap degree is greater than the first threshold, count the number of trajectory points located outside the restricted area within a preset time period based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to the second threshold, determine that the target category has a violation behavior, and determine the target confidence degree corresponding to the violation behavior based on the confidence degree corresponding to each target category. Step S405: When it is determined that the target category has a violation, different modal data including visual structured data, environmental sensor data and positioning data are acquired. The visual structured data includes the violation and the target confidence level corresponding to the violation. Step S406: Calculate the spatiotemporal confidence score based on the visual structured data, environmental sensor data, and positioning data; Step S407: Calculate the multimodal conflict index between different modal data based on spatiotemporal confidence. Step S408: If the multimodal conflict index is lower than the preset threshold, a comprehensive risk score is calculated based on the target confidence level, and the warning level is determined based on the comprehensive risk score calculation result.

[0079] It should be noted that since the intelligent security early warning method based on multimodal perception fusion in this embodiment is based on the same inventive concept as the intelligent security early warning system based on multimodal perception fusion described above, the corresponding content in the system embodiment is also applicable to this method embodiment, and will not be described in detail here.

[0080] Reference Figure 5 This application also provides an electronic device, which includes: At least one memory; At least one processor; At least one program; The program is stored in memory, and the processor executes at least one program to implement the intelligent security early warning method based on multimodal perception fusion described above in this disclosure.

[0081] This electronic device can be any smart terminal, including mobile phones, tablets, personal digital assistants (PDAs), and in-vehicle computers.

[0082] The electronic devices according to embodiments of this application will now be described in detail.

[0083] The processor 1600 can be implemented using a general-purpose central processing unit (CPU), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1700 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1700 can store the operating system and other applications. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1700 and is called and executed by the processor 1600 to implement the intelligent security early warning method based on multimodal perception fusion of the embodiments of this disclosure.

[0084] The input / output interface 1800 is used to implement information input and output. The communication interface 1900 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 2000 transmits information between various components of the device (e.g., processor 1600, memory 1700, input / output interface 1800, and communication interface 1900); The processor 1600, memory 1700, input / output interface 1800 and communication interface 1900 are connected to each other within the device via bus 2000.

[0085] This disclosure also provides a storage medium, which is a computer-readable storage medium storing computer-executable instructions for causing a computer to execute the above-described intelligent security early warning method based on multimodal perception fusion.

[0086] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0087] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0088] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0089] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0090] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0091] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0092] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0094] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0095] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause an electronic device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. The embodiments of this application have been described in detail above with reference to the accompanying drawings, but this application is not limited to the above embodiments. Various changes can be made within the scope of knowledge possessed by those skilled in the art without departing from the spirit of this application.

[0097] The embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of this application.

Claims

1. A multimodal perception fusion intelligent security early warning system, characterized in that, The system includes: The video intelligent recognition subsystem is used to decode the input video stream data into raw image frames, preprocess the raw image frames to obtain preprocessed images, perform neural network inference and post-processing on the preprocessed images, and output multiple target categories, target bounding boxes corresponding to each of the multiple target categories, confidence scores corresponding to each of the multiple target categories, and timestamps corresponding to each of the multiple target categories. A trajectory spatiotemporal violation determination subsystem is used to perform multi-target tracking on the multi-target categories. Based on the timestamp, a trajectory history queue arranged in chronological order is constructed for each target category. The trajectory history queue includes the trajectory point coordinates of each target category within a preset time period. Based on the target bounding box and prohibited area corresponding to each target category, the overlap is calculated. If the overlap is greater than a first threshold, the number of trajectory points located outside the prohibited area within the preset time period is counted based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to a second threshold, the target category is determined to have a violation. Based on the confidence level corresponding to each target category, the target confidence level corresponding to the violation is determined. A multimodal fusion early warning subsystem is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data, when determining that the target category has violated regulations. The visual structured data includes the violation and the target confidence level corresponding to the violation. Based on the visual structured data, the environmental sensor data, and the positioning data, a spatiotemporal confidence level is calculated. Based on the spatiotemporal confidence level, a multimodal conflict index between the different modal data is calculated. If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score is calculated based on the target confidence level, and an early warning level is determined based on the comprehensive risk score calculation result.

2. The intelligent security early warning system based on multimodal perception fusion according to claim 1, characterized in that, The video intelligent recognition subsystem includes: The video decoding unit is used to decode the video stream data through a hardware decoder to obtain the original image frames; An image preprocessing unit is used to perform preprocessing operations on the original image frame to obtain a preprocessed image; The neural network inference unit is used to input the preprocessed image into the neural network model to obtain the target detection result, which includes multiple category results, the confidence score and bounding box corresponding to each category result; The post-detection processing unit is used to perform non-maximum suppression and confidence filtering on the target detection results to remove redundant bounding boxes and retain high-confidence categories, thereby obtaining multiple target categories, the confidence scores corresponding to each of the multiple target categories, and the target bounding boxes corresponding to each of the multiple target categories. An adaptive adjustment unit is used to adjust the sampling interval of the original image frame by the image preprocessing unit; The result output unit is used to output the multi-object category, the target bounding box corresponding to each of the multi-object categories, the confidence level corresponding to each of the multi-object categories, and the timestamp corresponding to each of the multi-object categories to the trajectory spatiotemporal violation determination subsystem.

3. The intelligent security early warning system based on multimodal perception fusion according to claim 1, characterized in that, The trajectory spatiotemporal violation determination subsystem includes: A multi-target tracking module is used to perform cross-frame tracking of the multiple target categories and assign a unique tracking identifier to each target category; The trajectory history maintenance module is used to construct a trajectory history queue arranged in chronological order for each target category based on the timestamp. The trajectory history queue is used to record the trajectory point coordinates, timestamps, and attribute status of each target category in a preset time period. The attribute status includes the unique tracking identifier of each target category and whether it is in a restricted area. The regional intrusion detection module is used to calculate the overlap between the current target bounding box of the target category and the preset restricted area. The overlap is the ratio of the intersection area of ​​the target bounding box and the restricted area to the total area of ​​the target bounding box. The real intrusion determination module is used to: if the overlap is greater than the first threshold, count the number of trajectory points located outside the restricted area within the preset time period based on the trajectory point coordinates; if the number of trajectory points is greater than or equal to the second threshold, determine that the target category has violated regulations, and determine the target confidence level corresponding to the violation based on the confidence level corresponding to each target category; if the number of trajectory points is less than the second threshold, determine that the target category does not violate regulations and does not trigger an alarm. The violation status management module is used to manage the status of the violation based on a finite state machine. The violation includes area intrusion, failure to wear security protective equipment, and abnormal gathering of people. The status includes normal status, first violation status, and continuous violation status. If the target category is judged to be in violation for the first time, an alarm is triggered and the target category enters the first violation status. If the target category continues to violate the rules and the time since the last alarm exceeds a preset reminder interval, an alarm is triggered again and the continuous violation status is maintained. If the target category leaves the restricted area or the trajectory of the target category has not been updated for a long time, the status is reset to normal status.

4. The intelligent security early warning system based on multimodal perception fusion according to claim 1, characterized in that, The multimodal fusion early warning subsystem includes: The data integration module is used to acquire different modal data, including visual structured data, environmental sensor data, and positioning data. The spatiotemporal confidence calculation module is used to calculate the spatiotemporal confidence based on the visual structured data, the environmental sensor data, and the positioning data. The multimodal consistency analysis module is used to calculate the multimodal conflict index between the different modal data based on the spatiotemporal confidence level. The comprehensive risk score calculation module is used to calculate a comprehensive risk score based on the target confidence level if the multimodal conflict index is lower than a preset threshold. The early warning response module is used to determine the early warning level based on the comprehensive risk score calculation results, and to trigger the corresponding early warning response based on the early warning level.

5. The intelligent security early warning system based on multimodal perception fusion according to claim 4, characterized in that, Based on the visual structured data, the environmental sensor data, and the positioning data, the spatiotemporal confidence score is calculated, including: Based on a time sliding window, the historical baselines corresponding to the visual structured data, the environmental sensor data, and the positioning data are calculated respectively. Initial confidence weights are assigned to the visual structured data, the environmental sensor data, and the positioning data, respectively. Based on the historical benchmark and the initial confidence weight, the spatiotemporal confidence scores corresponding to the visual structured data, the environmental sensor data, and the positioning data are calculated respectively.

6. The intelligent security early warning system based on multimodal perception fusion according to claim 4, characterized in that, The calculation of the multimodal conflict index between the different modal data based on the spatiotemporal confidence level includes: ； in, This indicates the number of modal data types involved in the fusion. Indicates the multimodal conflict index. Indicates the first The spatiotemporal confidence of modal data at the current moment. Indicates that within the same spatiotemporal window, the first The strength of evidence value for class modal data that negates or weakens the violation determined by the visual modal data.

7. The intelligent security early warning system based on multimodal perception fusion according to claim 4, characterized in that, The calculation of the comprehensive risk score based on the target confidence level includes: ； in, This represents the overall risk score. This indicates the number of modal data types involved in the fusion. Indicates the first The spatiotemporal confidence of modal data at the current moment. Indicates the first The evidence strength value of the modal data at the current moment, wherein the evidence strength value is equal to the target confidence level.

8. A multimodal perception fusion intelligent security early warning method, characterized in that, The method is applied to the multimodal perception fusion intelligent security early warning system, including: The video stream data is decoded into raw image frames, and the raw image frames are preprocessed to obtain a preprocessed image. The preprocessed image is input into a neural network model to obtain target detection results. The target detection results are then post-processed to obtain multiple target categories, target bounding boxes corresponding to each of the multiple target categories, confidence scores corresponding to each of the multiple target categories, and timestamps corresponding to each of the multiple target categories. Based on the timestamp, a trajectory history queue arranged in chronological order is constructed for each target category. The trajectory history queue includes the coordinates of trajectory points for each target category within a preset time period. Based on the target bounding box and restricted area corresponding to each target category, the overlap is calculated. If the overlap is greater than a first threshold, the number of trajectory points located outside the restricted area within the preset time period is counted based on the trajectory point coordinates. If the number of trajectory points is greater than or equal to a second threshold, the target category is determined to have a violation. Based on the confidence level corresponding to each target category, the target confidence level corresponding to the violation is determined. When determining that the target category has a violation, different modal data including visual structured data, environmental sensor data and positioning data are acquired. The visual structured data includes the violation and the target confidence level corresponding to the violation. Based on the visual structured data, the environmental sensor data, and the positioning data, calculate the spatiotemporal confidence score; Based on the spatiotemporal confidence level, calculate the multimodal conflict index between the different modal data; If the multimodal conflict index is lower than a preset threshold, a comprehensive risk score is calculated based on the target confidence level, and the warning level is determined based on the comprehensive risk score calculation result.

9. An electronic device, characterized in that, It includes at least one control processor and a memory for communicatively connecting to the at least one control processor; the memory stores instructions executable by the at least one control processor, which, when executed by the at least one control processor, enable the at least one control processor to perform the intelligent security early warning method of multimodal perception fusion as described in claim 8.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions for causing a computer to execute the intelligent security early warning method of multimodal perception fusion as described in claim 8.

Images