Security and disinfection integrated multi-modal fusion embedded edge calculation method and system

Abstract

The invention relates to the technical field of fire protection and security, in particular to a security and disinfection integrated multi-modal fusion embedded edge calculation method and system, and provides a security and disinfection integrated multi-modal fusion embedded edge calculation method which comprises the following steps: constructing a data set; the AI visual recognition model uses an improved YOLOv8 model to train a data set and execute target detection, meanwhile, an optical flow method is adopted to extract motion features of flames or smog in continuous key frame images in a monitoring video, and a spreading prediction model is established to send out an early warning signal in advance; and the fusion processing module establishes a dynamic threshold adjustment model based on the environmental parameters to optimize an early warning trigger threshold, and transmits the early warning trigger threshold to the fire-fighting linkage control module to verify and trigger fire-fighting linkage operation. AI intelligent analysis and fire-fighting linkage disposal of fire hazards are realized, false alarm of early warning information is effectively prevented, the discovery and disposal efficiency of the fire hazards is improved, and the fire hazards are eliminated in the bud.

Description

Technical Field

[0001] The present invention relates to the field of fire protection and security technology, and in particular to a multi-modal fusion embedded edge computing method and system for integrated fire protection and security. Background Art

[0002] With the acceleration of urbanization and the increasing complexity of building structures, fire hazard prevention and control has become a major concern in public safety management. Currently, common fire warning systems primarily rely on single physical sensing devices, such as smoke detectors and temperature sensors, to detect fire activity by monitoring changes in environmental parameters. While these systems are easy to deploy and relatively low-cost, they are limited in their ability to identify early fires in complex scenarios. This is particularly true in the presence of wind disturbances, high humidity, obstructions, or non-open flames. These systems are prone to false alarms and missed alarms, resulting in limited reliability and practicality.

[0003] Furthermore, traditional fire monitoring systems typically lack video image processing capabilities, making them unable to dynamically identify and analyze fire evolution. Furthermore, these systems often operate independently, making it difficult to coordinate and control subsystems such as surveillance video, security access control, and electrical control. This results in slow response times and fragmented response processes, making it difficult to meet the demands of modern buildings for intelligent, integrated, and real-time fire prevention and control.

[0004] Therefore, the existing technology urgently needs an embedded fire warning system with multimodal information fusion capabilities, edge intelligent analysis capabilities and security and fire protection system linkage control capabilities to improve the accuracy of fire identification, the timeliness of response and the overall intelligence level of system operation, thereby meeting the actual needs of early fire intelligent identification and emergency linkage in complex scenarios. Summary of the Invention

[0005] This application proposes a multi-modal fusion embedded edge computing method and system for integrated fire safety and firefighting. The system aims to achieve AI intelligent analysis of fire hazards and firefighting linkage disposal by integrating AI visual recognition algorithm modules, multi-sensor data acquisition and fusion processing, and firefighting system linkage control functions, effectively improving the level of fire safety management. To achieve the above objectives, the present invention is implemented through the following technologies: The present invention provides a multi-modal fusion embedded edge computing method for integrated security and fire protection, including: Step S1: Select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, fusion processing module and fire linkage control module into an embedded system and encapsulate it in an edge computing box; Step S2: extract key frame images from the adaptively preprocessed surveillance video to construct a dataset, and extend the dataset by simulating fireworks images in different scenarios using a generative adversarial network; Step S3: The AI ​​visual recognition model uses the improved YOLOv8 model as the detection model to train the dataset and perform target detection, returning the category and location of the smoke and fire features in the image. At the same time, the optical flow method is used to extract the motion features of the flame or smoke in the continuous key frame images in the surveillance video. Based on the motion features, a spread prediction model is established to predict the path and time of fire spread. An early warning signal is issued based on the analysis results of the spread prediction model. Step S4: The fusion processing module accesses one or more sensors to obtain perception information, and establishes a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold. The perception information that exceeds the warning trigger threshold and the results of the AI ​​visual recognition module are transmitted to the fire linkage control module, and the fire linkage operation is triggered through verification through a double verification mechanism.

[0006] Furthermore, in step S2, key frame images are extracted from the adaptively pre-processed surveillance video to construct a data set, including: Step S21: Obtain a surveillance video, and perform dynamic denoising on the surveillance video using a wavelet transform or a non-local means denoising algorithm to suppress noise caused by environmental changes in the surveillance video; Step S22: using a Retinex algorithm to enhance the visibility of fireworks features in a dark environment; Step S23: Based on background modeling technology, identify dynamic areas in the surveillance video and perform ROI focusing; Step S24: extract key frame images from the surveillance video to construct a data set, where the key frame images include visible light images, infrared images, and thermal imaging images.

[0007] Furthermore, in step S2, a generative adversarial network is used to simulate and generate an extended dataset of fireworks images in different scenarios, and the dataset is preprocessed, including: Step S25: simulating the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; Step S26: Automatically label the unlabeled dataset images using the pre-trained model, generate labels, and perform verification; Step S27: Divide the dataset into a test set, a training set, and a validation set, and perform target detection through the detection model. The training set is used to train and adjust weights and optimize parameters, the validation set is used to optimize hyperparameters during training, and the test set is used to evaluate model performance.

[0008] Furthermore, in step S3, the AI ​​visual recognition model uses the improved YOLOv8 model as the detection model to train the dataset and perform target detection, returning the category and location of the fireworks features in the image, including: Step S31: The input layer of the detection model receives the image in the training set as input; Step S32: extracting fireworks features from the image using a feature extraction network training set, including thermal imaging temperature features, smoke features, and flame features, and performing multi-scale fusion through a feature fusion network; Step S33: The detection head introduces the CBAM attention mechanism to focus on the target area in the image and suppress background interference. The detection head performs target detection to obtain the category and position of the fireworks features in the image.

[0009] Furthermore, in step S3, the optical flow method is used to extract the motion features of flames or smoke in continuous key frame images in the surveillance video, and a spread prediction model is established based on the motion features to predict the path and time of fire spread. An early warning signal is issued based on the analysis results of the spread prediction model, including: Step S34: Detecting fireworks features based on the detection model and acquiring continuous key frame images from the surveillance video; Step S35: Calculate the movement direction or speed of the smoke feature or flame feature using the optical flow method, and output the corresponding motion feature vector; Step S36: The motion feature vectors of each continuous key frame image are organized into time series data in chronological order, and the time series data is processed through the RNN network to establish a fire spread prediction model, predict the direction and path of fire spread, and issue an early warning signal for warning based on the analysis results.

[0010] Furthermore, in step S4, the fusion processing module accesses one or more sensors to obtain perception information and establishes a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold, including: Step S41: Connect multiple sensors to the fusion processing module through a wireless or wired communication interface, and transmit the collected perception information to the fusion processing module through a communication protocol, wherein the sensors include temperature and humidity sensors, smoke sensors, and wind speed sensors; Step S42: Initializing the initial threshold of the sensor, which is a safety standard value; Step S43: using multiple linear regression to establish a relationship model between environmental parameters including temperature, humidity, and wind speed and smoke concentration, and training the relationship model with historical data to calculate the influence coefficient corresponding to each environmental parameter; Step S44: input the environmental parameters collected in real time into the relationship model, calculate the corresponding adjustment range, and update the safety standard value of the sensor.

[0011] Furthermore, in step S4, the perception information exceeding the warning trigger threshold and the results of the AI ​​visual recognition module are transmitted to the fire linkage control module, and the fire linkage operation is triggered through a double verification mechanism for verification, including: Step S45: When the fire linkage control module receives the warning information from the AI ​​visual recognition module or the alarm information from the fusion processing module, it automatically reviews the alarm information or warning information from the other module, triggers a local alarm and pushes it to the monitoring center; Step S46: When the fire linkage control module receives the early warning information or the alarm information at the same time, it automatically links the fire fighting equipment, performs operations according to the return results of the detection model, and sends an emergency signal to the fire department.

[0012] Based on the same inventive concept, the present invention provides a multi-modal fusion embedded edge computing system for integrated security and fire protection, which adopts the above-mentioned multi-modal fusion embedded edge computing method, including: A deployment configuration module is used to select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, fusion processing module, and fire linkage control module into an embedded system and encapsulate it in an edge computing box; A dataset construction module is used to extract key frame images from adaptively pre-processed surveillance videos to construct a dataset, and to expand the dataset by using a generative adversarial network to simulate and generate fireworks images in different scenarios; The target detection module, used by the AI ​​visual recognition model, uses an improved YOLOv8 model as a detection model to train the dataset and perform target detection. It returns the category and location of smoke and fire features in the image. It also uses optical flow to extract the motion features of flames or smoke in consecutive keyframe images in the surveillance video. Based on the motion features, it builds a spread prediction model to predict the path and time of fire spread. The analysis results of the spread prediction model are used to issue early warning signals. The data processing module is used to integrate the processing module to access one or more sensors to obtain perception information, and establish a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold. The perception information that exceeds the warning trigger threshold and the AI ​​visual recognition results are transmitted to the fire linkage control module, and the fire linkage operation is triggered through verification through a double verification mechanism.

[0013] Furthermore, the dataset construction module includes, The dataset collection unit is used to acquire surveillance videos and dynamically denoise them using wavelet transform or non-local means denoising algorithms to suppress noise caused by environmental changes in the surveillance videos. The unit also uses the Retinex algorithm to enhance the visibility of fireworks features in low-light environments. Based on background modeling technology, the unit identifies dynamic regions in the surveillance videos and performs ROI focusing. Keyframe images are extracted from the surveillance videos to construct a dataset. Keyframe images include visible light images, infrared images, and thermal images. The dataset expansion unit is used to simulate the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; use the pre-trained model to automatically annotate the unlabeled dataset images, generate labels and perform verification; divide the dataset into test set, training set and validation set, and perform target detection through the detection model, among which the training set is used to train and adjust weights and optimize parameters, the validation set is used for hyperparameter optimization during training, and the test set is used to evaluate model performance.

[0014] Furthermore, the target detection module includes, The fireworks feature detection unit, used for the input layer of the detection model, receives images from the training set as input. It uses a feature extraction network to extract fireworks features from the images, including thermal imaging temperature features, smoke features, and flame features, and performs multi-scale fusion through a feature fusion network. The detection head introduces a CBAM attention mechanism to focus on the target area in the image and suppress background interference. The detection head then performs target detection to obtain the category and location of the fireworks features in the image. The fire spread prediction unit is used to detect smoke and fire features based on the detection model and obtain continuous key frame images from the monitoring video; calculate the movement direction or speed of smoke features or flame features through the optical flow method, and output the corresponding motion feature vector; organize the motion feature vectors of each continuous key frame image into time series data in chronological order, and process the time series data through the RNN network to establish a fire spread prediction model, predict the direction and path of fire spread, and issue an early warning signal for warning based on the analysis results.

[0015] Compared with the prior art, the present invention has at least one of the following beneficial effects: This invention integrates AI visual recognition, multi-sensor data fusion, and fire system linkage control to achieve AI-powered intelligent analysis of fire hazards and coordinated fire response. This effectively prevents false alarms, improves the efficiency of fire hazard detection and response, and eliminates fire hazards in their infancy, preventing them from causing significant losses. Furthermore, this multimodal fusion service enhances the intelligence of fire safety management and provides a strong guarantee for public safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is a flowchart of the steps of the multi-modal fusion embedded edge computing method for integrated security and fire protection of the present invention. DETAILED DESCRIPTION

[0017] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0018] Those skilled in the art will appreciate that, unless otherwise stated, the singular forms "a," "an," "said," and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the stated features, integers, steps, operations, elements, and / or components, but does not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0019] First embodiment Existing fire warning systems typically rely on a single smoke or temperature sensor to detect fire hazards. However, single sensors can suffer from high false alarm rates and delayed response times in complex environments. For example, in conditions of high humidity or large temperature fluctuations, traditional sensors may be unable to accurately determine the onset of a fire and are susceptible to interference from environmental factors. Furthermore, traditional systems lack sufficient flexibility and adaptability in the face of rapidly changing fire scenarios, resulting in poor accuracy and timeliness in fire warnings.

[0020] In order to solve the above problems, the inventors proposed a multi-modal fusion embedded edge computing method for integrated security and firefighting based on the shortcomings of the existing technology. Through AI visual recognition technology and multi-sensor data fusion, combined with edge computing technology, multi-dimensional information processing and dynamic response are achieved. Specifically, the system monitors environmental parameters such as temperature, humidity, wind speed, etc. in real time, and combines video image analysis to more accurately judge the risk of fire and dynamically adjust the alarm threshold, thereby reducing false alarms and missed alarms, and significantly improving the response speed and reliability of the system. The specific implementation method is as follows: like Figure 1 As shown, the present invention provides a multi-modal fusion embedded edge computing method for integrated security and fire protection, including: Step S1: Select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, fusion processing module and fire linkage control module into an embedded system and encapsulate it in an edge computing box; Specifically, the edge computing box serves as the system's core processing unit, equipped with a high-performance processor and storage module. It supports edge computing to reduce data transmission latency. This edge computing box is deployed in fire-prone locations, such as high-rise buildings, industrial parks, and new energy charging stations, as well as warehouses for flammable and explosive materials. The edge computing box is installed on-site and connected to devices such as cameras and smoke detectors. After the system boots up, the AI ​​visual recognition module begins processing video images in real time while simultaneously collecting information from the smoke detectors. When the system detects a fire hazard, it immediately issues a warning signal and connects to the fire extinguishing control unit of the fire protection system via a communication module, triggering the activation of the fire extinguishing device. The system also uploads the warning information to the monitoring center, enabling management personnel to take timely action. The lightweight hardware and software deployment in this application primarily relies on the use of NVIDIA GPUs to accelerate convolution operations, reducing the inference latency of the visual algorithm to less than 20ms. The software is optimized based on the YOLOv8 algorithm, compressing the model parameters to less than 10MB through channel pruning and knowledge distillation. This software is compatible with NVIDIA edge computing devices and enables real-time analysis and computation of 1080P video streams.

[0021] Step S2: extract key frame images from the adaptively preprocessed surveillance video to construct a dataset, and extend the dataset by simulating fireworks images in different scenarios using a generative adversarial network; Step S3: The AI ​​visual recognition model uses the improved YOLOv8 model as the detection model to train the dataset and perform target detection, returning the category and location of the smoke and fire features in the image. At the same time, the optical flow method is used to extract the motion features of the flame or smoke in the continuous key frame images in the surveillance video. Based on the motion features, a spread prediction model is established to predict the path and time of fire spread. An early warning signal is issued based on the analysis results of the spread prediction model. Step S4: The fusion processing module accesses one or more on-site firefighting sensors to acquire and collect sensory information. A dynamic threshold adjustment model is established based on environmental parameters to optimize the warning trigger threshold. Sensory information exceeding the warning trigger threshold, along with the results of the AI ​​visual recognition module, is transmitted to the fire linkage control module. A double-check mechanism verifies and triggers the fire linkage operation, preventing warning errors caused by false alarms from a single sensor or module.

[0022] Furthermore, in step S2, key frame images are extracted from the adaptively pre-processed surveillance video to construct a data set, including: Step S21: Acquire a surveillance video, and perform dynamic denoising on the surveillance video using a wavelet transform or a non-local means denoising algorithm to suppress noise caused by environmental changes in the surveillance video; Step S22: using a Retinex algorithm to enhance the visibility of fireworks features in a dark environment; Step S23: Based on background modeling technology, identify dynamic areas in the surveillance video and perform ROI focusing; Step S24: extract key frame images from the surveillance video to construct a data set, where the key frame images include visible light images, infrared images, and thermal imaging images.

[0023] Specifically, front-end cameras, such as visible light, infrared, and thermal imaging cameras, transmit captured video streams to the AI ​​visual recognition module via wired or wireless channels. The module then performs preprocessing on the received video streams. This includes dynamic denoising of the surveillance video using wavelet transforms or non-local means denoising algorithms to reduce motion blur and illumination fluctuations in the surveillance footage. The Retinex algorithm enhances the visibility of fire and smoke features in low-light environments, improving fire detection rates by 15% in nighttime warehouse scenarios, for example. Based on background modeling techniques, dynamic regions in the surveillance video are identified and focused on, targeting high-frequency activity areas and reducing inefficient computation. Keyframe images are extracted from the surveillance video to construct a dataset. These keyframe images include visible light, infrared, and thermal images. Visible light images provide morphological and texture features for identifying smoke, flames, or other visual features. In particular, in fires, flames have irregular edges, while smoke appears diffuse or floating. Texture features of smoke and flames can also be extracted using texture analysis methods. Infrared images primarily provide thermal imaging temperature features, primarily thermal radiation from objects, which are used to detect areas of abnormal temperature. It can effectively detect heat sources and identify potential fire hazards based on abnormal temperature areas in infrared images, which are high-temperature areas. Thermal imaging features are used to obtain temperature distribution information, which reflects the intensity of thermal radiation from an object. High-temperature areas (red / pink) contrast sharply with low-temperature areas (blue / green). This temperature distribution information not only displays static temperature distribution but also dynamically monitors the rate of temperature change, a key feature used to determine the spread of fire.

[0024] Furthermore, in step S2, a generative adversarial network is used to simulate and generate an extended dataset of fireworks images in different scenarios, and the dataset is preprocessed, including: Step S25: simulating the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; Step S26: Automatically label the unlabeled dataset images using the pre-trained model, generate labels, and perform verification; Step S27: Divide the dataset into a test set, a training set, and a validation set, and perform target detection through the detection model. The training set is used to train and adjust weights and optimize parameters, the validation set is used to optimize hyperparameters during training, and the test set is used to evaluate model performance.

[0025] Furthermore, in step S3, the AI ​​visual recognition model uses the improved YOLOv8 model as the detection model to train the dataset and perform target detection, returning the category and location of the fireworks features in the image, including: Step S31: The input layer of the detection model receives the image in the training set as input; Step S32: extracting fireworks features from the image using a feature extraction network training set, including thermal imaging temperature features, smoke features, and flame features, and performing multi-scale fusion through a feature fusion network; Step S33: The detection head introduces the CBAM attention mechanism to focus on the target area in the image and suppress background interference, such as focusing on the core area of ​​the flame and suppressing background interference, such as flame-like neon lights, red clothing, etc.; and performs target detection through the detection head to obtain the category and location of the fireworks features in the image.

[0026] Specifically, the CBAM module uses two stages, spatial attention and channel attention, to enhance focus on important image regions, such as flames and smoke. It optimizes the input feature map by calculating attention weights, allowing the model to focus on more important areas and suppress background interference. Consequently, based on the feature map optimized by the CBAM module, detailed image features (such as texture, edges, and shape) as well as deep semantic features (such as the shape and position of the flames) are enhanced for subsequent object detection.

[0027] At the same time, it also includes spatiotemporal consistency verification, the specific steps are: Continuous frame detection: Target detection is performed on continuous frame images extracted from surveillance video. Targets in continuous frame images are identified and located, such as flame features and smoke features. In three consecutive frames, the detection targets are detected and confirmed to be consistent. Spatial overlap calculation: Position matching is performed on the objects detected in three consecutive frames, and the spatial overlap in each frame is calculated. The overlap ratio is measured using the IoU metric. The calculation method is: compare the bounding boxes of the objects in the two frames, calculate their intersection area and union area, and then find the ratio of the intersection to the union area. When the spatial overlap of targets in three consecutive frames exceeds a preset value, such as 70%, it is determined that a fire exists. When the detection target appears in a single frame or part of the frame and the spatial overlap does not exceed a preset value, such as 70%, the detection target is regarded as a momentary interference and the fire spread prediction model is not activated.

[0028] The loss function and optimizer selection for this detection model include using a focal loss to address class imbalance and improve the model's ability to identify minority classes, such as fire. The CIoU regression loss function optimizes the accuracy of flame bounding box positioning, reducing the fire point localization error to within 0.5 meters. The AdamW optimizer, combined with a cosine annealing learning rate strategy, accelerates model convergence and improves stability.

[0029] Furthermore, in step S3, the optical flow method is used to extract the motion features of flames or smoke in continuous key frame images in the surveillance video, and a spread prediction model is established based on the motion features to predict the path and time of fire spread. An early warning signal is issued based on the analysis results of the spread prediction model, including: Step S34: Detecting fireworks features based on the detection model and acquiring continuous key frame images from the surveillance video; Step S35: Calculate the movement direction or speed of the smoke feature or flame feature using the optical flow method, and output the corresponding motion feature vector; Step S36: The motion feature vectors of each continuous key frame image are organized into time series data in chronological order, and the time series data is processed through the RNN network to establish a fire spread prediction model, predict the direction and path of fire spread, and issue an early warning signal for warning based on the analysis results.

[0030] It should be noted that the motion calculation by optical flow method is to calculate the pixel motion between consecutive frames and determine the moving direction and speed of the target in the video: Extract the visible light images of the current frame and the previous frame for optical flow analysis. Use the bounding box of the detection target output by the detection model in each frame as the area for optical flow calculation and extract the pixel motion in this area. The Lucas-Kanade optical flow method or the Farnebäck optical flow method is used to generate the motion direction and speed of each pixel, and estimate the motion direction of the detection target, such as the flame spread direction and smoke diffusion speed. Based on the analysis results generated by the spread prediction model, an early warning signal is sent to the fire linkage control module for reminder or cross-verification with the fusion processing module to trigger fire linkage operations, such as activating fire-fighting equipment such as sprinklers and smoke exhaust systems, and sending emergency signals to the fire department.

[0031] Furthermore, in step S4, the fusion processing module accesses one or more sensors to obtain perception information and establishes a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold, including: Step S41: Connect multiple sensors to the fusion processing module through a wireless or wired communication interface, and transmit the collected perception information to the fusion processing module through a communication protocol, wherein the sensors include temperature and humidity sensors, smoke sensors, and wind speed sensors; Step S42: Initialize the sensor's initial threshold. This initial threshold is a safety standard value to ensure proper system operation and timely response in the early stages of a fire. For example, the smoke sensor's initial threshold is set between 5% OBS / M and 15% OBS / M (optical density). This range represents the range of smoke concentration. When the smoke concentration reaches or exceeds 15% OBS / M, indicating the presence of severe smoke, the smoke sensor triggers an alarm. The smoke sensor uses a photosensor to sense changes in smoke particle concentration. When the smoke concentration exceeds the set threshold, the sensor converts the change in smoke concentration into an electrical signal for further processing.

[0032] Step S43: using multiple linear regression to establish a relationship model between environmental parameters including temperature, humidity, and wind speed and smoke concentration, and training the relationship model with historical data to calculate the influence coefficient corresponding to each environmental parameter; Step S44: input the environmental parameters collected in real time into the relationship model, calculate the corresponding adjustment range, and update the safety standard value of the sensor.

[0033] Specifically, environmental factors do have a certain impact on the performance and sensitivity of smoke sensors. This is especially true in everyday environments, where sensors may be interfered with by non-fire factors, leading to false alarms or delayed responses. The following are some environmental factors that may affect smoke sensor performance and their impact on fire warning systems: For example, higher temperatures may increase air flow or cause changes in smoke particles, affecting sensor readings. Humidity has a significant impact on the sensitivity of smoke sensors. High humidity can cause smoke particles to aggregate or change their light scattering properties, affecting the working principle of photoelectric smoke sensors. Furthermore, air flow can affect the diffusion of smoke particles, which in turn affects smoke sensor detection. High wind speeds can cause smoke particles to disperse too quickly, making the smoke concentration unstable within the sensor's sampling range or causing false alarms. In this embodiment, the temperature and humidity sensors, smoke sensors, and wind speed sensors are connected to the fusion processing module through the communication interface to collect and transmit environmental data. The temperature and humidity sensors are used to measure the temperature and humidity of the environment. The smoke sensor is used to sense changes in smoke particle concentration through a photosensor and detect the smoke concentration (unit: %OBS / M). The wind speed sensor is used to measure wind speed and analyze the impact of the fire spread direction. The sensors periodically collect data and transmit it to the central fusion processing module through a communication protocol (wireless or wired). The communication protocol can use wired communication (such as Modbus, RS-485) or wireless communication (such as Wi-Fi, ZigBee, LoRa, etc.) to transmit data to the fusion module. Initialize sensor threshold settings to ensure proper operation and timely response to early fires. For example, the smoke sensor's initial threshold is set between 5% OBS / M and 15% OBS / M, representing the range of smoke concentration. If the smoke concentration exceeds 15% OBS / M, indicating severe smoke, the smoke sensor triggers an alarm. When smoke concentration reaches or exceeds the threshold, an alarm is triggered and appropriate measures are taken.

[0034] A multivariate linear regression method was used to establish a model for the relationship between the environmental parameters temperature, humidity, wind speed, and smoke concentration. The regression model was trained using historical data on temperature, humidity, wind speed, and smoke concentration. The least squares method was used to optimize the regression coefficients to minimize the model's fitting error to the historical data. The resulting regression model was then used to predict smoke concentration for given temperature, humidity, and wind speed.

[0035] Input the real-time collected environmental parameters into the relational model, calculate the corresponding adjustment range, and update the safety standard value of the sensor. Furthermore, in step S4, the perception information exceeding the warning trigger threshold and the results of the AI ​​visual recognition module are transmitted to the fire linkage control module, and the fire linkage operation is triggered through a double verification mechanism for verification, including: Step S45: When the fire linkage control module receives the warning information from the AI ​​visual recognition module or the alarm information from the fusion processing module, it automatically reviews the alarm information or warning information from the other module, triggers a local alarm and pushes it to the monitoring center; Step S46: When the fire linkage control module receives the early warning information or the alarm information at the same time, it automatically links the fire fighting equipment, performs operations according to the return results of the detection model, and sends an emergency signal to the fire department.

[0036] Second embodiment Based on the same inventive concept, the present invention provides a multi-modal fusion embedded edge computing system for integrated security and fire protection, which adopts the above-mentioned multi-modal fusion embedded edge computing method, including: A deployment configuration module is used to select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, fusion processing module, and fire linkage control module into an embedded system and encapsulate it in an edge computing box; A dataset construction module is used to extract key frame images from adaptively pre-processed surveillance videos to construct a dataset, and to expand the dataset by using a generative adversarial network to simulate and generate fireworks images in different scenarios; The target detection module, used by the AI ​​visual recognition model, uses an improved YOLOv8 model as a detection model to train the dataset and perform target detection. It returns the category and location of smoke and fire features in the image. It also uses optical flow to extract the motion features of flames or smoke in consecutive keyframe images in the surveillance video. Based on the motion features, it builds a spread prediction model to predict the path and time of fire spread. The analysis results of the spread prediction model are used to issue early warning signals. The data processing module is used to integrate the processing module to access one or more sensors to obtain perception information, and establish a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold. The perception information that exceeds the warning trigger threshold and the AI ​​visual recognition results are transmitted to the fire linkage control module, and the fire linkage operation is triggered through verification through a double verification mechanism.

[0037] Furthermore, the dataset construction module includes, The dataset collection unit is used to acquire surveillance videos and dynamically denoise them using wavelet transform or non-local means denoising algorithms to suppress noise caused by environmental changes in the surveillance videos. The unit also uses the Retinex algorithm to enhance the visibility of fireworks features in low-light environments. Based on background modeling technology, the unit identifies dynamic regions in the surveillance videos and performs ROI focusing. Keyframe images are extracted from the surveillance videos to construct a dataset. Keyframe images include visible light images, infrared images, and thermal images. The dataset expansion unit is used to simulate the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; use the pre-trained model to automatically annotate the unlabeled dataset images, generate labels and perform verification; divide the dataset into test set, training set and validation set, and perform target detection through the detection model, among which the training set is used to train and adjust weights and optimize parameters, the validation set is used for hyperparameter optimization during training, and the test set is used to evaluate model performance.

[0038] Furthermore, the target detection module includes, The fireworks feature detection unit, used for the input layer of the detection model, receives images from the training set as input. It uses a feature extraction network to extract fireworks features from the images, including thermal imaging temperature features, smoke features, and flame features, and performs multi-scale fusion through a feature fusion network. The detection head introduces a CBAM attention mechanism to focus on the target area in the image and suppress background interference. The detection head then performs target detection to obtain the category and location of the fireworks features in the image. The fire spread prediction unit is used to detect smoke and fire features based on the detection model and obtain continuous key frame images from the monitoring video; calculate the movement direction or speed of smoke features or flame features through the optical flow method, and output the corresponding motion feature vector; organize the motion feature vectors of each continuous key frame image into time series data in chronological order, and process the time series data through the RNN network to establish a fire spread prediction model, predict the direction and path of fire spread, and issue an early warning signal for warning based on the analysis results.

[0039] While various embodiments of the present disclosure have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A multi-modal fusion embedded edge computing method for integrated security and fire protection, characterized in that: include: Step S1: Select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, the fusion processing module and the fire linkage control module into an embedded system and encapsulate it in an edge computing box; Step S2: extracting key frame images from the adaptively pre-processed surveillance video to construct a dataset, and expanding the dataset by simulating and generating fireworks images in different scenarios using a generative adversarial network; Step S3: The AI ​​visual recognition model uses the improved YOLOv8 model as a detection model to train the dataset and perform target detection, returning the category and location of the smoke and fire features in the image. At the same time, the optical flow method is used to extract the motion features of the flame or smoke in the continuous key frame images in the surveillance video, and a spread prediction model is established based on the motion features to predict the path and time of fire spread. An early warning signal is issued based on the analysis results of the spread prediction model. Step S4: The fusion processing module accesses one or more of the sensors to obtain perception information, and establishes a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold, and transmits the perception information that exceeds the warning trigger threshold and the result of the AI ​​visual recognition module to the fire linkage control module, and verifies through a double verification mechanism to trigger the fire linkage operation.

2. The multimodal fusion embedded edge computing method according to claim 1, characterized in that: In step S2, key frame images are extracted from the adaptively preprocessed surveillance video to construct a dataset, including: Step S21: Acquire the surveillance video, and perform dynamic denoising on the surveillance video using a wavelet transform or a non-local means denoising algorithm to suppress noise caused by environmental changes in the surveillance video; Step S22: enhancing the visibility of the fireworks features in a dark environment by using a Retinex algorithm; Step S23: Based on background modeling technology, identify the dynamic area in the surveillance video and perform ROI focusing; Step S24: extracting key frame images from the surveillance video to construct the data set, wherein the key frame images include visible light images, infrared images and thermal imaging images.

3. The multimodal fusion embedded edge computing method according to claim 2, characterized in that: In step S2, the firework images in different scenes are simulated and generated using a generative adversarial network to expand the dataset, and image preprocessing is performed on the dataset, including: Step S25: simulating the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; Step S26: automatically labeling the unlabeled images in the dataset using the pre-trained model, generating labels and performing verification; Step S27: Divide the dataset into a test set, a training set, and a validation set, and perform target detection using the detection model, wherein the training set is used for training weight adjustment and parameter optimization, the validation set is used for hyperparameter optimization during training, and the test set is used for evaluating model performance.

4. The multimodal fusion embedded edge computing method according to claim 3, characterized in that: In step S3, the AI ​​visual recognition model uses the improved YOLOv8 model as a detection model to train the dataset and perform target detection, returning the category and location of the fireworks features in the image, including: Step S31: the input layer of the detection model receives the image in the training set as input; Step S32: extracting the fireworks features of the image from the training set using a feature extraction network, including thermal imaging temperature features, smoke features, and flame features, and performing multi-scale fusion through a feature fusion network; Step S33: The detection head introduces the CBAM attention mechanism to focus on the target area in the image and suppress background interference, and performs the target detection through the detection head to obtain the category and position of the fireworks feature in the image.

5. The multimodal fusion embedded edge computing method according to claim 4, characterized in that: In step S3, the optical flow method is used to extract the motion features of the flame or smoke in the continuous key frame images in the monitoring video, and a spread prediction model is established based on the motion features to predict the path and time of fire spread. An early warning signal is issued according to the analysis results of the spread prediction model, including: Step S34: detecting the fireworks features based on the detection model, and acquiring continuous key frame images from the surveillance video; Step S35: Calculate the movement direction or speed of the smoke feature or flame feature using an optical flow method, and output a corresponding motion feature vector; Step S36: The motion feature vectors of each continuous key frame image are organized into time series data in chronological order, and the time series data is processed through an RNN network to establish the fire spread prediction model, predict the direction and path of the fire spread, and issue the early warning signal for warning based on the analysis results.

6. The multimodal fusion embedded edge computing method according to claim 5, characterized in that: In step S4, the fusion processing module accesses one or more sensors to obtain perception information, and establishes a dynamic threshold adjustment model based on environmental parameters to optimize the warning trigger threshold, including: Step S41: connecting the plurality of sensors to the fusion processing module via a wireless or wired communication interface, and transmitting the collected sensing information to the fusion processing module via a communication protocol, wherein the sensors include a temperature and humidity sensor, a smoke sensor, and a wind speed sensor; Step S42: Initializing the initial threshold of the sensor, wherein the initial threshold is a safety standard value; Step S43: using multiple linear regression to establish a relationship model between the environmental parameters including temperature, humidity and wind speed and smoke concentration, and training the relationship model with historical data to calculate the influence coefficient corresponding to each environmental parameter; Step S44: inputting the environmental parameters collected in real time into the relationship model, calculating the corresponding adjustment range, and updating the safety standard value of the sensor.

7. The multimodal fusion embedded edge computing method according to claim 6, characterized in that: In step S4, the perception information exceeding the warning trigger threshold and the result of the AI ​​visual recognition module are transmitted to the fire linkage control module, and the fire linkage operation is triggered by verification through a double verification mechanism, including: Step S45: When the fire linkage control module receives the warning information from the AI ​​visual recognition module or the alarm information from the fusion processing module, it automatically reviews the alarm information or the warning information from the other module, triggers a local alarm, and pushes it to the monitoring center; Step S46: When the fire linkage control module receives the early warning information or the alarm information at the same time, it automatically links the fire fighting equipment, performs operations according to the return results of the detection model, and sends an emergency signal to the fire department.

8. A multimodal fusion embedded edge computing system integrating security and fire protection, adopting the multimodal fusion embedded edge computing method according to any one of claims 1 to 7, characterized in that: include: A deployment configuration module is used to select an AI visual recognition module suitable for edge computing scenarios, combine it with a fusion processing module that can access one or more types of sensors, and integrate the AI ​​visual recognition module, the fusion processing module, and the fire linkage control module into an embedded system and encapsulate it in an edge computing box; A dataset construction module is used to extract key frame images from the adaptively pre-processed surveillance video to construct a dataset, and to expand the dataset by simulating and generating fireworks images in different scenarios using a generative adversarial network; A target detection module is used for the AI ​​visual recognition model to use the improved YOLOv8 model as a detection model to train the dataset and perform target detection, return the category and location of the smoke and fire features in the image, and simultaneously use the optical flow method to extract the motion features of the flame or smoke in the consecutive key frame images in the surveillance video, and establish a spread prediction model based on the motion features to predict the path and time of fire spread, and issue an early warning signal based on the analysis results of the spread prediction model; The data processing module is used for the fusion processing module to access one or more of the sensors to obtain perception information, and to establish a dynamic threshold adjustment model based on environmental parameters to optimize the early warning trigger threshold, and transmit the perception information that exceeds the early warning trigger threshold and the AI ​​visual recognition result to the fire linkage control module, and verify through a double verification mechanism to trigger the fire linkage operation.

9. The multimodal fusion embedded edge computing system according to claim 8, characterized in that: The data set construction module includes: A data set collection unit is configured to acquire the surveillance video, perform dynamic denoising on the surveillance video using a wavelet transform or a non-local means denoising algorithm to suppress noise caused by environmental changes in the surveillance video; enhance the visibility of the fireworks features in a low-light environment using a Retinex algorithm; and identify dynamic areas in the surveillance video and perform ROI focusing based on background modeling technology. Extracting the key frame images from the surveillance video to construct the data set, wherein the key frame images include visible light images, infrared images, and thermal imaging images; A dataset expansion unit is used to simulate the changes in fireworks morphology under different environments and wind conditions based on the generative adversarial network to generate corresponding fireworks images for expanding the dataset; automatically annotate unlabeled dataset images using a pre-trained model, generate labels, and perform verification and validation; divide the dataset into a test set, a training set, and a validation set, and perform target detection through the detection model, wherein the training set is used for training to adjust weights and optimize parameters, the validation set is used for hyperparameter optimization during training, and the test set is used to evaluate model performance.

10. The multimodal fusion embedded edge computing system according to claim 9, characterized in that: The target detection module includes: a fireworks feature detection unit, configured for an input layer of the detection model to receive images in the training set as input; A feature extraction network is used to extract the fireworks features of the image from the training set, including thermal imaging temperature features, smoke features, and flame features, and multi-scale fusion is performed through a feature fusion network. A detection head introduces a CBAM attention mechanism to focus on the target area in the image and suppress background interference. The detection head performs the target detection to obtain the category and position of the fireworks features in the image. A fire spread prediction unit is configured to detect the smoke and fire features based on the detection model, obtain continuous key frame images from the surveillance video, calculate the movement direction or speed of the smoke or flame features by an optical flow method, and output corresponding motion feature vectors; organize the motion feature vectors of each continuous key frame image into time series data in chronological order, process the time series data through an RNN network, establish the fire spread prediction model, predict the direction and path of the fire spread, and issue the early warning signal for warning based on the analysis results.

Images