An AI-based linkage response and control system

Abstract

The present invention discloses a linkage response and control system based on AI, which belongs to the field of security monitoring technology and aims to solve the comprehensive and real-time collection, accurate analysis, timely warning and decision-making of environmental data to achieve early warning and rapid response of fire. It includes a data acquisition module, an AI analysis module, an AI recognition module, an AI decision module and an execution terminal robot; the data acquisition module is used to collect environmental data of monitoring sensors and surveillance cameras in the target scene; the AI ​​analysis module is used to process and analyze the environmental data obtained by the monitoring sensor, and output the sensor abnormal area; the AI ​​recognition module is used to perform real-time analysis on the environmental video stream obtained by the surveillance camera, and output the camera abnormal area; the AI ​​decision module is used to further evaluate the abnormal area and divide the abnormal level; the execution terminal robot is used to respond and control the abnormal area in the control room in real time according to the abnormal level.

Description

Technical Field

[0001] The present invention relates to the field of security monitoring technology, and more specifically to an AI-based linkage response and control system. Background Art

[0002] In recent years, the rapid development of technologies such as the Internet of Things, big data, and cloud computing has provided strong technical support for the application of artificial intelligence (AI) in the field of fire protection. By collecting a large amount of data through sensors and combining it with advanced AI algorithms for analysis, real-time monitoring and early warning of fire conditions can be achieved, thereby greatly improving the efficiency and accuracy of fire prevention and control. Internet of Things technology connects various sensor devices to the Internet to achieve real-time data collection and transmission. These sensors can monitor key indicators such as ambient temperature, smoke concentration, and gas composition, and immediately trigger an alarm mechanism once an abnormality is detected. Big data technology can store, process, and analyze these massive amounts of data to uncover potential fire risk factors. Cloud computing technology provides AI algorithms with powerful computing power and storage resources, making complex data analysis and model training possible.

[0003] The Chinese patent application with publication number CN118409520A discloses an integrated system for monitoring and early warning of safety risks based on AI, big data and the Internet of Things, including a collection end, an IoT intelligent control terminal for transmitting data from the collection end and issuing operation instructions, an operation processing end that receives the operation instructions of the IoT intelligent control terminal, and a display end. Although the existing technology can realize the simultaneous collection of multiple dangerous factors, realize automatic emergency disposal of terminal dangers, realize integrated synchronization of data and video, realize multi-party emergency joint defense and linkage, and realize government online inspection and supervision, through a set of integrated systems and a control terminal, it can enhance the regulatory department's ability to control enterprise safety production, solve the interconnection of multiple scenarios of enterprises, and reduce the cost of enterprises in the field of safety production. Through early warning and alarm of dangerous factors, it can automatically and quickly deal with them in the shortest time, eliminate hazards in the bud, and prevent and reduce the occurrence of accidents. It still fails to solve the problems of real-time and accuracy, especially in the early warning and rapid response to fires in complex environments. Therefore, in order to overcome these limitations, the present invention proposes a linkage response and control system based on AI. Summary of the invention

[0004] In view of the shortcomings of the prior art, the purpose of the present invention is to provide an AI-based linkage response and control system, which solves the problem of comprehensive and real-time collection, accurate analysis, timely warning and decision-making of environmental data to achieve early warning and rapid response to fire. By integrating data acquisition module, AI analysis module, AI recognition module, AI decision module and execution terminal robot, it automatically judges and outputs abnormal areas of monitoring sensors and abnormal areas of surveillance cameras, and formulates execution plans according to abnormal levels.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] An AI-based linkage response and control system, including a data acquisition module, an AI analysis module, an AI recognition module, an AI decision-making module and an execution terminal robot;

[0007] The data acquisition module is used to collect environmental data from monitoring sensors and surveillance cameras in the target scene;

[0008] The AI ​​analysis module is used to process and analyze the environmental data obtained by the monitoring sensors, identify abnormal trends, determine whether there are abnormalities in the environment, and output the sensor abnormal area;

[0009] The AI ​​recognition module is used to identify abnormal phenomena by real-time analysis of the environmental video stream obtained by the surveillance camera, and output the abnormal area of ​​the camera;

[0010] The AI ​​decision-making module is used to further evaluate the abnormal area and classify the abnormality level;

[0011] The execution terminal robot is used to respond and control abnormal areas in real time in the control room according to the abnormality level.

[0012] Specifically, the specific steps of sensor abnormal area judgment and output include:

[0013] Configuring anomaly thresholds ],in, is the number of monitoring sensor types, It is An abnormal threshold value of a monitoring sensor;

[0014] Determine whether the environmental data obtained by the monitoring sensor is abnormal according to the abnormal threshold. When the environmental data collected by the monitoring sensor is greater than the abnormal threshold, mark the environmental data point greater than the abnormal threshold as a potential abnormal point, and mark the sensor as a potential abnormal sensor;

[0015] Get the location coordinates of potential abnormal sensors [ X * , Y * , Z * ] , calculate the distance between the monitoring sensor and the potential abnormal sensor, and use a weighted method to focus on the horizontal distance between the monitoring sensors, that is:

[0016]

[0017] in, It is The distance between each monitoring sensor and the potential abnormal sensor, [X i , Y i , Z i ] It is The location coordinates of the monitoring sensors, is a weighting factor.

[0018] Specifically, the specific steps of sensor abnormal area judgment and output include:

[0019] Configuring distance threshold , filter the distance from the potential abnormal sensor within the distance threshold The monitoring sensors inside

[0020]

[0021] when When it is 1, it is marked as the target monitoring sensor W=[ W 1 , W 2 ,..., W m ] ,in is the number of target monitoring sensors within the distance threshold, It is target monitoring sensors;

[0022] Configuring Time Thresholds , obtain the environmental data of the target monitoring sensor within each distance threshold within the time threshold W T t =[ w t * - T t , w t * - T t +1 ,..., w t * ] ,in, is Environmental data at all times, and It is used to characterize the moment of acquiring environmental data within the time threshold, and calculate the difference between the environmental data and the abnormal threshold of the target monitoring sensor as the abnormal score.

[0023] Specifically, the specific steps of sensor abnormal area judgment and output include:

[0024] The abnormal score of each target monitoring sensor is standardized to eliminate the influence of different dimensions. The first target monitoring sensor Anomaly score , its standard value is calculated as follows:

[0025]

[0026] in, and They are The mean and standard deviation of the target monitoring sensor data;

[0027] Each normalized anomaly score is spatially weighted to take into account the impact of space. The spatially weighted anomaly score is:

[0028]

[0029] in, It is The first target monitoring sensor The normalized anomaly score The spatially weighted anomaly score of It is The normalized distance between the target monitoring sensor and the potential abnormal sensor, It is the set of distances between all target monitoring sensors and potential abnormal sensors;

[0030] Calculate the comprehensive anomaly index, accumulate and sum the spatial weighted anomaly scores of all target monitoring sensors to obtain the comprehensive anomaly index, and configure the evaluation threshold , when the comprehensive abnormal index Greater than the evaluation threshold When , the potential abnormal sensor is taken as the center and the distance threshold The area with a radius of is marked as the sensor abnormal area, and the center coordinates and radius of the sensor abnormal area are output; when the comprehensive abnormal index Less than or equal to the evaluation threshold When , the potential abnormal sensor is taken as the center and the distance threshold The area with a radius of is marked as the sensor normal area.

[0031] Specifically, the specific steps of abnormal area judgment and output of the surveillance camera include:

[0032] Obtain key frame images from the environmental video stream captured by the surveillance camera, and input the key frame images into the smoke detection model to determine whether there is an abnormal smoke area. If there is an abnormal smoke area, output the boundary information of the abnormal smoke area;

[0033] Filter the list of surveillance cameras in the area where the smoke abnormality is located, obtain the environmental video streams of the surveillance cameras in the list, extract information, and build a recognition data set;

[0034] Based on the recognition data set, the smoke diffusion speed and direction are determined, the fire source location is predicted, and the abnormal camera area is determined and output.

[0035] Specifically, the specific steps of the smoke detection model to identify abnormal smoke areas include:

[0036] Identify key frame images of the environment video stream;

[0037] Preprocess the key frame images to extract the characteristic information related to smoke;

[0038] The extracted feature information is input into the smoke detection model, and the output of the binary classification of the smoke detection model is converted into a probability value using the softmax function;

[0039] Configuring the judgment threshold , used to determine whether there is a smoke abnormal area in the current key frame image. When the probability value is greater than the judgment threshold , it is determined that there is an abnormal smoke area in the current key frame image, and it is marked as an abnormal image. The abnormal image, the time when the abnormal image was collected, and the location of the surveillance camera that collected the abnormal image are output.

[0040] Specifically, the specific steps of the smoke detection model to identify the abnormal smoke area also include:

[0041] The abnormal image is binarized, the smoke abnormal area is marked as a high brightness value, and the non-smoke area is marked as a low brightness value;

[0042] Perform connected domain analysis to obtain connected high-brightness areas, use depth-first search to traverse the abnormal image, mark the pixels belonging to the same connected domain, calculate the circumscribed rectangle of each connected domain as the bounding box of the smoke anomaly area, and record the coordinate position of the bounding box of each smoke anomaly area.

[0043] Specifically, the specific steps of predicting the fire source location include:

[0044] According to the order of surveillance camera IDs, key frame images and marking information with smoke abnormality area markings are obtained from the recognition data set, and the timestamp and coordinates of the smoke abnormality area boundary box are obtained;

[0045] Calculate the diffusion speed of the abnormal smoke area in the key frame images of adjacent time stamps respectively, and calculate the diffusion speed and the diffusion direction of the smoke according to the position change of the abnormal smoke area at different time points;

[0046] Calculate and identify the diffusion speed and direction of the abnormal smoke area in the key frame images collected by all surveillance cameras in the dataset;

[0047] Using the diffusion speed and diffusion direction of the smoke abnormal area, the initial diffusion speed and initial diffusion direction are estimated by weighted average, that is:

[0048]

[0049]

[0050] in, and are the initial diffusion velocity and initial diffusion direction, It is The weight of the timestamp, and It is The first The diffusion speed and direction, For the Surveillance camera ID The timestamp of the key frame image, is the number of abnormal smoke areas in the identified key frame images.

[0051] Specifically, the specific steps of predicting the fire source location also include:

[0052] Using the initial diffusion speed and initial diffusion direction, trace back from the last known smoke anomaly area to the fire source, that is:

[0053]

[0054]

[0055]

[0056] in, It is The initial location of the fire source estimated by the surveillance camera ID, It is The last location of the smoke abnormality area in the surveillance camera ID; is the time interval to the fire source;

[0057] The center point coordinates are calculated based on the initial position of the fire source estimated by all surveillance cameras as the estimated final fire source position. , and calculate the average diffusion speed based on the current diffusion speed and diffusion direction and the average diffusion direction ;

[0058] Configuring the Diffusion Threshold , used to measure the time interval for the smoke anomaly area to be effectively controlled, and calculate the final position where the diffusion of the smoke anomaly area is effectively controlled );

[0059] The center point coordinates are calculated based on the final position of the smoke abnormal area estimated by all surveillance cameras to be effectively controlled, and used as the final position coordinates of the smoke abnormal area estimated to be effectively controlled. ,by As a starting point, As the end point, a three-dimensional rectangular area is constructed as the camera abnormal area.

[0060] Specifically, the specific steps of abnormal level classification include:

[0061] When an abnormal area of ​​the monitoring sensor is detected, the monitoring results of the abnormal area of ​​the monitoring camera in the area are retrieved. If the monitoring camera does not find any abnormality, it is classified as a first-level abnormality, otherwise it is a second-level abnormality;

[0062] When an abnormal area is detected by the surveillance camera, the abnormal area monitoring results of the monitoring sensor in the area are retrieved. If the monitoring sensor does not find any abnormality, it is classified as a second-level abnormality, otherwise it is a third-level abnormality.

[0063] Beneficial effects of the present invention:

[0064] 1. Through the cooperation of the data acquisition module and the AI ​​analysis module, environmental data can be collected in real time, abnormal trends can be identified, and abnormal areas of sensors can be output in a timely manner. This enables early warnings to be issued at the early stages of safety incidents such as fires, greatly improving response speed and accuracy.

[0065] 2. The AI ​​recognition module uses the environmental video stream obtained by the surveillance camera to identify abnormal smoke areas through the smoke detection model and predict the location of the fire source. Combined with the fire source location prediction algorithm, it accurately locates the potential fire source and provides a scientific basis for emergency response. At the same time, the AI ​​decision-making module evaluates and grades the abnormal areas, formulates corresponding execution plans, and ensures the effective allocation and rapid disposal of resources.

[0066] 3. It integrates a variety of data processing technologies, such as spatial weighted processing, time series analysis, and deep learning, to conduct in-depth mining and comprehensive evaluation of monitoring data. By considering the impact of different dimensions, calculating the comprehensive anomaly index and evaluation threshold, it can comprehensively and objectively evaluate environmental conditions and improve the scientificity and reliability of decision-making. This multi-dimensional data analysis capability enables the system to maintain efficient operation in complex environments, providing strong support for the prevention and control of various security risks. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 It is a structural diagram of an AI-based linkage response and control system;

[0068] Figure 2 A flowchart of the specific steps of sensor abnormal area judgment and output;

[0069] Figure 3 Schematic diagram of sensor abnormal area division;

[0070] Figure 4 A flowchart of the specific steps for abnormal area judgment and output of the surveillance camera;

[0071] Figure 5 Schematic diagram of the camera abnormal area division. DETAILED DESCRIPTION

[0072] See also Figure 1 , this embodiment introduces an AI-based linkage response and control system, including a data acquisition module, an AI analysis module, an AI recognition module, an AI decision module and an execution terminal robot;

[0073] The data acquisition module is used to collect environmental data from monitoring sensors and surveillance cameras in the target scene;

[0074] The AI ​​analysis module is used to process and analyze the environmental data obtained by the monitoring sensors, identify abnormal trends, determine whether there are abnormalities in the environment, and output the sensor abnormal area;

[0075] The AI ​​recognition module is used to identify abnormal phenomena by real-time analysis of the environmental video stream obtained by the surveillance camera, and output the abnormal area of ​​the camera;

[0076] The AI ​​decision-making module is used to further evaluate the abnormal area and classify the abnormality level;

[0077] The execution terminal robot is used to respond and control abnormal areas in real time in the control room according to the abnormality level.

[0078] In this embodiment, a variety of environmental monitoring sensors and surveillance cameras are deployed in the target scene to monitor parameters such as air quality, temperature, humidity, and harmful gas concentration in real time. The monitoring sensors include temperature sensors, humidity sensors, and harmful gas sensors. The monitoring sensor data is collected at fixed time intervals, and the surveillance camera is recorded 24 hours a day. Real-time monitoring ensures continuous tracking of the target scene environment, and any potential risks or problems can be discovered in time, providing accurate data support for subsequent analysis and decision-making. The environmental data obtained by the monitoring sensor is processed and analyzed. The collected data is judged to be abnormal, and then by identifying abnormal trends, it is determined whether the environment where the abnormal data point is located is abnormal, and the abnormal area is output. Automated analysis reduces dependence on manual monitoring, improves the speed and accuracy of abnormal detection, and enables timely response to potential safety issues. At the same time, by performing real-time analysis on the environmental data obtained by the surveillance camera, identifying abnormal phenomena, using image processing technology, such as deep learning models, to identify the appearance of flames or smoke, and output abnormal areas. Using advanced image processing technology, the AI ​​recognition module can identify the location of the fire source as soon as the fire occurs, which is crucial for taking emergency measures quickly. Further evaluate the abnormal area, divide the abnormal level, and formulate an execution plan according to the abnormal level. Exemplarily, a central control display is set in the control room, and the central control display obtains environmental data and displays the real-time picture taken by the surveillance camera in the central control display. The execution terminal robot is set as a humanoid robot, and a control unit for executing the fire scene linkage fire fighting facilities is configured in the control room. When responding and controlling, the execution terminal robot obtains the abnormal level, analyzes the abnormal level, and controls the corresponding control unit according to the abnormal level to perform fire control, so that the execution terminal robot quickly knows the location and abnormal level of the abnormal area through the camera picture in the central control display, and replaces people to perform a rapid fire linkage response. Exemplarily, when the execution terminal robot responds according to the abnormal level, for the first level abnormality, the preliminary inspection procedure is started, and the relevant equipment is quickly diagnosed. For the second level abnormality, the fire plan is started, and the fire fighters are organized to carry out the initial fire fighting work. For the third level abnormality, the emergency response mechanism is started and support is requested; the execution terminal robot quickly locates the designated abnormal area according to the execution plan formulated by the AI ​​decision module, and responds and controls the abnormal area in real time. For example, the emergency broadcast system is activated to push fire information and escape guidance broadcasts, and automatically start fire water pumps, sprinkler pumps, fire shutters, fire doors and other fire protection systems and facilities.

[0079] See also Figure 2 Specifically, the specific steps of sensor abnormal area judgment and output include:

[0080] Configuring anomaly thresholds ],in, is the number of monitoring sensor types, is the abnormal threshold of the nth monitoring sensor;

[0081] Determine whether the environmental data obtained by the monitoring sensor is abnormal according to the abnormal threshold. When the environmental data collected by the monitoring sensor is greater than the abnormal threshold, mark the environmental data point greater than the abnormal threshold as a potential abnormal point, and mark the sensor as a potential abnormal sensor;

[0082] Get the location coordinates of potential abnormal sensors [ X * , Y * , Z * ] , calculate the distance between the monitoring sensor and the potential abnormal sensor, and use a weighted method to focus on the horizontal distance between the monitoring sensors, that is:

[0083]

[0084] in, It is The distance between each monitoring sensor and the potential abnormal sensor, [X i , Y i , Z i ] It is The location coordinates of the monitoring sensors, is a weight factor used to adjust the proportion of horizontal distance and vertical distance. The closer it is to 1, the greater the weight of the vertical distance; The closer it is to 0, the smaller the weight of the vertical distance;

[0085] Configuring distance threshold , filter the distance from the potential abnormal sensor within the distance threshold The monitoring sensors inside

[0086]

[0087] when When it is 1, it is marked as the target monitoring sensor W=[ W 1 , W 2 ,..., W m ] ,in is the number of target monitoring sensors within the distance threshold, It is target monitoring sensors;

[0088] Configuring Time Thresholds , obtain the environmental data of the target monitoring sensor within each distance threshold within the time threshold W T t =[ w t * - T t , w t * - T t +1 ,..., w t * ] ,in, is Environmental data at all times, and It is used to characterize the moment of acquiring environmental data within the time threshold, and calculate the difference between the environmental data and the abnormal threshold of the target monitoring sensor as the abnormal score to measure the degree of abnormality of the environmental data;

[0089] The abnormal score of each target monitoring sensor is standardized to eliminate the influence of different dimensions. The first target monitoring sensor Anomaly score , its standard value is calculated as follows:

[0090]

[0091] in, and They are The mean and standard deviation of the target monitoring sensor data;

[0092] Each normalized anomaly score is spatially weighted to take into account the impact of space. The spatially weighted anomaly score is:

[0093]

[0094]

[0095] in, It is The first target monitoring sensor The normalized anomaly score The spatially weighted anomaly score of It is The normalized distance between the target monitoring sensor and the potential abnormal sensor, It is the set of distances between all target monitoring sensors and potential abnormal sensors;

[0096] Calculate the comprehensive anomaly index, and sum up the spatial weighted anomaly scores of all target monitoring sensors to obtain the comprehensive anomaly index, that is:

[0097]

[0098] in, Is a comprehensive anomaly index, configure the evaluation threshold , when the comprehensive abnormal index Greater than the evaluation threshold When , the potential abnormal sensor is taken as the center and the distance threshold The area with a radius of is marked as the sensor abnormal area, and the center coordinates and radius of the sensor abnormal area are output; when the comprehensive abnormal index Less than or equal to the evaluation threshold When , the potential abnormal sensor is taken as the center and the distance threshold The area with a radius of is marked as the sensor normal area.

[0099] In this embodiment, temperature sensors, humidity sensors and harmful gas concentration sensors are used to monitor the environmental quality of the target scene. Each sensor is arranged at different positions of the target scene to collect environmental data in real time. The use of multiple types of sensors enables comprehensive monitoring of environmental conditions and timely detection of various potential problems. First, the abnormal threshold is configured for the monitoring sensor. For example, for the temperature sensor, if the abnormal threshold is set to 40°C, when the temperature detected by a temperature sensor in the target scene exceeds 40°C, the data point is marked as a potential abnormal point, and the sensor is marked as a potential abnormal sensor. The position coordinates of the potential abnormal sensor are obtained to quickly screen out possible problem areas and provide a basis for subsequent analysis. Due to the irregularity of the installation position and the inconsistency of the hanging height of the monitoring sensor, the weighted distance is used to reduce the influence of the hanging height on the distance. By calculating the distance between the monitoring sensor and the potential abnormal sensor, the monitoring sensor within a certain range from the potential abnormal sensor is screened as the target monitoring sensor, and the environmental data within a certain time range is obtained. Through distance screening, the target monitoring sensor that is most likely to be affected by the abnormality is focused on, which reduces the interference of irrelevant data and improves the accuracy of the analysis. For each target monitoring sensor, the difference between its environmental data and the corresponding abnormal threshold is calculated as the abnormality score. The anomaly scores are normalized to eliminate the influence of different dimensions. Spatial weighting is performed based on the distance of each target monitoring sensor from the potential anomaly sensor to obtain a spatially weighted anomaly score. The anomaly score reflects the degree of deviation of each target monitoring sensor from the anomaly threshold, providing a quantitative indicator for evaluating the degree of anomaly of environmental data. Standardization ensures that different types of data are compared on the same scale, enhancing universality and fairness. Spatial weighting takes into account the influence of geographic location, so that sensor readings close to known anomaly points have a greater impact on the final result, thereby more accurately reflecting the actual situation. The spatially weighted anomaly scores of all target monitoring sensors are accumulated and summed to obtain a comprehensive anomaly index. The comprehensive anomaly index provides a global perspective to evaluate the abnormal conditions of the entire area, making decisions more comprehensive and objective. See Figure 3 If the comprehensive abnormality index is greater than the preset evaluation threshold, the area with the potential abnormal point as the center and the distance threshold as the radius is determined to be the abnormal area. By accurately defining the scope of the abnormal area, relevant personnel can be guided to quickly locate the problem and take corresponding countermeasures.

[0100] See also Figure 4 Specifically, the specific steps of abnormal area judgment and output of the surveillance camera include:

[0101] Obtain key frame images from the environmental video stream captured by the surveillance camera, and input the key frame images into the smoke detection model to determine whether there is an abnormal smoke area. If there is an abnormal smoke area, output the boundary information of the abnormal smoke area;

[0102] Filter the list of surveillance cameras in the area where the smoke abnormality is located, obtain the environmental video streams of the surveillance cameras in the list, extract information, and build a recognition data set;

[0103] Based on the recognition data set, the smoke diffusion speed and direction are determined, the fire source location is predicted, and the abnormal camera area is determined and output.

[0104] In this embodiment, the environmental video stream captured by the surveillance camera is input into the trained smoke detection model. The model will automatically identify the abnormal smoke area according to the input image data, analyze the results of the model output, and extract the location, size and other information of the smoke to help more accurately determine the location of the fire source and the abnormal area. According to the location information of the abnormal smoke area, the list of surveillance cameras in the area where the area is located is screened, and the environmental video stream is obtained from the screened cameras to construct a recognition data set containing multiple perspectives to contain more comprehensive information, which helps to more accurately determine the location of the fire source and the abnormal area. Analyze the position changes of smoke at different time points, and infer the diffusion speed and direction of smoke by judging the diffusion trend of smoke. According to the diffusion trend of smoke, predict the possible location of the fire source. In order to discover potential fire hazards in advance and take corresponding preventive measures. According to the predicted location of the fire source, divide the abnormal area. These areas are areas with higher fire risks, and fire prevention plans need to be activated to reduce the harm caused by the fire source.

[0105] Specifically, the construction and training of the smoke detection model includes:

[0106] Collect a dataset containing smoke and non-smoke images, and label each image, including smoke and non-smoke; the dataset images can come from public datasets, web crawlers, or self-collection to ensure the diversity and representativeness of the dataset, covering the smoke forms and characteristics in different scenarios.

[0107] Build and train the smoke detection model, select target detection algorithms, including the YOLO series target detection algorithms, convolutional neural networks, Faster R-CNN, and RetinaNet, and configure the model's structure and parameters; divide the image dataset into training set, validation set, and test set. The training set is used to train the model, the validation set is used to adjust hyperparameters and prevent overfitting, and the test set is used to evaluate model performance. Select loss functions and optimization algorithms for model training. For example, loss functions include cross entropy loss function and mean square error; optimization algorithms include Adam optimizer and stochastic gradient descent. Monitor indicators such as loss value and accuracy during training, and adjust hyperparameters as needed, including learning rate, batch size, etc. to improve model performance.

[0108] Specifically, the specific steps of the smoke detection model to identify abnormal smoke areas include:

[0109] Identify key frame images of the environment video stream, which is used to select frame images that can represent scene changes according to the motion or change of the video stream content, and apply motion detection algorithms, including optical flow method, background subtraction and inter-frame difference to detect motion and identify and output key frame images; Apply motion detection algorithms to identify key frame images in the environment video stream, which not only improves processing efficiency, but also retains important information that can represent scene changes;

[0110] Preprocess the key frame images, including denoising and contrast enhancement; extract smoke-related feature information, including color, texture, and shape; the feature information extracted after preprocessing the key frame images helps improve the judgment accuracy of the smoke detection model;

[0111] The extracted feature information is input into the smoke detection model, and the output of the binary classification of the smoke detection model is converted into a probability value using the softmax function. The probability value is used to indicate the possibility of smoke in the current key frame image.

[0112] Configuring the judgment threshold , used to determine whether there is a smoke abnormal area in the current key frame image. When the probability value is greater than the judgment threshold , it is determined that there is an abnormal smoke area in the current key frame image, and it is marked as an abnormal image, and the abnormal image, the time when the abnormal image was collected, and the location of the surveillance camera that collected the abnormal image are output;

[0113] The abnormal image is binarized, the smoke abnormal area is marked as a high brightness value, and the non-smoke area is marked as a low brightness value. A connected domain analysis is performed to obtain the connected high-brightness area. The abnormal image is traversed using a depth-first search, and the pixels belonging to the same connected domain are marked. The circumscribed rectangle of each connected domain is calculated as the bounding box of the smoke abnormal area, and the coordinate position of the bounding box of each smoke abnormal area is recorded. Through binarization and connected domain analysis, the position and size of the smoke abnormal area are accurately located, providing accurate basic data for subsequent analysis and processing.

[0114] The specific steps of constructing the recognition data set include:

[0115] Obtain the location information of the surveillance cameras in the target environment, including the surveillance camera ID and the area captured by the surveillance camera; to determine which surveillance cameras can cover the abnormal smoke area, so that the relevant video stream data can be accurately obtained later.

[0116] The location of the abnormal smoke area is matched with the monitoring range of each surveillance camera, and the surveillance cameras whose monitoring range overlaps with the abnormal smoke area are screened out. A list of surveillance cameras is generated, which includes the ID and location information of each camera; in order to further narrow the monitoring range and obtain surveillance cameras that may capture the abnormal smoke area.

[0117] Use the API interface provided by the surveillance camera to retrieve and obtain the real-time video stream of the surveillance camera in the list through the surveillance camera ID; to ensure that real-time video data related to the abnormal smoke area can be obtained, providing raw materials for subsequent analysis and processing.

[0118] Configuring the cutoff threshold , which is used to characterize the time period of the video stream for monitoring the abnormal smoke area; obtain the video stream data within the cutoff threshold and sort it according to the surveillance camera ID and timestamp to form an identification video stream data set, which is convenient for rapid retrieval and use when constructing the data set later, and filter out key frames from a large amount of video stream data to improve the efficiency of data processing.

[0119] The video stream in the recognition video stream dataset is processed, key frame images are extracted, and the key frame smoke abnormal area in the image is marked through the smoke detection model, and binarization processing and connected domain analysis are performed to record the coordinate position of the boundary box of the smoke abnormal area;

[0120] The key frame images with smoke abnormal area marks in the recognition video stream data set and their corresponding annotation information constitute the recognition data set. The processed key frame images and their corresponding annotation information are combined together to form a complete recognition data set.

[0121] Specifically, the specific steps of fire source location prediction include:

[0122] According to the order of surveillance camera IDs, key frame images and marking information with smoke abnormality area markings are obtained from the recognition data set, and the timestamp and coordinates of the smoke abnormality area boundary box are obtained;

[0123] Calculate the diffusion speed of the abnormal smoke area in the key frame images of adjacent time stamps respectively, and calculate the diffusion speed according to the position change of the abnormal smoke area at different time points, that is:

[0124]

[0125] in, Timestamp is The spatial coordinate system of the smoke anomaly area Axial diffusion velocity, Timestamp is The spatial coordinate system of the smoke anomaly area Axial diffusion velocity, Timestamp is The spatial coordinate system of the smoke anomaly area Axial diffusion velocity, are the adjacent timestamps of the keyframe images, and , The timestamp is The reference point of the smoke abnormality area of ​​the key frame image, the reference point includes the center, the centroid or any fixed point, The timestamp is The reference point of the smoke abnormality area in the key frame image;

[0126] According to the position change trend of the abnormal smoke area at different time points, the diffusion direction of the smoke is calculated, that is:

[0127]

[0128]

[0129]

[0130] in, ) is the timestamp The diffusion direction of the smoke anomaly area, The timestamp is The reference point of the smoke abnormal area in the key frame image, The timestamp is The reference point of the smoke abnormality area in the key frame image;

[0131] Calculate the diffusion speed of the abnormal smoke area in the key frame images collected by all surveillance cameras in the recognition data set V ID q =[ v t 2 q , v t 3 q ,..., v t l q ] and diffusion direction θ ID q =[ θ t 2 q , θ t 3 q ,..., θ t l q ] ,in and are the diffusion speed and diffusion direction calculated by the qth surveillance camera ID, is the ID of the qth surveillance camera The diffusion rate, is the ID of the qth surveillance camera diffusion direction, For the Surveillance camera ID The timestamp of the key frame image, is the number of smoke abnormal areas in the identified key frame image;

[0132] Using the diffusion speed and diffusion direction of the smoke abnormal area, the initial diffusion speed and initial diffusion direction are estimated by weighted average, that is:

[0133]

[0134]

[0135] in, and are the initial diffusion velocity and initial diffusion direction, It is The weight of the key frame image timestamp is a constant greater than 0. The larger the The diffusion speed and diffusion direction of each timestamp have a greater impact on the diffusion of the smoke anomaly area, and the timestamp closer to the initial smoke anomaly area has a higher weight; and They are The first The diffusion speed and diffusion direction of the abnormal smoke area in the key frame image, For the Surveillance camera ID The timestamp of the key frame image, is the number of smoke abnormal areas in the identified key frame image;

[0136] Using the initial diffusion speed and initial diffusion direction, trace back from the last known smoke anomaly area to the fire source, that is:

[0137]

[0138]

[0139]

[0140] in, It is The initial location of the fire source estimated by the surveillance camera ID, It is The last location of the smoke abnormality area in the surveillance camera ID; is the time interval to the fire source;

[0141] The center point coordinates are calculated based on the initial position of the fire source estimated by all surveillance cameras as the estimated final fire source position. , and calculate the average diffusion speed based on the current diffusion speed and diffusion direction and the average diffusion direction ;

[0142] Configuring the Diffusion Threshold , which is used to measure the time interval for the smoke anomaly area to be effectively controlled and calculate the final position where the diffusion of the smoke anomaly area is effectively controlled, that is:

[0143]

[0144]

[0145]

[0146] in, ) is the The final position where the diffusion of the abnormal smoke area estimated by the surveillance camera ID is effectively controlled, ) is the Surveillance camera ID The position of the abnormal smoke area in the key frame image, For the Surveillance camera ID The timestamp of the key frame image, is the average diffusion velocity, is the average diffusion direction, is a positive constant used to adjust the range of the smoke anomaly area;

[0147] See also Figure 5 , according to the final position of the smoke abnormal area estimated by all surveillance cameras to be effectively controlled, the center point coordinates are calculated as the final position coordinates of the estimated final smoke abnormal area where the diffusion is effectively controlled ,by As a starting point, is the end point, and because The existence of can ensure that the starting point and the end point are not in the same plane. Therefore, a three-dimensional rectangular area is constructed based on the starting point and the end point as the camera abnormal area.

[0148] Specifically, the specific steps of abnormal level classification include:

[0149] When an abnormal area of ​​the monitoring sensor is detected, the monitoring results of the abnormal area of ​​the monitoring camera in the area are retrieved. If the monitoring camera does not find any abnormality, it is classified as a first-level abnormality, otherwise it is a second-level abnormality;

[0150] When an abnormal area is detected by the surveillance camera, the abnormal area monitoring results of the monitoring sensor in the area are retrieved. If the monitoring sensor does not find any abnormality, it is classified as a second-level abnormality, otherwise it is a third-level abnormality.

[0151] In this embodiment, when a monitoring sensor detects an abnormality in a certain area, the monitoring sensor abnormal area is demarcated, and the surveillance camera records in the area are automatically retrieved to check whether there are visible abnormal conditions, including flames, smoke, etc. If the surveillance camera does not find any abnormality, then the abnormality is classified as a first-level abnormality. If the surveillance camera finds an abnormality and there is an abnormal surveillance camera area, it is classified as a second-level abnormality.

[0152] When a surveillance camera detects an abnormality in a certain area, there is an abnormal area of ​​the surveillance camera, and the sensor data of the abnormal area is automatically checked to determine whether there is an abnormality. If the sensor does not find an abnormality, the abnormality found by the surveillance camera is classified as a secondary abnormality. If the sensor also detects an abnormality, the abnormal area found by the surveillance camera is classified as a tertiary abnormality. In this way, the data of different sensors can be cross-validated to more accurately classify abnormal situations, so that corresponding countermeasures can be taken.

[0153] Specifically, real-time response and control of abnormal areas include:

[0154] For the first level anomaly, it means that the sensor has detected potential danger signals in the target scene, including smoke, high temperature or increased concentration of harmful gases. Start the preliminary inspection procedure, quickly diagnose the relevant equipment to determine whether it is a false alarm or equipment failure, notify the relevant personnel to conduct on-site verification, and inform the relevant departments and personnel of the abnormal situation to ensure timely handling of potential dangers;

[0155] For the second level abnormality, it means that a small fire has occurred, but the fire may not have developed into a large-scale fire. Execute all measures for the first level abnormality, and immediately activate the firefighting plan, strengthen monitoring, notify the security personnel in the abnormal area to prohibit the initial firefighting, and evacuate the surrounding personnel and materials to prevent the fire from expanding;

[0156] Level 3 abnormalities indicate that the fire has grown and may be spreading rapidly. The emergency response mechanism is activated to generate an escape plan, notify additional technical support, security experts, and emergency maintenance teams, contact the fire department to report the fire and request support, and organize the evacuation of personnel through broadcasting while waiting for the arrival of firefighters.

[0157] In this embodiment, for the first-level abnormality, the preliminary inspection procedure is immediately started, the relevant equipment is quickly diagnosed to determine whether there is a false alarm or equipment failure, and the relevant personnel are notified to conduct on-site verification to ensure that potential dangers are handled in a timely manner. Increase the monitoring frequency, strengthen the monitoring of the abnormal area, and increase the acquisition frequency of the data acquisition module to capture environmental changes more timely. Inform the relevant departments and personnel of the abnormal situation to ensure that all relevant personnel are aware of the current situation. Record the details of the abnormal event, including the time, location, nature and handling results, and report to the upper management. For the second-level abnormality, all measures for the first-level abnormality are implemented, more detailed data analysis and equipment inspection are added, and monitoring is strengthened to prevent small fires from developing into larger fires. Immediately start the fire prevention plan, including opening the fire sprinkler for initial fire extinguishing, closing the fire doors adjacent to the abnormal area, ensuring that the fire passage is unobstructed, organizing firefighters to carry out initial fire extinguishing, and preparing to evacuate surrounding personnel and materials, continuously monitoring the situation in the abnormal area, and sending the latest monitoring results to relevant personnel. Record all operation steps and results, and update the abnormal status and processing progress in a timely manner. For level 3 abnormalities, all measures for level 1 and level 2 abnormalities are implemented, the emergency response mechanism is activated immediately, the fire department is contacted immediately, the fire is reported and support is requested, and an emergency evacuation notice is issued to all personnel through the broadcasting system; through the response measures under each abnormal level, the importance of rapid response, detailed records, effective communication, and professional intervention is emphasized. Through the hierarchical response mechanism, potential losses and risks can be minimized while ensuring the safety of all relevant personnel.

[0158] Preferably, the generation of the personnel escape plan includes:

[0159] When a level 3 abnormality occurs, the video data of the cameras in the abnormal area of ​​the monitoring sensor and the abnormal area of ​​the surveillance camera are retrieved for real-time fire analysis, including the location of the fire, the size of the fire, the smoke concentration, and the distribution of people; in order to understand the on-site conditions and ensure a rapid response and accurate judgment of the fire situation.

[0160] Utilize video data analysis technology and target recognition to obtain all personnel information in the abnormal area, including the number of personnel, their location, and their movement trajectory; personnel information is the basis for formulating effective escape plans, ensuring a comprehensive understanding of personnel situations and timely response.

[0161] According to the number and location of identified personnel, combined with building information, including exit locations, stair locations, and fire-fighting facility locations, and based on the fire situation analysis results, an escape plan is generated by executing the terminal robot's built-in path planning algorithm, ensuring the safety and efficiency of the escape route and improving the efficiency and safety of personnel evacuation.

[0162] Exemplarily, generating an escape plan through a path planning algorithm includes:

[0163] Obtain the 3D environment model of the target scene, retrieve the 3D environment where the abnormal area is located, and extract all spatial coordinate information, including the exit location, staircase location, and specific location of fire-fighting facilities;

[0164] Based on the personnel information, in the 3D environment where the abnormal area is located, a human role model with a complex bone structure is created, and natural and smooth animation effects are added to it; by creating a human role model with a complex bone structure and adding natural and smooth animation effects to it, the behavior pattern of real people in an emergency can be simulated. This helps to more realistically reflect the reactions and actions of people in an emergency, thereby providing a reference for formulating more effective escape plans.

[0165] Each character model is regarded as an independent intelligent agent to simulate the behavior patterns of large-scale crowds in emergency situations, and the intelligent agent is able to perceive changes in the surrounding environment and respond accordingly. By allowing the intelligent agent to perceive changes in the surrounding environment and respond accordingly, the actions of people in emergency situations can be more accurately predicted, providing a basis for formulating more effective escape plans.

[0166] Configure safety criteria and generate multiple preliminary escape routes based on safety standards, personal characteristics and special needs. Safety criteria include minimizing casualties and escaping as quickly as possible. Personal characteristics include age, height, and special needs, including that the elderly, children or disabled people may need easier-to-walk routes or need the support of auxiliary equipment. By configuring safety criteria and generating multiple preliminary escape routes based on safety standards, personal characteristics and special needs, the safety and feasibility of the escape routes can be ensured. At the same time, generating personalized escape routes based on personal characteristics such as age and height can better meet the needs of different people and thus improve the overall evacuation efficiency. In addition, providing easier-to-walk routes or support of auxiliary equipment for people with special needs such as the elderly, children or disabled people can ensure that they can evacuate safely.

[0167] Recreate the crowd escape scenario driven by intelligent agents in a virtual 3D environment, dynamically adjust the action trajectory of each intelligent agent according to the escape route, including individual action mode, selected path and its change over time, evaluate the pros and cons of different preliminary escape routes, and select the final escape route. By observing the individual action mode, selected path and its change over time, the pros and cons of different preliminary escape routes can be evaluated to select the most suitable escape route. This helps to improve the accuracy and efficiency of decision-making.

[0168] The generated escape plan is communicated to all personnel in the target area through the broadcasting system and mobile phone text messages; during the evacuation process, the situation in the abnormal area is continuously monitored, and the escape plan is updated based on the latest personnel information, and personnel are re-notified and guided to evacuate; ensuring the timely transmission of information and the orderly evacuation of personnel, so that everyone can act according to the latest plan.

[0169] Record all key information of the entire evacuation process, including start time, end time, number of participants, evacuation route, etc., to facilitate post-analysis and improvement of emergency response plans. At the same time, provide detailed reports to senior management for further evaluation and decision-making, providing valuable experience and reference for subsequent emergency response.

[0170] Specifically, the execution terminal robot realizes the following functions: including seeing, hearing, speaking, and moving;

[0171] Exemplarily, the terminal robot determines whether there is a fire in the monitoring area through real-time video monitoring; Exemplarily, the terminal robot watches the video in real time through the monitoring camera and uses the algorithm to detect whether there is an abnormal smoke area. At the same time, the monitoring sensor collects environmental data, such as smoke concentration and temperature, to assist in determining whether there is a fire risk; when the smoke concentration exceeds the preset threshold or the temperature rises abnormally, the system will trigger an alarm mechanism to notify relevant personnel to take emergency measures.

[0172] The program algorithm is used to determine whether the monitoring channel is blocked or locked. For example, the execution terminal robot can be used to control the fire door in the target scene. When an abnormal situation is detected, the fire door can be opened and closed intelligently, and the camera monitoring data can be combined to determine whether the fire passage is unobstructed and whether the fire door is blocked. When the monitoring camera captures that the fire passage is blocked by objects or the fire door is illegally locked, the system will automatically issue an alarm and notify the management personnel to handle it.

[0173] Monitor whether the battery vehicles are illegally parked or charged in public areas; for example, the execution terminal robot can monitor the parking and charging of battery vehicles in real time through the surveillance cameras installed in the public areas. When the battery vehicles are found to be illegally parked or charged, the violation can be automatically recorded and the management personnel can be notified for handling. In addition, relevant regulations and precautions can be communicated to the public through voice prompts or display screens.

[0174] Whether there are people smoking in no-smoking places, whether there are people using fire in violation of regulations; for example, the execution terminal robot can use image recognition technology to analyze the monitoring screen to detect whether there are people smoking or using fire in no-smoking places. Once a violation is found, an alarm can be automatically issued and the management personnel can be notified to handle it.

[0175] Check the system self-check status in real time; for example, the execution terminal robot has a self-check function, which can check the operating status and performance indicators of its own system in real time. By performing self-check operations regularly, the stability and reliability of the system are ensured. Once an abnormal situation or fault problem is found, an alarm can be automatically issued and the maintenance personnel can be notified to handle it.

[0176] Exemplarily, listening includes: intelligently answering fire, emergency rescue and help calls in the fire monitoring room; exemplary, when someone calls the fire monitoring room, the call can be automatically answered through voice recognition technology, and corresponding help or guidance can be provided according to the caller's needs.

[0177] Real-time monitoring of abnormal signals in the monitoring room, and voice notification of relevant personnel to confirm the alarm situation after abnormal signals are found; for example, when the system detects that the smoke concentration exceeds the standard signal emitted by the smoke detector, it will automatically trigger the voice notification function, alert nearby security personnel and instruct them to go to the scene for confirmation. At the same time, all abnormal signal events are recorded for subsequent safety analysis and improvement.

[0178] For example, it includes: after confirming the fire alarm, calling the internal personnel to report the fire; for example, once the monitoring system confirms the abnormal area through video analysis or other sensor data, it will automatically call the preset internal contact number to notify the relevant personnel to take emergency actions. For example, according to the pre-set priority list, the department head, security personnel and fire department are called in turn to ensure the timely transmission of information.

[0179] After confirming the occurrence of a fire, the emergency broadcast system is automatically activated to push fire information and escape guidance broadcasts; for example, when a fire is confirmed, it is immediately switched to the emergency broadcast mode, and the fire alarm is broadcast to the personnel in the entire building through the speaker system, and clear escape route guidance and other safety instructions are provided. At the same time, according to the specific location and scale of the fire, the broadcast content and frequency are dynamically adjusted to ensure the effectiveness and coverage of the information.

[0180] According to the user's voice or interface input, the robot automatically answers and displays relevant fire information content; for example, the execution terminal robot has an interactive interface, through which the user can make query requests, such as asking for the location of the nearest fire extinguisher or how to close the gas valve in a specific area. The execution terminal robot can use the built-in knowledge base and natural language processing capabilities to provide accurate answers and operation guides. In addition, it can continuously optimize its own question and answer library based on user feedback to improve service quality.

[0181] For example, the actions include: if it is not a fire alarm, the alarm will be automatically muted and the smoke alarm controller will be reset; for example, if it is initially determined that the abnormal area is a false alarm or smoke caused by a non-fire situation, the alarm will be muted and the smoke detector will be remotely controlled to reset and restore its normal working state. At the same time, all the details of the false alarm event, including time, location, cause, etc., will be recorded to facilitate subsequent technical analysis and optimization.

[0182] After confirming the fire, automatically start the fire pump, sprinkler pump, fire shutter, fire door and other fire protection systems and facilities; for example, when the fire is confirmed, immediately activate all fire response measures, including starting the fire pump to supply water to the fire extinguishing system, starting the sprinkler pump to start automatic water spraying to extinguish the fire, lowering the fire shutter to isolate the spread of fire, and closing the fire door to prevent the spread of smoke. At the same time, the operating status of these devices will be monitored in real time to ensure that they can work normally and achieve the expected results.

[0183] The execution terminal robot can monitor the flame situation in the monitoring area in real time by seeing, hearing, speaking and moving. Once a flame is detected, an alarm is issued and relevant personnel are notified to handle it. This realizes the intelligent flame recognition, greatly improves the accuracy and timeliness of fire warning, and effectively prevents the occurrence of fire accidents.

[0184] The execution terminal robot can accurately identify the smoke generated by the fire through highly sensitive smoke sensors and data analysis technology. When the sensor data such as smoke concentration exceeds the preset threshold, the alarm mechanism can be automatically triggered to notify relevant personnel to take emergency measures. This realizes intelligent fire smoke recognition, which can detect the fire source in the early stage of the fire and buy precious time for fire fighting and rescue.

[0185] The execution terminal robot can analyze the real-time video of the monitoring area and perform target recognition. It can automatically identify whether the passage is blocked by objects or illegally occupied. Once a passage is blocked, an alarm will be immediately issued and the management personnel will be notified to handle it. This will help keep the fire passage unobstructed and ensure that people can be evacuated quickly in an emergency.

[0186] The execution terminal robot can monitor public places in real time and judge smoking behavior through image recognition. Once someone is found smoking in a non-smoking area, it will immediately issue an alarm and notify the management staff to dissuade and deal with the situation. This is to maintain a smoke-free environment in public places and protect the health and safety of the public.

[0187] The execution terminal robot can automatically identify the parking and charging status of battery vehicles by analyzing the real-time video of the monitoring area. When a battery vehicle is found to be parked or charged illegally, the violation is automatically recorded and the management personnel are notified to handle it. In addition, relevant regulations and precautions can be communicated to the public through voice prompts or display screens. This battery vehicle identification technology helps to standardize the parking and charging behavior of battery vehicles and reduce safety hazards caused by battery vehicles.

[0188] When a fire occurs, the execution terminal robot can automatically answer fire, emergency rescue and help calls in the fire monitoring room, and provide corresponding help or guidance according to the caller's needs. At the same time, it can also automatically start firefighting facilities such as fire pumps and sprinkler pumps, and start fire emergency plans. In addition, it also has big data human-computer interaction functions, which can automatically answer and display relevant fire information content based on the user's voice or interface input. This technology for intelligently handling fire accidents can greatly improve the efficiency and accuracy of fire response and reduce losses and casualties caused by fire.

[0189] When a fire occurs, the executive terminal robot can immediately switch to emergency broadcast mode, broadcast the fire alarm to people throughout the building through the loudspeaker, and provide clear escape route guidance and other safety instructions. This sound evacuation escape system can effectively guide people to evacuate quickly and reduce casualties and losses caused by panic and confusion.

[0190] Users can make inquiries through the interactive interface of the execution terminal robot, such as asking for the location of the nearest fire extinguisher or how to close the gas valve in a specific area. The execution terminal robot uses the built-in knowledge base and natural language processing capabilities to provide accurate answers and operation guides. This human-computer interaction system can provide users with convenient fire information services and improve fire safety awareness and self-rescue capabilities.

[0191] Working principle and effect:

[0192] The present invention provides an AI-based linkage response and control system, including a data acquisition module, an AI analysis module, an AI recognition module, an AI decision-making module, and an execution terminal robot. It solves the problem of comprehensive and real-time collection, accurate analysis, timely warning and decision-making of environmental data to achieve early warning and rapid response to fire.

[0193] The data acquisition module is responsible for collecting environmental data from monitoring sensors and surveillance cameras in the target scene in real time. The AI ​​analysis module processes this data, determines whether there are any abnormalities in the environment by identifying abnormal trends, and outputs the sensor abnormal area; specifically, it configures abnormal thresholds, determines whether the data is abnormal based on these thresholds, marks potential abnormal points and sensors, and then calculates the distance between monitoring sensors and performs weighted processing to focus on the impact of horizontal distance. Next, a time threshold is configured to filter the environmental data of the target monitoring sensor within the time threshold, calculates the anomaly score and performs normalization processing to eliminate the impact of different dimensions. Finally, the spatial weighted anomaly scores of all target monitoring sensors are accumulated and summed to obtain a comprehensive anomaly index. When the comprehensive anomaly index exceeds the evaluation threshold, it is determined to be a sensor abnormal area, and the center coordinates and radius of the area are output.

[0194] The AI ​​recognition module identifies abnormal phenomena and outputs the abnormal area of ​​the camera by performing real-time analysis on the environmental video stream obtained by the surveillance camera. It extracts key frame images from the video stream and inputs them into the smoke detection model to determine whether there is an abnormal smoke area. If so, the boundary information of the abnormal smoke area is output, and the abnormal image is binarized and connected domain analysis is performed to determine the bounding box of the abnormal smoke area. Subsequently, the location of the fire source is predicted based on the recognition data set, and the initial diffusion speed and direction are estimated by weighted average by calculating the diffusion speed and direction of the abnormal smoke area, and then the location of the fire source is traced back from the last known abnormal smoke area position. At the same time, the diffusion threshold is configured to measure the time interval for the effective control of the abnormal smoke area, and the final position is calculated as the abnormal camera area.

[0195] The AI ​​decision-making module further evaluates and grades the abnormal areas based on the above analysis results. When an abnormal area of ​​the monitoring sensor is detected, the monitoring results of the abnormal area of ​​the monitoring camera in the area will be retrieved. If the monitoring camera does not find any abnormality, it will be classified as a level 1 abnormality; otherwise, it will be a level 2 abnormality. Similarly, when an abnormal area of ​​the monitoring camera is detected, the monitoring results of the abnormal area of ​​the monitoring sensor in the area will also be retrieved. If the monitoring sensor does not find any abnormality, it will be classified as a level 2 abnormality; otherwise, it will be a level 3 abnormality. Such a division helps to locate the source and severity of the problem more accurately, thereby formulating more effective response measures.

[0196] The execution terminal robot performs different levels of response and control according to the abnormality level: for the first level abnormality, the preliminary inspection procedure is initiated, the relevant equipment is quickly diagnosed to determine whether it is a false alarm or equipment failure, the relevant personnel are notified to conduct on-site verification, and the abnormal situation is reported to the relevant departments and personnel; for the second level abnormal response, all measures for the first level abnormality are implemented, and the fire plan is immediately initiated, monitoring is strengthened, and the security personnel in the abnormal area are notified to prohibit initial firefighting and evacuate the surrounding personnel and materials; for the third level abnormality, the emergency response mechanism is initiated, additional technical support, security experts and emergency maintenance teams are notified, the fire department is contacted to report the fire and request support, and personnel are evacuated through broadcasting while waiting for the arrival of firefighters; appropriate response and control are carried out for abnormal situations of different levels. These measures have shown good results in improving response speed, ensuring personnel safety, mobilizing necessary resources, and ensuring the timeliness and accuracy of information notification.

[0197] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. An AI-based linkage response and control system, characterized in that: Including data collection module, AI analysis module, AI recognition module, AI decision module and execution terminal robot; The data acquisition module is used to collect environmental data of monitoring sensors and surveillance cameras in the target scene; The AI ​​analysis module is used to process and analyze the environmental data acquired by the monitoring sensor, determine whether the environment is abnormal by identifying abnormal trends, and output the sensor abnormal area; The AI ​​recognition module is used to identify abnormal phenomena by real-time analysis of the environmental video stream obtained by the surveillance camera, and output the abnormal area of ​​the camera; The AI ​​decision module is used to further evaluate the abnormal area, which includes the sensor abnormal area and the camera abnormal area, and divide the abnormal level; The execution terminal robot is used to respond and control the abnormal area in real time in the control room according to the abnormality level; The specific steps of determining and outputting the abnormal area of ​​the sensor include: Configure an abnormal threshold, and judge whether the environmental data obtained by the monitoring sensor is abnormal according to the abnormal threshold. When the environmental data collected by the monitoring sensor is greater than the abnormal threshold, the environmental data point greater than the abnormal threshold is marked as a potential abnormal point, and the sensor is marked as a potential abnormal sensor; Obtain the location coordinates of the potential abnormal sensor, calculate the distance between the monitoring sensor and the potential abnormal sensor, configure the distance threshold, and select the monitoring sensor whose distance from the potential abnormal sensor is within the distance threshold as the target detection sensor; Configuring Time Thresholds , obtain the environmental data of the target monitoring sensor within each distance threshold within the time threshold ,in, is Environmental data at all times, and It is used to characterize the moment of acquiring environmental data within the time threshold, and calculate the difference between the environmental data and the abnormal threshold of the target monitoring sensor as the abnormal score; The anomaly score of each target monitoring sensor is normalized, and each normalized anomaly score is spatially weighted; The spatially weighted anomaly scores of all target monitoring sensors are cumulatively summed to obtain a comprehensive anomaly index; Configure an evaluation threshold. When the comprehensive abnormality index is greater than the evaluation threshold, mark the area with the potential abnormal sensor as the center and the distance threshold as the radius as the sensor abnormal area, and output the center coordinates and radius of the sensor abnormal area; The specific steps of determining and outputting abnormal areas of the surveillance camera include: Obtain key frame images from the environmental video stream captured by the surveillance camera, and input the key frame images into the smoke detection model to determine whether there is an abnormal smoke area. If there is an abnormal smoke area, output the boundary information of the abnormal smoke area; Filter the list of surveillance cameras in the area where the smoke abnormality is located, obtain the environmental video streams of the surveillance cameras in the list, extract information, and build a recognition data set; According to the recognition data set, the smoke diffusion speed and direction are determined, the fire source location is predicted, and the abnormal camera area is determined and output; The specific steps of predicting the fire source location include: According to the order of surveillance camera IDs, key frame images and marking information with smoke abnormality area markings are obtained from the recognition data set, and the timestamp and coordinates of the boundary box of the smoke abnormality area are obtained; Calculate the diffusion speed of the abnormal smoke area in the key frame images of adjacent time stamps respectively, and calculate the diffusion speed and the diffusion direction of the smoke according to the position change of the abnormal smoke area at different time points; Calculate the diffusion speed and diffusion direction of the smoke abnormal area in the key frame images collected by all surveillance cameras in the recognition data set, and estimate the initial diffusion speed and initial diffusion direction by weighted average; Using the initial diffusion speed and initial diffusion direction, trace back from the last known abnormal smoke area to the fire source location; The center point coordinates are calculated based on the initial position of the fire source estimated by all surveillance cameras as the estimated final fire source position. , and calculate the average diffusion speed and average diffusion direction according to the current diffusion speed and diffusion direction; Configure the diffusion threshold to measure the time interval for the smoke anomaly area to be effectively controlled and calculate the final position where the diffusion of the smoke anomaly area is effectively controlled; The center point coordinates are calculated based on the final position of the smoke abnormal area estimated by all surveillance cameras to be effectively controlled, and used as the final position coordinates of the smoke abnormal area estimated to be effectively controlled. ,by As a starting point, As the end point, a three-dimensional rectangular area is constructed as the camera abnormal area.

2. The AI-based linkage response and control system according to claim 1, characterized in that: The calculation formula of the distance between the monitoring sensor and the potential abnormal sensor is as follows: ； in, It is The distance between each monitoring sensor and the potential abnormal sensor, It is The location coordinates of the monitoring sensors, is a weighting factor.

3. The AI-based linkage response and control system according to claim 2, characterized in that: The step of obtaining the spatial weighted anomaly score comprises: The abnormal score of each target monitoring sensor is standardized to eliminate the influence of different dimensions. The first target monitoring sensor Anomaly score , its standard value is calculated as follows: ； in, and They are The mean and standard deviation of the target monitoring sensor data; Each normalized anomaly score is spatially weighted to take into account the impact of space. The spatially weighted anomaly score is: ； in, It is The first target monitoring sensor The normalized anomaly score The spatially weighted anomaly score of It is The normalized distance between the target monitoring sensor and the potential abnormal sensor, It is the set of distances between all target monitoring sensors and potential abnormal sensors.

4. The AI-based linkage response and control system according to claim 1, characterized in that: The specific steps of the smoke detection model for identifying abnormal smoke areas include: Identify key frame images of the environment video stream; Preprocess the key frame images to extract the characteristic information related to smoke; The extracted feature information is input into the smoke detection model, and the output of the binary classification of the smoke detection model is converted into a probability value using the softmax function; Configuring the judgment threshold , used to determine whether there is a smoke abnormal area in the current key frame image. When the probability value is greater than the judgment threshold , it is determined that there is an abnormal smoke area in the current key frame image, and it is marked as an abnormal image. The abnormal image, the time when the abnormal image was collected, and the location of the surveillance camera that collected the abnormal image are output.

5. The AI-based linkage response and control system according to claim 4, characterized in that: The specific steps of the smoke detection model for identifying abnormal smoke areas also include: The abnormal image is binarized, the smoke abnormal area is marked as a high brightness value, and the non-smoke area is marked as a low brightness value; Perform connected domain analysis to obtain connected high-brightness areas, use depth-first search to traverse the abnormal image, mark the pixels belonging to the same connected domain, calculate the circumscribed rectangle of each connected domain as the bounding box of the smoke anomaly area, and record the coordinate position of the bounding box of each smoke anomaly area.

6. The AI-based linkage response and control system according to claim 1, characterized in that: The calculation formulas for the initial diffusion velocity and the initial diffusion direction are as follows: ； ； in, and are the initial diffusion velocity and initial diffusion direction, It is The weight of the timestamp, and It is The first The diffusion speed and direction, For the Surveillance camera ID The timestamp of the key frame image, is the number of abnormal smoke areas in the identified key frame images.

7. The AI-based linkage response and control system according to claim 6, characterized in that: The calculation formula for the reverse tracking of the fire source position is as follows: ； ； ； in, It is The initial location of the fire source estimated by the surveillance camera ID, It is The last location of the smoke abnormality area in the surveillance camera ID; is the time interval to the fire source.

8. The AI-based linkage response and control system according to claim 1, characterized in that: The specific steps of abnormal level classification include: When an abnormal area of ​​the monitoring sensor is detected, the monitoring results of the abnormal area of ​​the monitoring camera in the area are retrieved. If the monitoring camera does not find any abnormality, it is classified as a first-level abnormality, otherwise it is a second-level abnormality; When an abnormal area is detected by the surveillance camera, the abnormal area monitoring results of the monitoring sensor in the area are retrieved. If the monitoring sensor does not find any abnormality, it is classified as a second-level abnormality, otherwise it is a third-level abnormality.

Images