Micro-power-consumption state machine vision intelligent surface fire monitoring and identifying system

Abstract

A micro-power-consumption state machine vision intelligent surface fire monitoring and recognition system is characterized in that a micro-power-consumption camera module, an MCU microcontroller, a communication module and an alarm module are arranged, the MCU microcontroller is in bidirectional connection with the micro-power-consumption camera module and the communication module, the communication module is connected with the alarm module, and the alarm module is connected with the micro-power-consumption camera module. A deep learning algorithm module is arranged in the MCU microcontroller; the MCU microcontroller controls the micro-power-consumption camera module to acquire image data and transmit the image data to the deep learning algorithm module, the deep learning algorithm module classifies and identifies the image data to obtain a classification and identification result, and when the classification and identification result is abnormal, the MCU microcontroller activates the communication module and sends the communication module to the micro-power-consumption camera module. And the fire alarm and the key video evidence are uploaded to a cloud, and an alarm module is triggered to give an alarm. The system has the advantages that state machine control and an efficient image recognition algorithm are achieved on the MCU microcontroller, system power consumption is remarkably reduced, and accuracy and real-time performance of monitoring and recognition are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of surface fire monitoring and identification, and in particular to a micro-power state machine visual intelligent surface fire monitoring and identification system. Background Art

[0002] At present, surface fire monitoring technology has been widely used in forest fire prevention, environmental protection and other aspects. Traditional surface fire monitoring methods mainly rely on manual inspections, satellite monitoring and various sensor networks. However, manual inspection methods are inefficient, costly, and subject to great restrictions on weather, terrain and other conditions; although satellite monitoring has a wide coverage, it has problems such as long monitoring cycles and poor real-time performance; although various sensor networks can provide relatively real-time monitoring data, in large-area monitoring scenarios, the cost of sensor deployment and maintenance is high, and the sensor battery life is limited.

[0003] In recent years, with the advancement of microelectronics and artificial intelligence technologies, surface fire monitoring methods based on image recognition technology have gradually emerged. However, when traditional image recognition methods are applied to surface fire monitoring, there are the following problems:

[0004] 1. High power consumption: Traditional cameras and processing units consume high power in continuous working mode and are not suitable for long-term, unattended monitoring scenarios.

[0005] 2. Poor real-time performance: Some existing image recognition methods based on cloud computing require uploading the captured images or video streams to the cloud for processing. There is a certain network delay and it cannot meet the requirements of real-time fire alarm.

[0006] 3. Insufficient recognition accuracy: Some simple image processing algorithms cannot accurately identify surface fires and are prone to false positives or omissions. Summary of the invention

[0007] The present invention provides a micro-power state machine visual intelligent surface fire monitoring and identification system, which can significantly reduce system power consumption while ensuring monitoring accuracy.

[0008] To achieve the above-mentioned purpose, the present invention provides a micro-power state machine visual intelligent surface fire monitoring and identification system, the key of which is: a micro-power camera module, an MCU microcontroller, a communication module and an alarm module are provided, the MCU microcontroller is bidirectionally connected to the micro-power camera module and the communication module respectively, the communication module is connected to the alarm module, and a deep learning algorithm module is provided in the MCU microcontroller;

[0009] The MCU microcontroller controls the micro-power camera module to acquire image data and transmits it to the deep learning algorithm module. The deep learning algorithm module classifies and identifies the image data to obtain a classification and identification result. When the classification and identification result is an abnormal situation, the MCU microcontroller activates the communication module, uploads the fire alarm and key video evidence to the cloud, and triggers the alarm module to alarm;

[0010] The MCU microcontroller is constructed with a state machine having four working modes: power on, power off, half-working and full-working.

[0011] Through the above design, the MCU microcontroller is responsible for the core control of the entire system. It has built a state machine with four working modes: shutdown, startup, half-working and full-working, to dynamically adjust the system's operating status and achieve efficient power consumption management and monitoring functions.

[0012] The micro-power camera module is controlled by the MCU microcontroller, and a state machine with four working modes is built in the MCU microcontroller. Combined with the deep learning algorithm model trained by MobileNetV4, efficient and low-power surface fire monitoring and identification can be achieved.

[0013] The communication and alarm mechanism ensures the rapid transmission and processing of fire information. Once the system confirms that the surface fire is abnormal, the MCU immediately activates the communication module, uploads the fire alarm and key video evidence to the cloud, and triggers the real-time alarm process. This immediate response mechanism is crucial to accelerate the implementation of fire response measures and help minimize the damage that may be caused by the fire.

[0014] Preferably: the micro-power consumption camera module is provided with two working modes: micro-power consumption and full-power consumption. In the micro-power consumption mode, the micro-power consumption camera module takes a single photo and performs low-frequency periodic detection; in the full-power mode, the micro-power consumption camera module continuously shoots video streams and performs high-frequency depth monitoring and identification.

[0015] Through the above design, a camera module that supports micro-power consumption mode and full working mode is adopted. In different working modes, the camera module works at different power consumption levels, further reducing the overall power consumption of the system. While ensuring the accuracy of surface fire detection and recognition, the system power consumption is effectively reduced, thereby reducing the system operating cost.

[0016] The micro-power camera module uses a camera component with both micro-power and full-power modes. The system can intelligently adjust the working intensity of the camera module according to actual monitoring needs, effectively switching freely between low-frequency monitoring and high-frequency accurate identification. This design not only improves energy utilization efficiency, but also ensures the flexibility and effectiveness of monitoring tasks.

[0017] As a preference: the deep learning algorithm module is provided with a single-frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B , the single frame image classification and recognition algorithm model M A For identifying single photo image data in the semi-working mode, the video stream classification and recognition algorithm model M B Used to identify video stream image data in full-time mode.

[0018] Single-frame image classification and recognition algorithm model M established by MobileNetV4 training A And video stream classification and recognition algorithm model M B , which are used for surface fire monitoring and identification of single-frame images and video streams, respectively, to ensure the accuracy and real-time performance of identification, thereby improving the accuracy and reliability of surface fire identification.

[0019] In terms of recognition technology, this paper integrates advanced deep learning algorithms and uses MobileNetV4 framework customized M A With M B Two sets of algorithm models are respectively aimed at fast initial screening of static images and in-depth analysis of dynamic video streams. A The model ensures instant response to single-frame images with its high speed and high accuracy; B By introducing time series analysis, the model strengthens the ability to capture surface fire features in continuous videos and improves the precision and reliability of identification. The two work together to greatly enhance the system's monitoring and identification efficiency.

[0020] As a preference: the single-frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The current wind direction information is obtained by identifying the flame tilt direction and smoke drift trajectory in the image data; the video stream classification and recognition algorithm model M B The current wind force information is obtained by identifying the flame swing frequency and smoke diffusion speed in the video stream image data;

[0021] Flame tilt direction recognition: the single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The wind direction is determined by detecting the tilt direction of the flame, and the main tilt axis of the flame is calculated using an image processing algorithm. The expression is as follows:

[0022]

[0023] Where I represents the input image, F represents the flame area, (x, y) represents the pixel coordinates in the flame area, and θ represents the tilt angle of the flame;

[0024] Smoke drift trajectory recognition: the single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The wind direction is determined by tracking the movement path of the smoke in the video, and the moving vector field V of the smoke is calculated using the optical flow algorithm. The expression is as follows:

[0025] V=OpticalFlow(I t-1 ,I t )

[0026] Among them I t-1 represents the image data at time t-1, I t represents the image data at time t, and OpticalFlow represents the optical flow algorithm;

[0027] Flame swing frequency recognition: the video stream classification and recognition algorithm model M B The wind force is determined by analyzing the swing frequency of the flame. Stronger wind force will cause the flame to swing more frequently. The center of gravity of the flame area (x f ,y f ) is as follows:

[0028]

[0029] The expression for calculating the swing frequency f of the flame center of gravity is:

[0030] f=Frequency({(x f (t),y f (t))})

[0031] Smoke diffusion speed recognition: the video stream classification and recognition algorithm model M B The wind speed is determined by measuring the speed of smoke diffusion. The stronger the wind speed, the faster the smoke diffuses. The expression for calculating the smoke diffusion speed v through the optical flow vector field V is as follows:

[0032]

[0033] Through the above design, the wind direction is judged according to the inclination direction of the flame and the trajectory of the smoke drifting, and the wind force is judged by the flame swing frequency and the smoke diffusion speed. It not only accurately identifies the fire situation, but also effectively predicts the fire spread trend, pointing out the direction for subsequent fire prevention and disaster relief work.

[0034] As a preference: the single-frame image classification and recognition algorithm model M A An image preprocessing unit, a feature extraction unit and a classification and discrimination unit are provided in the apparatus;

[0035] The image preprocessing unit is used to normalize and resize the captured single-frame image, and the expression is as follows:

[0036]

[0037] Among them, I norm represents the processed image, I represents the input image, μ represents the mean of the image, and σ represents the standard deviation of the image;

[0038] The feature extraction is used to extract high-dimensional features in the image and capture the characteristic information of the surface fire. The expression is as follows:

[0039] A=MobileNetV4(I norm )

[0040] Where A represents the extracted high-dimensional feature vector;

[0041] The classification and discrimination unit is used to classify the extracted image features, judge the surface fire conditions in the image, and classify them as normal or abnormal conditions. The expression is as follows:

[0042]

[0043] Among them, W represents the weight matrix, b represents the bias vector, Represents the predicted class probability distribution.

[0044] The single frame image classification and recognition algorithm model M A For the preliminary recognition of single-frame images, the algorithm is trained using the MobileNetV4 model, which has high recognition accuracy and low computational complexity. The image preprocessing module performs preprocessing operations such as normalization and size adjustment on the captured single-frame images to meet the input requirements of the MobileNetV4 model.

[0045] In micro-power consumption mode, image capture and recognition need to be completed within 450ms to ensure real-time performance.

[0046] As a preferred embodiment: the video stream classification and recognition algorithm model M B A video frame preprocessing unit, a feature extraction and time series analysis unit, and a classification and recognition unit are sequentially arranged;

[0047] The video frame preprocessing unit is used to normalize and resize each frame in the video stream, and the expression is as follows:

[0048]

[0049] Among them, I t represents the t-th frame image;

[0050] The feature extraction and time series analysis unit is used to extract high-dimensional features of each frame, and combined with the time series analysis method, obtain the time series relationship features between frames, and the expression is as follows:

[0051] A t =MobileNetV4(I t )

[0052] A 1:T =[A1,A2,…,A T ]

[0053] Among them, A t represents the feature sequence of the tth frame, A 1:T is the feature sequence from frame 1 to frame T;

[0054] The classification and recognition unit is used to classify the video stream features as a whole, determine the surface fire conditions in the video, and classify them as normal or abnormal. The expression is as follows:

[0055]

[0056] Among them, LSTM represents long short-term memory network, and softmax represents activation function.

[0057] After confirming the abnormal situation, the MCU microcontroller controls the communication module to upload the fire alarm message and short video to the cloud to issue a fire alarm.

[0058] The video stream classification and recognition algorithm model M B Used for deep recognition of continuous video streams. The algorithm is also trained using the MobileNetV4 model, but on this basis, it combines the time series analysis method to conduct more in-depth surface fire monitoring and recognition of video streams. The algorithm considers the temporal relationship between frames and effectively improves the accuracy of surface fire classification and recognition.

[0059] As a preference: the single-frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B Trained by MobileNetV4.

[0060] Among them, the single-frame image classification and recognition algorithm model M A Based on MobileNetV4, the following optimizations were made:

[0061] Data augmentation: Add surface fire images under different wind conditions to the training data to improve the generalization ability of the model.

[0062] Feature fusion: The features of flame tilt direction and smoke drift trajectory are added to the model, so that the model can more accurately identify the fire and infer the wind direction.

[0063] Multi-task learning: Simultaneously train fire identification and wind direction identification tasks, so that the model can identify fire and determine wind direction at the same time.

[0064] Below is M A The general structure of the model is:

[0065]

[0066] Among them, MultiScaleFeatures represents multi-scale features, A base Represents the basic features, A multi Represents multi-scale features, A flame_angle Indicates flame characteristics, A smoke_trajectory Indicates smoke characteristics, A combined Represents comprehensive characteristics, Indicates fire output. Indicates wind direction output, Indicates wind power output.

[0067] The video stream classification and recognition algorithm model M B Based on MobileNetV4, the following optimizations were made:

[0068] Time series analysis: Combined with time series analysis methods, a long short-term memory network (LSTM) is used to capture the dynamic changes of flames and smoke in the video and improve the prediction accuracy of wind speed and direction.

[0069] Multi-scale feature extraction: A multi-scale feature extraction module is added to the model to capture the characteristics of flames and smoke at different scales and enhance the model's ability to identify wind direction and wind force.

[0070] Optical flow features: Optical flow features are added to the model to analyze the movement trajectory of smoke and flames in the video, so as to more accurately infer wind direction and wind speed.

[0071] Below is M B The general structure of the model is:

[0072]

[0073]

[0074] As a preferred embodiment: in shutdown mode, the MCU microcontroller is in uA-level sleep state, the micro-power camera module is in 0-power shutdown state, and the communication module is in working state. The power consumption expression in shutdown mode is as follows:

[0075] P off =0

[0076] Among them, P off Indicates the power consumption in shutdown mode;

[0077] In the power-on mode, the MCU microcontroller is in the mA-level working state. The MCU microcontroller controls the micro-power camera module to capture photos and perform preliminary recognition. The power consumption expression in the power-on mode is as follows:

[0078] P on =P MCU +P camera

[0079] Among them, P on Indicates the power consumption in the on mode, P MCU is the power consumption of the microcontroller unit, P camera is the power consumption of the camera module;

[0080] In semi-working mode: When the single-frame image classification and recognition algorithm model M A When the recognition result is normal, the MCU microcontroller enters sleep mode, the micro-power camera module enters micro-power mode, and the MCU microcontroller wakes up the micro-power camera module every 1-3 seconds to take pictures and classify and recognize the images. The power consumption expression in semi-working mode is as follows:

[0081] P half =P MCU +P camera,low

[0082] Among them, P half Indicates the power consumption in half-duty mode, P camera,low Indicates the power consumption of the camera module in micro power consumption mode;

[0083] In full working mode: When the single frame image classification recognition algorithm model M A When the recognition result is abnormal, the MCU microcontroller controls the micro-power camera module to enter the full-power mode. The MCU microcontroller controls the micro-power camera module to continuously shoot the video stream, and performs deep recognition and alarm processing on the video stream. The power consumption expression in the full-power mode is as follows:

[0084] P full =P MCU +P camera,high

[0085] Among them, P full Indicates the power consumption in full working mode, P camera,high Indicates the power consumption of the micro-power camera module in full power mode;

[0086] The transition conditions of the state machine working mode can be expressed as follows:

[0087]

[0088] Among them, t no activity Indicates the time without activity, T off is the shutdown time threshold, is the output of the classifier.

[0089] By building a state machine with four modes of shutdown, startup, half-operation and full-operation inside the MCU microcontroller, the system can flexibly respond to network commands and dynamically adjust the operating status. This not only achieves seamless switching from ultra-low power sleep to high-intensity monitoring, but also ensures the optimal balance between system power consumption and monitoring performance. For example, the system can automatically enter shutdown mode when there is no monitoring demand, greatly reducing energy consumption; and quickly switch to full-operation mode when necessary, providing continuous video stream capture and high-level fire identification and alarm, fully adapting to different safety demand scenarios.

[0090] As a preferred embodiment, the micro-power camera module is provided with n cameras, and the n camera arrays are distributed in the area to be detected. When the images taken by one or more cameras are detected by the single-frame image classification and recognition algorithm model M A When an abnormal state is identified, the MCU microcontroller wakes up the abnormal state camera and its surrounding cameras to enter full power mode, and continuously shoots video streams for detection; if the surrounding cameras also detect abnormal conditions, the MCU microcontroller further expands the wake-up range to make the cameras in a larger range enter full power mode;

[0091] The wake-up strategy of the camera module is as follows:

[0092] (1) Fire detection: Let I represent the input image and f(I) represent the fire detection function. When f(I) = 1, it means that a fire is detected and the system enters the full-power mode. Otherwise, it remains in the micro-power mode.

[0093] (2) Initial wake-up: Set C i represents the i-th camera, R(C i ) indicates that C i The initial wake-up range is centered on m cameras, and the camera set within the wake-up range is {C j |C j ∈R(C i )}, where C j represents the jth camera;

[0094] (3) Gradual expansion: Let t represent the time step, the initial time t = 0, when there is C j ∈R(C i) and f(C j )=1, then the wake-up range R(C i ,t+1)=R(C i ,t)+ΔR, where ΔR represents the extended range; the set of cameras with all wake-up ranges extended between time step 0 and T is:

[0095]

[0096] Through the above design, after detecting an abnormal situation, the system will prioritize waking up cameras within a certain range for further monitoring. If the newly awakened camera also detects an abnormal situation, the system will continue to expand the monitoring range and wake up more cameras for collaborative monitoring and identification. This gradually expanded wake-up mechanism ensures dynamic adjustment of the monitoring range, which can not only capture and identify fires in a timely manner at the early stage of the fire, but also provide wider monitoring coverage when the fire expands, improving the efficiency and accuracy of the overall monitoring system.

[0097] By setting up a gradually expanding mechanism, the monitoring scope can be flexibly adjusted according to the actual spread of the fire, ensuring that the monitoring system covers all areas affected by the fire. This wake-up strategy not only improves the real-time and accuracy of monitoring, but also effectively reduces the overall power consumption of the system, ensuring that the system can quickly respond and handle the fire at critical moments.

[0098] As a preferred embodiment: the video stream classification and recognition algorithm model M B According to the flame tilt direction or smoke drift trajectory and its horizontal displacement in the video stream image data, combined with the position information of the corresponding camera, the environmental wind direction information is comprehensively judged; the environmental wind direction information includes the environmental wind direction and the wind direction deviation angle, and the judgment process is as follows:

[0099] The video stream classification and recognition algorithm model M B A camera is selected from the abnormal state cameras as a forward camera, and the orientation information of the forward camera is obtained, and then, according to the orientation information, the flame tilt direction, flame height or smoke drift direction, and smoke height in the video stream image data captured by the forward camera are judged, and the forward horizontal displacement of the flame or smoke is judged according to the flame tilt direction, flame height or smoke drift direction, and smoke height;

[0100] The height of the flame or smoke in the video stream image data captured by the forward camera is used as the standard height, and a camera whose orientation information is close to the tilt direction of the flame or the drift direction of the smoke and is perpendicular to the orientation information of the current camera is selected as the side camera, and the lateral horizontal displacement of the flame or smoke in the video stream image data captured by the side camera at the standard height is determined;

[0101] According to the forward horizontal displacement and the lateral horizontal displacement, the ambient wind direction and the wind direction deviation angle are determined. The calculation formula of the wind direction deviation angle is as follows:

[0102]

[0103] Among them, α represents the wind direction deviation angle, H1 represents the positive horizontal displacement of the flame or smoke, and H2 represents the lateral horizontal displacement of the flame or smoke.

[0104] Through the above design, the specific environmental wind direction information can be comprehensively judged and the environmental wind direction changes can be understood in real time. The change of wind direction directly affects the spread direction of the fire. Therefore, understanding and adapting to the change of wind direction is the key to improving the efficiency of fire fighting and rescue and ensuring the safety of personnel.

[0105] The beneficial effects of the present invention are as follows: by implementing state machine control and efficient image recognition algorithms on the MCU microcontroller, the system power consumption is significantly reduced, and the accuracy and real-time performance of monitoring and identification are improved. It can not only detect fire conditions and issue fire alarms, but also detect the current environmental wind direction and wind speed, predict the fire spread trend, and provide direction for subsequent fire prevention and disaster relief.

[0106] The present invention provides a comprehensive and efficient solution for the field of surface fire monitoring by comprehensively using intelligent state machine control, micro-power management, deep learning recognition, and an efficient communication alarm system, reflecting the great potential of technological innovation in improving public safety and environmental protection. BRIEF DESCRIPTION OF THE DRAWINGS

[0107] Figure 1 It is a structural schematic diagram of the present invention;

[0108] Figure 2 This is a schematic diagram of the working process of the micro-power state machine;

[0109] Figure 3 Schematic diagram of the calculation coordinates of the ambient wind direction angle in the embodiment. DETAILED DESCRIPTION

[0110] The present invention is further described in detail below in conjunction with the accompanying drawings and specific examples. The following examples or drawings are used to illustrate the present invention, but are not used to limit the scope of the present invention.

[0111] like Figure 1 As shown: a micro-power state machine visual intelligent surface fire monitoring and identification system, which is provided with a micro-power camera module, an MCU microcontroller, a communication module and an alarm module, wherein the MCU microcontroller is bidirectionally connected with the micro-power camera module and the communication module respectively, the communication module is connected with the alarm module, and a deep learning algorithm module is provided in the MCU microcontroller;

[0112] The MCU microcontroller controls the micro-power camera module to acquire image data and transmits it to the deep learning algorithm module. The deep learning algorithm module classifies and identifies the image data to obtain a classification and identification result. When the classification and identification result is an abnormal situation, the MCU microcontroller activates the communication module, uploads the fire alarm and key video evidence to the cloud, and triggers the alarm module to alarm;

[0113] The MCU microcontroller is constructed with a state machine having four working modes: power on, power off, half-working and full-working.

[0114] The micro-power camera module is provided with two working modes: micro-power consumption and full-power consumption. In the micro-power consumption mode, the micro-power consumption camera module takes a single photo and performs low-frequency periodic detection; in the full-power mode, the micro-power consumption camera module continuously shoots video streams and performs high-frequency depth monitoring and identification.

[0115] The deep learning algorithm module is provided with a single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B , the single frame image classification and recognition algorithm model M A For identifying single photo image data in the semi-working mode, the video stream classification and recognition algorithm model M B Used to identify video stream image data in full-time mode.

[0116] The single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The current wind direction information is obtained by identifying the flame tilt direction and smoke drift trajectory in the image data; the video stream classification and recognition algorithm model M B The current wind force information is obtained by identifying the flame swing frequency and smoke diffusion speed in the video stream image data;

[0117] Flame tilt direction recognition: the single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The wind direction is determined by detecting the tilt direction of the flame, and the main tilt axis of the flame is calculated using an image processing algorithm. The expression is as follows:

[0118]

[0119] Where I represents the input image, F represents the flame area, (x, y) represents the pixel coordinates in the flame area, and θ represents the tilt angle of the flame;

[0120] Smoke drift trajectory recognition: the single frame image classification and recognition algorithm model M AAnd video stream classification and recognition algorithm model M B The wind direction is determined by tracking the movement path of the smoke in the video, and the moving vector field V of the smoke is calculated using the optical flow algorithm. The expression is as follows:

[0121] V=OpticalFlow(I t-1 ,I t )

[0122] Among them I t-1 represents the image data at time t-1, I t represents the image data at time t, and OpticalFlow represents the optical flow algorithm;

[0123] Flame swing frequency recognition: the video stream classification and recognition algorithm model M B The wind force is determined by analyzing the swing frequency of the flame. Stronger wind force will cause the flame to swing more frequently. The center of gravity of the flame area (x f ,y f ) is as follows:

[0124]

[0125] The expression for calculating the swing frequency f of the flame center of gravity is:

[0126] f=Frequency({(x f (t),y f (t))})

[0127] Smoke diffusion speed recognition: the video stream classification and recognition algorithm model M B The wind speed is determined by measuring the speed of smoke diffusion. The stronger the wind speed, the faster the smoke diffuses. The expression for calculating the smoke diffusion speed v through the optical flow vector field V is as follows:

[0128]

[0129] The single frame image classification and recognition algorithm model M A An image preprocessing unit, a feature extraction unit and a classification and discrimination unit are provided in the apparatus;

[0130] The image preprocessing unit is used to normalize and resize the captured single-frame image, and the expression is as follows:

[0131]

[0132] Among them, I norm represents the processed image, I represents the input image, μ represents the mean of the image, and σ represents the standard deviation of the image;

[0133] The feature extraction is used to extract high-dimensional features in the image and capture the characteristic information of the surface fire. The expression is as follows:

[0134] A=MobileNetV4(I norm )

[0135] Where A represents the extracted high-dimensional feature vector;

[0136] The classification and discrimination unit is used to classify the extracted image features, judge the surface fire conditions in the image, and classify them as normal or abnormal conditions. The expression is as follows:

[0137]

[0138] Among them, W represents the weight matrix, b represents the bias vector, Represents the predicted class probability distribution.

[0139] The video stream classification and recognition algorithm model M B A video frame preprocessing unit, a feature extraction and time series analysis unit, and a classification and recognition unit are sequentially arranged;

[0140] The video frame preprocessing unit is used to normalize and resize each frame in the video stream, and the expression is as follows:

[0141]

[0142] Among them, I t represents the t-th frame image;

[0143] The feature extraction and time series analysis unit is used to extract high-dimensional features of each frame, and combined with the time series analysis method, obtain the time series relationship features between frames, and the expression is as follows:

[0144] A t =MobileNetV4(I t )

[0145] A 1:T =[A1,A2,…,A T ]

[0146] Among them, A t represents the feature sequence of the tth frame, A 1:T is the feature sequence from frame 1 to frame T;

[0147] The classification and recognition unit is used to classify the video stream features as a whole, determine the surface fire conditions in the video, and classify them as normal or abnormal. The expression is as follows:

[0148]

[0149] Among them, LSTM represents long short-term memory network, and softmax represents activation function.

[0150] The single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B Trained by MobileNetV4.

[0151] like Figure 2 As shown: In shutdown mode, the MCU microcontroller is in uA-level sleep state, the micro-power camera module is in 0-power shutdown state, and the communication module is in working state. The power consumption expression in shutdown mode is as follows:

[0152] P off =0

[0153] Among them, P off Indicates the power consumption in shutdown mode;

[0154] In the power-on mode, the MCU microcontroller is in the mA-level working state. The MCU microcontroller controls the micro-power camera module to capture photos and perform preliminary recognition. The power consumption expression in the power-on mode is as follows:

[0155] P on =P MCU +P camera

[0156] Among them, P on Indicates the power consumption in the on mode, P MCU is the power consumption of the microcontroller unit, P camera is the power consumption of the camera module;

[0157] In semi-working mode: When the single-frame image classification and recognition algorithm model M A When the recognition result is normal, the MCU microcontroller enters sleep mode, the micro-power camera module enters micro-power mode, and the MCU microcontroller wakes up the micro-power camera module every 1-3 seconds to take pictures and classify and recognize the images. The power consumption expression in semi-working mode is as follows:

[0158] P half =P MCU +P camera,low

[0159] Among them, P half Indicates the power consumption in half-duty mode, P camera,low Indicates the power consumption of the camera module in micro power consumption mode;

[0160] In full working mode: When the single frame image classification recognition algorithm model M AWhen the recognition result is abnormal, the MCU microcontroller controls the micro-power camera module to enter the full-power mode. The MCU microcontroller controls the micro-power camera module to continuously shoot the video stream, and performs deep recognition and alarm processing on the video stream. The power consumption expression in the full-power mode is as follows:

[0161] P full =P MCU +P camera,high

[0162] Among them, P full Indicates the power consumption in full working mode, P camera,high Indicates the power consumption of the micro-power camera module in full power mode;

[0163] The transition conditions of the state machine working mode can be expressed as follows:

[0164]

[0165] Among them, t no activity Indicates the time without activity, T off is the shutdown time threshold, is the output of the classifier.

[0166] The micro-power camera module is provided with n cameras, and the n camera arrays are distributed in the area to be detected. When the images taken by one or more cameras are detected by the single-frame image classification and recognition algorithm model M A When an abnormal state is identified, the MCU microcontroller wakes up the abnormal state camera and its surrounding cameras to enter full power mode, and continuously shoots video streams for detection; if the surrounding cameras also detect abnormal conditions, the MCU microcontroller further expands the wake-up range to make the cameras in a larger range enter full power mode;

[0167] The wake-up strategy of the camera module is as follows:

[0168] (1) Fire detection: Let I represent the input image and f(I) represent the fire detection function. When f(I) = 1, it means that a fire is detected and the system enters the full-power mode. Otherwise, it remains in the micro-power mode.

[0169] (2) Initial wake-up: Set C i represents the i-th camera, R(C i ) indicates that C i The initial wake-up range is centered on m cameras, and the camera set within the wake-up range is {C j |C j ∈R(C i )}, where C j represents the jth camera;

[0170] (3) Gradual expansion: Let t represent the time step, the initial time t = 0, when there is C j ∈R(C i ) and f(C j )=1, then the wake-up range R(C i ,t+1)=R(C i ,t)+ΔR, where ΔR represents the extended range; the set of cameras with all wake-up ranges extended between time step 0 and T is:

[0171]

[0172] The video stream classification and recognition algorithm model M B According to the flame tilt direction or smoke drift trajectory and its horizontal displacement in the video stream image data, combined with the position information of the corresponding camera, the environmental wind direction information is comprehensively judged; the environmental wind direction information includes the environmental wind direction and the wind direction deviation angle, and the judgment process is as follows:

[0173] The video stream classification and recognition algorithm model M B A camera is selected from the abnormal state cameras as a forward camera, and the orientation information of the forward camera is obtained, and then, according to the orientation information, the flame tilt direction, flame height or smoke drift direction, and smoke height in the video stream image data captured by the forward camera are judged, and the forward horizontal displacement of the flame or smoke is judged according to the flame tilt direction, flame height or smoke drift direction, and smoke height;

[0174] The height of the flame or smoke in the video stream image data captured by the forward camera is used as the standard height, and a camera whose orientation information is close to the tilt direction of the flame or the drift direction of the smoke and is perpendicular to the orientation information of the current camera is selected as the side camera, and the lateral horizontal displacement of the flame or smoke in the video stream image data captured by the side camera at the standard height is determined;

[0175] According to the forward horizontal displacement and the lateral horizontal displacement, the ambient wind direction and the wind direction deviation angle are determined. The calculation formula of the wind direction deviation angle is as follows:

[0176]

[0177] Among them, α represents the wind direction deviation angle, H1 represents the positive horizontal displacement of the flame or smoke, and H2 represents the lateral horizontal displacement of the flame or smoke.

[0178] like Figure 3 As shown: the front camera in this embodiment faces east and the side camera faces south.

[0179] The present invention constructs a state machine including four working modes of shutdown, startup, half-work and full-work in the MCU microcontroller, and dynamically switches each working mode by receiving instructions from network communication. The design of this state machine can effectively balance the power consumption and monitoring efficiency of the system. For example, in shutdown mode, the MCU is in an ultra-low power sleep state, and the camera module is completely shut down to achieve the lowest power consumption; in startup mode, the MCU and the camera module start to perform basic capture and recognition operations; in half-work mode, the system periodically captures and recognizes images in a timed manner to achieve continuous monitoring under low power consumption; in full-work mode, the camera module continuously shoots video streams and performs more in-depth recognition and alarm processing to deal with high-risk surface fires.

[0180] The present invention adopts a camera module that supports micro-power consumption mode and full working mode, so that the camera module can work at different power consumption levels in different working modes. In micro-power consumption mode, the camera module can only take a single photo, which is suitable for low-frequency monitoring tasks; in full working mode, the camera module can continuously shoot video streams, which is suitable for high-frequency and high-precision monitoring tasks. By dynamically switching the working mode of the camera module, the overall power consumption of the system can be significantly reduced while ensuring the monitoring effect.

[0181] The present invention adopts MobileNetV4 training to establish M A Algorithm Model and M B The algorithm model is used for surface fire monitoring and recognition in single-frame images and video streams. A The algorithm model is used to perform preliminary recognition of a single-frame image. It has high recognition accuracy and low computational complexity. It can complete image capture and recognition within 450ms, ensuring the real-time performance of the system. B The algorithm model is used to perform deep recognition on continuous video streams. By combining the time series analysis method and considering the temporal relationship between frames, the recognition accuracy is improved. The combination of the two algorithm models not only ensures the rapidity of preliminary recognition, but also takes into account the accuracy of deep recognition.

[0182] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A micro-power state machine visual intelligent surface fire monitoring and identification system, characterized by: A micro-power camera module, an MCU microcontroller, a communication module and an alarm module are provided, wherein the MCU microcontroller is bidirectionally connected to the micro-power camera module and the communication module respectively, the communication module is connected to the alarm module, and a deep learning algorithm module is provided in the MCU microcontroller; The MCU microcontroller controls the micro-power camera module to acquire image data and transmits it to the deep learning algorithm module. The deep learning algorithm module classifies and identifies the image data to obtain a classification and identification result. When the classification and identification result is an abnormal situation, the MCU microcontroller activates the communication module, uploads the fire alarm and key video evidence to the cloud, and triggers the alarm module to alarm; The MCU microcontroller is constructed with a state machine having four working modes: power on, power off, half-working and full-working.

2. According to the micro-power state machine visual intelligent surface fire monitoring and identification system of claim 1, it is characterized by: The micro-power camera module is provided with two working modes: micro-power consumption and full-power consumption. In the micro-power consumption mode, the micro-power consumption camera module takes a single photo and performs low-frequency periodic detection; in the full-power mode, the micro-power consumption camera module continuously shoots video streams and performs high-frequency depth monitoring and identification.

3. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 1 is characterized by: The deep learning algorithm module is provided with a single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B , the single frame image classification and recognition algorithm model M A For identifying single photo image data in the semi-working mode, the video stream classification and recognition algorithm model M B Used to identify video stream image data in full-time mode.

4. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 3 is characterized by: The single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The current wind direction information is obtained by identifying the flame tilt direction and smoke drift trajectory in the image data; the video stream classification and recognition algorithm model M B The current wind force information is obtained by identifying the flame swing frequency and smoke diffusion speed in the video stream image data; Flame tilt direction recognition: the single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B The wind direction is determined by detecting the tilt direction of the flame, and the main tilt axis of the flame is calculated using an image processing algorithm. The expression is as follows: Where I represents the input image, F represents the flame area, (x, y) represents the pixel coordinates in the flame area, and θ represents the tilt angle of the flame; Smoke drift trajectory recognition: the single frame image classification recognition algorithm model M A And video stream classification and recognition algorithm model M B The wind direction is determined by tracking the movement path of the smoke in the video, and the moving vector field V of the smoke is calculated using the optical flow algorithm. The expression is as follows: V=OpticalFlow(I t-1 ,I t ) Among them I t-1 represents the image data at time t-1, I t represents the image data at time t, and OpticalFlow represents the optical flow algorithm; Flame swing frequency recognition: the video stream classification and recognition algorithm model M B The wind force is determined by analyzing the swing frequency of the flame. Stronger wind force will cause the flame to swing more frequently. The center of gravity of the flame area (x f ,y f ) is as follows: The expression for calculating the swing frequency f of the flame center of gravity is: f=Frequency({(x f (t),y f (t))}) Smoke diffusion speed recognition: the video stream classification and recognition algorithm model M B The wind speed is determined by measuring the speed of smoke diffusion. The stronger the wind speed, the faster the smoke diffuses. The expression for calculating the smoke diffusion speed v through the optical flow vector field V is as follows:

5. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 3 is characterized by: The single frame image classification and recognition algorithm model M A An image preprocessing unit, a feature extraction unit and a classification and discrimination unit are provided in the apparatus; The image preprocessing unit is used to normalize and resize the captured single-frame image, and the expression is as follows: Among them, I norm represents the processed image, I represents the input image, μ represents the mean of the image, and σ represents the standard deviation of the image; The feature extraction is used to extract high-dimensional features in the image and capture the characteristic information of the surface fire. The expression is as follows: A=MobileNetV4(I norm ) Where A represents the extracted high-dimensional feature vector; The classification and discrimination unit is used to classify the extracted image features, judge the surface fire conditions in the image, and classify them as normal or abnormal conditions. The expression is as follows: Among them, W represents the weight matrix, b represents the bias vector, Represents the predicted class probability distribution.

6. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 3 is characterized by: The video stream classification and recognition algorithm model M B A video frame preprocessing unit, a feature extraction and time series analysis unit, and a classification and recognition unit are sequentially arranged; The video frame preprocessing unit is used to normalize and resize each frame in the video stream, and the expression is as follows: Among them, I t represents the t-th frame image; The feature extraction and time series analysis unit is used to extract high-dimensional features of each frame, and combined with the time series analysis method, obtain the time series relationship features between frames, and the expression is as follows: A t =MobileNetV4(I t ) <h2 style=";text-align:left;direction:ltr">A<h2 style=";text-align:left;direction:ltr"> 1:T <h2 style=";text-align:left;direction:ltr"> (A1,A2,…,A)<h2 style=";text-align:left;direction:ltr"> T <h2 style=";text-align:left;direction:ltr"> ] Among them, A t represents the feature sequence of the tth frame, A 1:T is the feature sequence from frame 1 to frame T; The classification and recognition unit is used to classify the video stream features as a whole, determine the surface fire conditions in the video, and classify them as normal or abnormal. The expression is as follows: Among them, LSTM represents long short-term memory network, and softmax represents activation function.

7. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 3 is characterized by: The single frame image classification and recognition algorithm model M A And video stream classification and recognition algorithm model M B Trained by MobileNetV4.

8. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 1 is characterized by: In shutdown mode, the MCU microcontroller is in uA-level sleep state, the micro-power camera module is in 0-power shutdown state, and the communication module is in working state. The power consumption expression in shutdown mode is as follows: P off ＝0 Among them, P off Indicates the power consumption in shutdown mode; In the power-on mode, the MCU microcontroller is in the mA-level working state. The MCU microcontroller controls the micro-power camera module to capture photos and perform preliminary recognition. The power consumption expression in the power-on mode is as follows: P on =P MCU +P camera Among them, P on Indicates the power consumption in the on mode, P MCU is the power consumption of the microcontroller unit, P camera is the power consumption of the camera module; In semi-working mode: When the single-frame image classification and recognition algorithm model M A When the recognition result is normal, the MCU microcontroller enters sleep mode, the micro-power camera module enters micro-power mode, and the MCU microcontroller wakes up the micro-power camera module every 1-3 seconds to take pictures and classify and recognize the images. The power consumption expression in semi-working mode is as follows: P half =P MCU +P camera,low Among them, P half Indicates the power consumption in half-duty mode, P camera,low Indicates the power consumption of the camera module in micro power consumption mode; In full working mode: When the single frame image classification recognition algorithm model M A When the recognition result is abnormal, the MCU microcontroller controls the micro-power camera module to enter the full-power mode. The MCU microcontroller controls the micro-power camera module to continuously shoot the video stream, and performs deep recognition and alarm processing on the video stream. The power consumption expression in the full-power mode is as follows: P full =P MCU +P camera,high Among them, P full Indicates the power consumption in full working mode, P camera,high Indicates the power consumption of the micro-power camera module in full power mode; The transition conditions of the state machine working mode can be expressed as follows: Among them, t no activity Indicates the time without activity, T off is the shutdown time threshold, is the output of the classifier.

9. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 8 is characterized by: The micro-power camera module is provided with n cameras, and the n camera arrays are distributed in the area to be detected. When the images taken by one or more cameras are detected by the single-frame image classification and recognition algorithm model M A When an abnormal state is identified, the MCU microcontroller wakes up the abnormal state camera and its surrounding cameras to enter full power mode, and continuously shoots video streams for detection; If the surrounding cameras also detect an abnormal situation, the MCU microcontroller further expands the wake-up range, so that the cameras in a larger range enter the full power mode; The wake-up strategy of the camera module is as follows: (1) Fire detection: Let I represent the input image and f(I) represent the fire detection function. When f(I) = 1, it means that a fire is detected and the system enters the full-power mode. Otherwise, it remains in the micro-power mode. (2) Initial wake-up: Set C i represents the i-th camera, R(C i ) indicates that C i The initial wake-up range is centered on m cameras, and the camera set within the wake-up range is {C j |C j ∈R(C i )}, where C j represents the jth camera; (3) Gradual expansion: Let t represent the time step, the initial time t = 0, when there is C j ∈R(C i ) and f(C j )=1, then the wake-up range R(C i ,t+1)=R(C i ,t)+ΔR, where ΔR represents the extended range; the set of cameras with all wake-up ranges extended between time step 0 and T is:

10. The micro-power state machine visual intelligent surface fire monitoring and identification system according to claim 3 is characterized by: The video stream classification and recognition algorithm model M B According to the flame tilt direction or smoke drift trajectory and its horizontal displacement in the video stream image data, combined with the position information of the corresponding camera, the environmental wind direction information is comprehensively judged; the environmental wind direction information includes the environmental wind direction and the wind direction deviation angle, and the judgment process is as follows: The video stream classification and recognition algorithm model M B A camera is selected from the abnormal state cameras as a forward camera, and the orientation information of the forward camera is obtained, and then, according to the orientation information, the flame tilt direction, flame height or smoke drift direction, and smoke height in the video stream image data captured by the forward camera are judged, and the forward horizontal displacement of the flame or smoke is judged according to the flame tilt direction, flame height or smoke drift direction, and smoke height; The height of the flame or smoke in the video stream image data captured by the forward camera is used as the standard height, and a camera whose orientation information is close to the tilt direction of the flame or the drift direction of the smoke and is perpendicular to the orientation information of the current camera is selected as the side camera, and the lateral horizontal displacement of the flame or smoke in the video stream image data captured by the side camera at the standard height is determined; According to the forward horizontal displacement and the lateral horizontal displacement, the ambient wind direction and the wind direction deviation angle are determined. The calculation formula of the wind direction deviation angle is as follows: Among them, α represents the wind direction deviation angle, H1 represents the positive horizontal displacement of the flame or smoke, and H2 represents the lateral horizontal displacement of the flame or smoke.

Images