Fire safety inspection method and system based on machine vision recognition

Abstract

The invention relates to the technical field of fire safety inspection, in particular to a fire safety inspection method and system based on machine vision recognition. The method comprises the following steps: carrying out inspection equipment deployment on an inspection area to obtain a camera deployment scheme and a sensor network deployment scheme; performing multispectral dynamic sensing according to the camera deployment scheme to obtain a time sequence spectral flow; performing multispectral channel motion estimation on the time sequence spectral flow, and performing motion vector noise suppression to obtain a filtered motion vector diagram; performing spectral characteristic-based motion enhancement on the filtered motion vector diagram to obtain a spectral enhanced motion vector; and performing motion amplification algorithm application on the time sequence spectrum flow according to the spectrum enhanced motion vector, and constructing an enhanced motion vector field to obtain the enhanced motion vector field. According to the invention, through multi-modal information fusion, context awareness and capturing of early fine features, early and accurate fire hazard identification and early warning are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of fire safety inspection, and in particular to a fire safety inspection method and system based on machine vision recognition. Background Art

[0002] Traditional fire safety inspections mainly rely on inspectors to conduct visual, auditory, olfactory and other sensory inspections in the inspection area regularly or irregularly to check whether the fire protection facilities are intact, whether there are any violations, whether there are any abnormal odors or smoke, etc. This method is inefficient, highly subjective, prone to fatigue and negligence, difficult to cover all areas and time, and difficult to detect early and subtle fire hazards. At present, there are also various types of fire detectors, such as smoke detectors, temperature detectors, flame detectors, etc., which constitute an automatic fire alarm system. The response speed is faster than manual inspections and can achieve 24-hour monitoring.

[0003] However, existing methods have problems such as insufficient ability to identify fire hazards at an early stage and high false alarm rate:

[0004] 1. Insufficient ability to identify early fire hazards: Existing fire detection methods mainly focus on obvious smoke, flames and high temperatures, while the characteristics of early fires are often very subtle, such as slight temperature rise, thermal disturbances, small amounts of smoke or the release of specific gases. These subtle features are difficult to capture with traditional methods, making it difficult to detect early fires in a timely manner.

[0005] 2. High false alarm rate: Existing fire detection methods are easily affected by environmental interference, such as lighting changes, background motion, other high-temperature objects, etc., which can lead to false alarms. For example, car exhaust, sunlight reflection, welding sparks, etc. are mistakenly identified as fires. Summary of the invention

[0006] Based on this, it is necessary to provide a fire safety inspection method and system based on machine vision recognition to solve at least one of the above technical problems.

[0007] To achieve the above purpose, a fire safety inspection method based on machine vision recognition includes the following steps:

[0008] Step S1: Deploy inspection equipment in the inspection area to obtain a camera deployment plan and a sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain a time-series spectral stream;

[0009] Step S2: performing multi-spectral channel motion estimation on the time series spectral stream and performing motion vector noise suppression to obtain a filtered motion vector map; performing motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; applying a motion amplification algorithm to the time series spectral stream according to the spectrally enhanced motion vector, and performing an enhanced motion vector field construction to obtain an enhanced motion vector field;

[0010] Step S3: extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map;

[0011] Step S4: performing multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain a fused feature vector; performing risk confidence prediction and generation on the fused feature vector to obtain a risk confidence heat map;

[0012] Step S5: Collect and preprocess the situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness warning based on the associated risk area information and the situational information data set to obtain an intelligent warning report.

[0013] The present invention realizes comprehensive perception and information collection of the inspection area by deploying multispectral cameras and sensor networks. Multispectral dynamic perception and motion amplification technology effectively highlights the weak motion information in the early stage of the fire. Thermal texture feature extraction and dynamic change analysis capture the thermal anomaly characteristics related to the fire. Multimodal feature fusion technology combines spectral, motion and thermal texture information to improve the accuracy of fire risk identification. Situational information collection and preprocessing provide richer contextual information for risk assessment, and through situational awareness and early warning, intelligent assessment and early warning of fire risks are realized, and finally an intelligent early warning report containing the location of the risk area, warning level, potential cause analysis and disposal suggestions is generated, thereby significantly improving the efficiency and reliability of early fire detection and early warning. Therefore, the present invention provides a fire safety inspection method based on machine vision recognition, which effectively solves the problems of insufficient ability to identify early fire hazards and high false alarm rate of existing methods through multimodal information fusion, situational awareness and capture of early subtle features, and realizes earlier and more accurate fire hazard identification and early warning.

[0014] Preferably, step S1 comprises the following steps:

[0015] Step S11: deploying a multispectral camera in the inspection area and deploying a sensor network to obtain a camera deployment plan and a sensor network deployment plan;

[0016] Step S12: performing high-speed acquisition of multispectral image sequences according to the camera deployment scheme to obtain an original multispectral image stream;

[0017] Step S13: performing multispectral image preprocessing on the original multispectral image stream to obtain a corrected multispectral image sequence;

[0018] Step S14: performing multispectral image channel alignment and fusion on the corrected multispectral image sequence to obtain a multispectral image stack;

[0019] Step S15: integrating the time-series spectral information of the multispectral image stack to obtain a time-series spectral stream.

[0020] The present invention achieves comprehensive coverage and precise monitoring of the inspection area and effectively captures early signs of fire through multispectral camera deployment and sensor network deployment solutions. High-speed acquisition of multispectral image sequences and preprocessing, including radiation calibration, denoising and geometric correction, ensures the accuracy and reliability of image data. Multispectral image channel alignment and fusion, as well as time-series spectral information integration, construct a multidimensional data set containing time, space and spectral information, providing a rich data foundation for subsequent early fire identification and early warning, and retaining the original information to the greatest extent through precise time synchronization and lossless compression storage, improving the accuracy and efficiency of early fire identification.

[0021] Preferably, step S2 comprises the following steps:

[0022] Step S21: performing multi-spectral channel motion estimation on the time series spectral stream to obtain a channel motion vector diagram;

[0023] Step S22: performing motion vector noise suppression and screening on the channel motion vector map to obtain a filtered motion vector map;

[0024] Step S23: performing motion enhancement based on spectral characteristics on the filtered motion vector map according to the time-series spectral stream to obtain a spectrally enhanced motion vector;

[0025] Step S24: using the spectral enhancement motion vector as guidance information, applying a motion amplification algorithm to the temporal spectral stream to obtain an amplified image sequence;

[0026] Step S25: constructing an enhanced motion vector field for the amplified image sequence according to the spectral enhanced motion vector to obtain an enhanced motion vector field.

[0027] The present invention effectively extracts the weak motion information caused by early fires through multi-spectral channel motion estimation, noise suppression and screening, and motion enhancement based on spectral characteristics, and suppresses the interference of background noise and irrelevant motion. The spectral enhanced motion vector guided motion amplification algorithm highlights the small motion related to the fire, making it more visually obvious and easy to observe and identify. The enhanced motion vector field finally constructed integrates spectral information and motion information, providing a more accurate basis for subsequent fire feature extraction and identification, and improving the sensitivity and reliability of early fire detection.

[0028] Preferably, step S23 includes the following steps:

[0029] Step S231: Calculate the spectral characteristic response graph of the time series spectral stream to obtain a spectral characteristic response graph set;

[0030] Step S232: constructing a spectral similarity weight map according to the spectral feature response atlas to obtain a spectral similarity weight atlas;

[0031] Step S233: fusing the filtered motion vector map with the spectral weight according to the spectral similarity weight atlas to obtain a spectral weighted motion vector map;

[0032] Step S234: performing time domain motion vector smoothing and accumulation on the spectrally weighted motion vector map to obtain a time domain smoothed motion vector map;

[0033] Step S235: performing motion vector enhancement according to the time-domain smoothed motion vector image and the filtered motion vector image to obtain a spectrally enhanced motion vector.

[0034] The present invention integrates spectral information into motion vector calculation by calculating spectral feature response graph and constructing spectral similarity weight graph, which effectively enhances the motion information related to fire characteristics. Through the weighted average fusion strategy, the spectral weights of multiple fire characteristics are combined to improve the accuracy of motion vector. Time domain motion vector smoothing and accumulation, using Kalman filtering, effectively suppress noise interference and highlight continuous motion information. The final motion vector enhancement step further amplifies the motion related to the fire while retaining the original motion information, providing a more reliable basis for subsequent motion amplification and fire identification, thereby improving the sensitivity and robustness of fire detection.

[0035] Preferably, step S3 comprises the following steps:

[0036] Step S31: extracting an infrared image sequence from the time-series spectral stream, and performing a reverse geometric transformation using an enhanced motion vector field to obtain an aligned infrared image sequence;

[0037] Step S32: performing thermal image preprocessing on the aligned infrared image sequence to obtain a preprocessed thermal image sequence;

[0038] Step S33: extracting thermal texture features from the preprocessed thermal image sequence to obtain a thermal texture feature map;

[0039] Step S34: performing a thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map;

[0040] Step S35: locating and characterizing thermal texture anomalies of the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map.

[0041] The present invention effectively eliminates the image deformation caused by motion amplification and extracts the thermal information related to the fire by performing infrared image extraction and inverse geometric transformation on the time series spectral stream. Thermal image preprocessing enhances the image contrast and lays the foundation for subsequent feature extraction. The LBP operator is used to extract thermal texture features and analyze their dynamic changes, effectively capturing thermal texture anomalies in the early stages of a fire. Finally, by locating and characterizing thermal texture anomalies and combining temperature and texture features, accurate identification and classification of potential fire areas are achieved, improving the accuracy and reliability of early fire detection.

[0042] Preferably, step S34 includes the following steps:

[0043] Step S341: Calculate the texture difference feature map on the thermal texture feature map to obtain the texture difference feature map;

[0044] Step S342: Establishing a motion amplitude characteristic map for the enhanced motion vector field to obtain a motion amplitude characteristic map;

[0045] Step S343: fusing the texture difference feature map and the motion amplitude feature map to obtain a fused feature score map;

[0046] Step S344: performing time series anomaly detection on the fused feature score graph to obtain an anomaly score graph;

[0047] Step S345: Generate a thermal texture change map according to the abnormal score map to obtain a thermal texture change map.

[0048] The present invention calculates the texture difference feature map and the motion amplitude feature map and fuses them, effectively combining the thermal texture change and motion information, thereby more comprehensively describing the dynamic characteristics of the early stage of the fire. The time series anomaly detection method is used to analyze the changing trend of the fused feature score, effectively distinguishing between normal thermal texture fluctuations and abnormal changes caused by fire. The thermal texture change map finally generated highlights the potential fire area in the form of anomaly scores, providing a more reliable basis for early warning of fires and improving the sensitivity and accuracy of fire detection.

[0049] Preferably, step S35 includes the following steps:

[0050] Step S351: generating a binary anomaly mask for the thermal texture change map to obtain a binary anomaly mask;

[0051] Step S352: Connect the binary anomaly masks by component labeling to obtain a connected component labeling graph;

[0052] Step S353: extracting abnormal regions from the preprocessed thermal image sequence according to the connected component label graph to obtain an abnormal region image block set;

[0053] Step S354: Calculate the regional temperature statistical characteristics of the abnormal region image block set to obtain a regional temperature characteristic table;

[0054] Step S355: Calculate the regional texture features of the abnormal region image block set to obtain a regional texture feature table;

[0055] Step S356: classifying and marking abnormal patterns according to the regional temperature feature table and the regional texture feature table to obtain a marked abnormal region map;

[0056] Step S357: Generate a thermal texture anomaly map based on the marked abnormal area map, the regional temperature feature table, and the regional texture feature table to obtain a thermal texture anomaly map.

[0057] The present invention realizes the precise positioning, characterization and classification of thermal texture anomalies through a series of steps. First, the areas with significant thermal texture changes are extracted through binarization and connected component analysis. Then, the temperature statistical characteristics and texture characteristics of each abnormal area are calculated respectively, which provides a basis for subsequent classification. Using the pre-trained classification model, the abnormal areas are classified and marked, and different types of early fire phenomena are effectively distinguished. The thermal texture anomaly map finally generated not only contains the location information of the abnormal area, but also provides detailed temperature and texture characteristics, as well as anomaly type labels, which provides more comprehensive and intuitive information for early fire warning and emergency response, thereby improving the efficiency and accuracy of fire detection.

[0058] Preferably, step S4 comprises the following steps:

[0059] Step S41: aligning multimodal feature data of the time series spectral stream, the enhanced motion vector field, and the thermal texture anomaly map to obtain an aligned multimodal feature set;

[0060] Step S42: extracting multimodal deep features from the aligned multimodal feature set to obtain multimodal deep features;

[0061] Step S43: performing feature-level fusion on the multimodal deep features and performing feature interactive learning to obtain a fused feature vector;

[0062] Step S44: predict and generate risk confidence for the fused feature vector to obtain a risk confidence heat map.

[0063] The present invention ensures the consistency of time series spectral flow, enhanced motion vector field and thermal texture anomaly map in space and time through multimodal feature data alignment, laying the foundation for subsequent multimodal feature fusion. The deep learning model is used to extract the deep features of different modalities respectively, effectively capturing the fire-related information contained in different data sources. Feature-level fusion and interactive learning based on the attention mechanism not only fuse the features of different modalities, but also learn the relationship between them, highlight important features, and suppress noise interference. The risk confidence heat map finally generated shows the spatial distribution of fire risk in an intuitive way, providing a reliable decision-making basis for early warning and precise prevention and control of fires, and significantly improving the efficiency and accuracy of fire detection.

[0064] Preferably, step S5 comprises the following steps:

[0065] Step S51: collecting and preprocessing context information of the inspection area according to the sensor network deployment plan to obtain a context information data set;

[0066] Step S52: extracting risk areas from the risk confidence heat map to obtain risk area data; performing spatial information matching on the risk area data and the context information data set, and correlating the risk areas to obtain correlated risk area information;

[0067] Step S53: weighting the situational factors according to the associated risk area information to obtain situational factor weighted data; performing risk correction according to the situational factor weighted data to obtain situational corrected risk confidence;

[0068] Step S54: performing warning level classification according to the scenario-corrected risk confidence level to obtain warning level classification information; dynamically adjusting the threshold of the warning level classification information according to the scenario information data set to obtain warning level information;

[0069] Step S55: Generate and push an intelligent warning report according to the warning level information, the associated risk area information and the time series spectrum flow to obtain an intelligent warning report.

[0070] The present invention realizes intelligent assessment and early warning of fire risks by combining situational information, risk confidence heat map and time-series spectral stream collected by sensor networks. Situational information collection and preprocessing provide rich contextual information for risk correction. Spatial information matching and risk area association link risks with specific scenarios and equipment to achieve accurate risk positioning. Situational factor weighting and risk correction effectively improve the accuracy of risk assessment. Warning level division and dynamic adjustment of thresholds optimize the warning level according to real-time situational information to avoid false alarms and missed alarms. Finally, the generation and push of intelligent warning reports provide fire safety management personnel with timely, accurate and comprehensive fire risk information and disposal suggestions, thereby effectively improving the efficiency and reliability of fire prevention and control.

[0071] Preferably, the present invention further provides a fire safety inspection system based on machine vision recognition, which is used to execute the fire safety inspection method based on machine vision recognition as described above. The fire safety inspection system based on machine vision recognition includes:

[0072] The multi-spectral dynamic perception module is used to deploy inspection equipment in the inspection area and obtain the camera deployment plan and the sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain the time-series spectral flow;

[0073] The micro-motion amplification module is used to perform multi-spectral channel motion estimation on the time-series spectral stream and perform motion vector noise suppression to obtain a filtered motion vector map; perform motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; apply a motion amplification algorithm to the time-series spectral stream according to the spectrally enhanced motion vector, and perform an enhanced motion vector field construction to obtain an enhanced motion vector field;

[0074] The thermal texture anomaly analysis module is used to extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence based on the thermal texture change map to obtain a thermal texture anomaly map;

[0075] The multimodal feature fusion module is used to perform multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain the fused feature vector; the risk confidence level of the fused feature vector is predicted and generated to obtain the risk confidence heat map;

[0076] The situational awareness and early warning module is used to collect and preprocess situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness and early warning based on the associated risk area information and the situational information data set to obtain an intelligent early warning report. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of the steps of a fire safety inspection method based on machine vision recognition;

[0078] Figure 2 Detailed implementation flow chart of step S2 in the present invention;

[0079] Figure 3 It is a schematic diagram of the detailed implementation steps of step S3 in the present invention.

[0080] The purpose, features and advantages of the present invention will be further described with reference to the accompanying drawings in conjunction with the embodiments. DETAILED DESCRIPTION

[0081] The technical method of the present invention is described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by technicians in this field without creative work are within the scope of protection of the present invention.

[0082] In addition, the accompanying drawings are only schematic illustrations of the present invention and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. The functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor methods and / or microcontroller methods.

[0083] It should be understood that, although the terms "first", "second", etc. may be used herein to describe various units, these units should not be limited by these terms. These terms are used only to distinguish one unit from another unit. For example, without departing from the scope of the exemplary embodiments, the first unit may be referred to as the second unit, and similarly the second unit may be referred to as the first unit. The term "and / or" used herein includes any and all combinations of one or more of the listed associated items.

[0084] To achieve this, please refer to Figures 1 to 3, a fire safety inspection method based on machine vision recognition, comprising the following steps:

[0085] Step S1: Deploy inspection equipment in the inspection area to obtain a camera deployment plan and a sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain a time-series spectral stream;

[0086] Step S2: performing multi-spectral channel motion estimation on the time series spectral stream and performing motion vector noise suppression to obtain a filtered motion vector map; performing motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; applying a motion amplification algorithm to the time series spectral stream according to the spectrally enhanced motion vector, and performing an enhanced motion vector field construction to obtain an enhanced motion vector field;

[0087] Step S3: extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map;

[0088] Step S4: performing multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain a fused feature vector; performing risk confidence prediction and generation on the fused feature vector to obtain a risk confidence heat map;

[0089] Step S5: Collect and preprocess the situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness warning based on the associated risk area information and the situational information data set to obtain an intelligent warning report.

[0090] In the embodiment of the present invention, reference Figure 1 FIG. 1 is a flow chart of the steps of the fire safety inspection method based on machine vision recognition of the present invention. In this example, the fire safety inspection method based on machine vision recognition includes the following steps:

[0091] Step S1: Deploy inspection equipment in the inspection area to obtain a camera deployment plan and a sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain a time-series spectral stream;

[0092] In the embodiment of the present invention, a multispectral camera with a specific band and resolution is selected, optimized and installed according to the layout of the inspection area and environmental factors, and initially calibrated to form a camera deployment plan. A wireless network containing multiple sensors is deployed simultaneously to form a sensor network deployment plan. Subsequently, multispectral images are collected synchronously at a high frame rate, and preprocessing such as radiation calibration, denoising and geometric correction is performed, and channel alignment and fusion are performed, and finally integrated into a time-series spectral stream containing multispectral channel information.

[0093] Step S2: performing multi-spectral channel motion estimation on the time series spectral stream and performing motion vector noise suppression to obtain a filtered motion vector map; performing motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; applying a motion amplification algorithm to the time series spectral stream according to the spectrally enhanced motion vector, and performing an enhanced motion vector field construction to obtain an enhanced motion vector field;

[0094] In the embodiment of the present invention, high-precision motion estimation is performed on each spectral channel, and noise is suppressed by using methods such as median filtering and amplitude threshold to screen out small motion vectors. Then, the motion vectors related to early fire characteristics are selectively enhanced in combination with spectral feature information. Next, the enhanced motion vectors are used to guide the motion amplification algorithm to highlight small motions. Finally, the motion of the amplified image is re-estimated and fused with the spectral enhanced motion vector to construct an enhanced motion vector field containing rich small motion information.

[0095] Step S3: extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map;

[0096] In an embodiment of the present invention, first, an infrared image sequence in a time-series spectral stream is extracted and preprocessed. Then, the LBP operator is used to extract local texture features of the thermal image, and the difference in texture features between adjacent frames is calculated to obtain a thermal texture change map. A threshold is set to generate a binary anomaly mask, and independent abnormal areas are identified through connected component analysis. Next, image blocks of these abnormal areas are extracted from the preprocessed thermal image, and their average temperature, maximum temperature, temperature standard deviation, and LBP histogram statistical features are calculated. Finally, based on these temperature and texture features, a pre-trained classification model or set rules is used to classify the abnormal pattern, such as distinguishing it as overheating, smoke, or flame, and marking different types of abnormal areas on the thermal image to generate a thermal texture anomaly map containing location, type, and detailed feature information.

[0097] Step S4: performing multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain a fused feature vector; performing risk confidence prediction and generation on the fused feature vector to obtain a risk confidence heat map;

[0098] In an embodiment of the present invention, the time series spectral stream, enhanced motion vector field and thermal texture anomaly map are precisely aligned in space and time, and the data format is unified. Then, for data of different modalities, pre-trained 3D-CNN and 2D-CNN are used to extract spatiotemporal spectral features, spatial motion features and thermal texture depth features. Next, a fusion strategy based on the attention mechanism is adopted to learn the importance weights of different modal features, and the weighted features are spliced ​​and processed by the fully connected layer for feature interactive learning. Finally, through a Sigmoid-activated fully connected layer, the fused feature vector is mapped to a fire risk confidence between 0 and 1, and the confidence value of each pixel is visualized as a risk confidence heat map.

[0099] Step S5: collect and preprocess the situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness warning based on the associated risk area information and the situational information data set to obtain an intelligent warning report;

[0100] In an embodiment of the present invention, environmental data is collected from temperature, humidity and smoke sensors according to the sensor network deployment scheme, and data cleaning is performed to construct a context information data set. According to the risk confidence heat map generated in step S44, the risk area with a confidence higher than 0.6 is extracted, and its minimum circumscribed rectangle and center point coordinates are calculated. The risk area is spatially matched with the context information data set, and nearby equipment, flammable and explosive items, environmental parameters and historical alarm records are associated to generate associated risk area information. Different context factors (for example, flammable and explosive items, electrical equipment, high temperature, low humidity, historical alarms) are weighted according to preset weight values, and the context correction factor is calculated to adjust the original risk confidence to obtain the context-corrected risk confidence. Then, according to the risk confidence after context correction, the warning level threshold is dynamically adjusted in combination with the current environmental information, and the risk area is divided into different warning levels such as low, medium, high, and emergency. Finally, for medium, high, and emergency risk areas, visual annotations are made on the visible light images of the time-series spectral stream, and combined with the associated equipment, environmental parameters, and historical alarm information, potential causes of fire hazards are analyzed, and an intelligent warning report is generated that includes the location of the risk area, warning level, potential causes, and disposal suggestions. The report is pushed to relevant personnel via text messages and emails.

[0101] Preferably, step S1 comprises the following steps:

[0102] Step S11: deploying a multispectral camera in the inspection area and deploying a sensor network to obtain a camera deployment plan and a sensor network deployment plan;

[0103] Step S12: performing high-speed acquisition of multispectral image sequences according to the camera deployment scheme to obtain an original multispectral image stream;

[0104] Step S13: performing multispectral image preprocessing on the original multispectral image stream to obtain a corrected multispectral image sequence;

[0105] Step S14: performing multispectral image channel alignment and fusion on the corrected multispectral image sequence to obtain a multispectral image stack;

[0106] Step S15: integrating the time-series spectral information of the multispectral image stack to obtain a time-series spectral stream.

[0107] In the embodiment of the present invention, for the inspection area, a multispectral camera including visible light, near infrared and short-wave infrared bands and a resolution of not less than 1920×1080 pixels is selected according to the pre-determined coverage range, potential fire hazard types and early abnormal spectral features to be captured. Based on the layout diagram of the inspection area, and taking into account the lighting conditions and potential obstructions, the specific installation position, pitch angle and horizontal angle of the multispectral camera are determined to ensure that the target area is covered without blind spots. The laser rangefinder is used to accurately measure the distance between the camera installation height and the target area, and the camera posture is calibrated using a level to ensure the geometric accuracy of image acquisition. The camera model, serial number, installation location coordinates, pitch angle, horizontal angle and initial spectral calibration parameters are recorded to form a camera deployment plan. The sensor network is deployed simultaneously, and wireless sensor nodes including temperature sensors, humidity sensors and wind speed sensors are selected. The deployment density and location of the sensor nodes are determined according to the distribution of environmental characteristics of the inspection area, such as flammable and explosive storage areas and electrical equipment concentration areas. All sensor nodes are connected to the central data acquisition gateway using a star topology and are configured with a unique network address and data transmission protocol. Record the model, serial number, deployment location coordinates, and data transmission frequency of each sensor node to form a sensor network deployment plan.

[0108] According to the camera deployment plan formulated in step S11, the deployed multispectral camera is started. The image acquisition frame rate of the multispectral camera is set to 10 frames per second to capture rapidly changing early signs of fire. The image acquisition system is configured to ensure that images of the three spectral channels of visible light, near infrared and short-wave infrared can be acquired synchronously, and the time synchronization error is controlled within 1 millisecond. A hardware trigger method or a precise timestamp synchronization mechanism is used to avoid analysis errors caused by the time difference of image acquisition between channels. The collected original multispectral images are stored in a solid-state drive array in chronological order, and the storage format is a lossless compressed TIFF format to ensure image quality. A timestamp accurate to the millisecond level is recorded for each image frame, and metadata containing the camera ID, frame number, and spectral channel information is attached. This process produces a sequence of original multispectral images that have not been processed, are arranged in chronological order, and contain three spectral channels of visible light, near infrared, and short-wave infrared.

[0109] The original multispectral image stream generated in step S12 is preprocessed frame by frame. First, radiation calibration correction is performed for the three spectral channels of visible light, near infrared and short-wave infrared. The digital grayscale value of the original image is converted into a radiation brightness value with physical significance using the factory calibration parameters of the camera to eliminate the influence of sensor response non-uniformity and ambient light changes. The dark current correction method is used to remove the dark current noise of the sensor, and the calibration image is illuminated by an integrating sphere uniform light source for flat field correction to eliminate the response difference between sensor pixels. Secondly, a denoising algorithm based on wavelet transform, such as Daubechies wavelet, is used to suppress the noise of the image of each spectral channel, reduce random noise interference, and improve the image signal-to-noise ratio. During the denoising process, the number of wavelet decomposition layers is set to 4 layers, and the soft threshold method is used for threshold processing to retain the detailed information of the image as much as possible. Finally, if the camera is slightly displaced after deployment, a feature point-based image registration algorithm, such as SIFT or ORB features, is used to geometrically correct the current frame image and the reference frame image, and the correction accuracy is controlled at the sub-pixel level. This process generates a radiometrically calibrated, denoised, and geometrically corrected multispectral image sequence.

[0110] For the corrected multispectral image sequence generated in step S13, accurate alignment and fusion between spectral channels are performed. First, an image registration algorithm based on mutual information maximization is used. Taking the visible light channel image as the reference, the images of the near infrared and short-wave infrared channels are accurately registered respectively to ensure that the images of the three spectral channels achieve accurate pixel-level correspondence in space. The search window size of the mutual information maximization algorithm is set to 15×15 pixels. A multi-resolution registration strategy is adopted. Coarse registration is first performed on the low-resolution image, and then fine registration is performed on the high-resolution image. After the registration is completed, the spatial alignment accuracy between the images of different spectral channels is checked and fine-tuned to ensure that the position deviation of the key feature points is less than 0.5 pixels. Secondly, the registered images of the three channels of visible light, near infrared and short-wave infrared are stacked in channel order to form a three-channel image data structure. The image at each time point contains the information of the three spectral channels and can be regarded as a three-dimensional data cube with a data type of float32, which is convenient for subsequent unified processing. This process generates a multispectral image stack that is accurately aligned in space.

[0111] For the multispectral image stack generated in step S14, it is integrated in time order to form a four-dimensional data structure containing the time dimension. The multispectral image stacks of continuous time points are arranged in order from small to large according to the acquisition timestamp, and a four-dimensional tensor is constructed, whose dimensions are time, height, width and channel. The time dimension represents the time order of image acquisition, the height and width represent the spatial resolution of the image, and the channel dimension represents different spectral channels (visible light, near infrared, short-wave infrared). In order to reduce the noise interference in time, the sliding average filtering method is used to smooth the time series. The sliding window size is set to 3 frames, and the radiance brightness values ​​of each pixel in each spectral channel of 3 consecutive frames are averaged to obtain the smoothed spectral information. This process generates a final continuous image sequence containing three spectral channel information of visible light, near infrared and short-wave infrared, which records the spectral changes of the scene in the time dimension, and the data type is float32.

[0112] Preferably, step S2 comprises the following steps:

[0113] Step S21: performing multi-spectral channel motion estimation on the time series spectral stream to obtain a channel motion vector diagram;

[0114] Step S22: performing motion vector noise suppression and screening on the channel motion vector map to obtain a filtered motion vector map;

[0115] Step S23: performing motion enhancement based on spectral characteristics on the filtered motion vector map according to the time-series spectral stream to obtain a spectrally enhanced motion vector;

[0116] Step S24: using the spectral enhancement motion vector as guidance information, applying a motion amplification algorithm to the temporal spectral stream to obtain an amplified image sequence;

[0117] Step S25: constructing an enhanced motion vector field for the amplified image sequence according to the spectral enhanced motion vector to obtain an enhanced motion vector field.

[0118] As an example of the present invention, refer to Figure 2 As shown, in this example, step S2 includes:

[0119] Step S21: performing multi-spectral channel motion estimation on the time series spectral stream to obtain a channel motion vector diagram;

[0120] In an embodiment of the present invention, for the time-series spectral stream generated in step S15, motion estimation is performed on the image sequences of the three spectral channels of visible light, near infrared and short-wave infrared. A motion estimation algorithm based on pixel-level phase correlation is used to calculate the motion vector of each pixel between two adjacent frames. Specifically, for the image sequence of each spectral channel, an image block of 32×32 pixels is selected, and a search window is set around the corresponding position of the adjacent frame, and the search window size is 64×64 pixels. The phase correlation between the image block of the current frame and the image blocks of all possible positions in the search window of the next frame is calculated by fast Fourier transform, and the displacement corresponding to the peak of the phase correlation is the motion vector of the central pixel of the image block. The motion vector contains components in both horizontal and vertical directions, and the accuracy reaches the sub-pixel level. For each spectral channel, a vector field with the same size as the image is generated, and each pixel stores the motion vector information of the pixel in the horizontal and vertical directions between adjacent frames. This process generates motion vector maps of the visible light channel, the near infrared channel, and the short-wave infrared channel, respectively.

[0121] Step S22: performing motion vector noise suppression and screening on the channel motion vector map to obtain a filtered motion vector map;

[0122] In an embodiment of the present invention, noise suppression and screening are performed for the three channel motion vectors of visible light, near infrared and short-wave infrared generated in step S21. First, the median filtering method is used to suppress the noise in the motion vector. For the motion vector of each channel, a filter window of size 3×3 is selected, and the horizontal and vertical components of all motion vectors in the window are sorted respectively, and the median is taken as the motion vector component after filtering of the current pixel. This operation effectively removes isolated noise motion vectors with large amplitudes. Secondly, preliminary screening is performed based on the amplitude of the motion vector. The amplitude threshold is set to 0.1 pixel, and the motion vector with an amplitude less than the threshold is regarded as a small motion and retained; the motion vector with an amplitude greater than the threshold is regarded as a significant motion, such as caused by the movement of background objects, and is eliminated. Finally, for the motion vector of each channel, the consistency of the motion vector in the local area is checked. If the motion vector direction of a pixel point differs from the direction of most of the motion vectors in the surrounding 8 neighborhoods by more than 45 degrees, the motion vector is considered to be an outlier and is eliminated. This process generates filtered visible light channel motion vectors, near infrared channel motion vectors, and short-wave infrared channel motion vectors, respectively.

[0123] Step S23: performing motion enhancement based on spectral characteristics on the filtered motion vector map according to the time-series spectral stream to obtain a spectrally enhanced motion vector;

[0124] In the embodiment of the present invention, the motion enhancement based on spectral characteristics is performed for the filtered motion vector map generated in step S22 and the time series spectral stream generated in step S15. First, for each pixel, the radiance values ​​of the three spectral channels of visible light, near infrared and short-wave infrared are extracted to form a spectral feature vector. Spectral feature models of two early fire characteristics, early hot air disturbance and slight smoke, are established. The hot air disturbance model is set to have a higher radiance value in the short-wave infrared band, while the slight smoke model is set to have a higher scattering intensity in the visible light and near-infrared bands. The cosine similarity between the spectral feature vector of each pixel and the spectral models of hot air disturbance and slight smoke is calculated. Then, a spectral similarity weight map is constructed. For the motion vector of each pixel, if the similarity between its corresponding spectral feature and the hot air disturbance model is higher than 0.8, a higher weight, such as 1.5, is assigned to the motion vector; if the similarity with the slight smoke model is higher than 0.8, a weight of 1.2 is assigned; otherwise, a weight of 1.0 is assigned. Next, each motion vector in the filtered motion vector map is multiplied by the corresponding spectral similarity weight value to obtain a spectrally weighted motion vector map. Finally, the spectrally weighted motion vector maps of 5 consecutive frames are smoothed using the temporal filtering method. Exponential smoothing filtering is used with a smoothing factor set to 0.2 to enhance continuous micro-motion and suppress random noise interference. This process generates spectrally enhanced visible light channel motion vectors, near infrared channel motion vectors, and short-wave infrared channel motion vectors, respectively.

[0125] Step S24: using the spectral enhancement motion vector as guidance information, applying a motion amplification algorithm to the temporal spectral stream to obtain an amplified image sequence;

[0126] In an embodiment of the present invention, for the time-series spectral flow generated in step S15, the spectral enhancement motion vector generated in step S23 is used as the guiding information, and the motion amplification algorithm based on optical flow is applied. Specifically, the image sequences of the three spectral channels of visible light, near infrared and short-wave infrared are processed separately. The TV-L1 optical flow algorithm is selected to calculate the dense optical flow field between adjacent frames, and the optical flow field is the spectral enhancement motion vector. The motion amplification factor is set to 10, which means that the amplitude of the small movement is amplified by 10 times. According to the calculated optical flow field, the image Warping technology is used to perform pixel displacement on the image of the current frame according to the instruction of the optical flow field to generate the next frame image after amplification. For example, if the optical flow vector of a pixel point indicates that it has moved 0.1 pixels to the right, the information of the pixel point and its surrounding pixels is moved to the position of 0.1 pixels on the right according to a certain interpolation method. The image sequences of the three channels of visible light, near infrared and short-wave infrared are subjected to motion amplification processing respectively. This process generates a visible light image sequence, a near infrared image sequence and a short-wave infrared image sequence after motion amplification.

[0127] Step S25: constructing an enhanced motion vector field for the amplified image sequence according to the spectral enhanced motion vector to obtain an enhanced motion vector field;

[0128] In an embodiment of the present invention, an enhanced motion vector field is constructed for the amplified image sequence generated in step S24 in combination with the spectrally enhanced motion vector generated in step S23. First, the motion vector of the amplified image sequence is re-estimated. The motion estimation algorithm based on pixel-level phase correlation as in step S21 is used to calculate the motion vector of each pixel between adjacent frames after amplification. Since the tiny motion has been amplified, the amplitude of the motion vector estimated at this time is larger and easier to detect. Then, the spectrally enhanced motion vector is fused with the re-estimated motion vector. For each pixel, if the amplitude of its spectrally enhanced motion vector is greater than the amplitude of the re-estimated motion vector, the spectrally enhanced motion vector is selected as the final motion vector; otherwise, the re-estimated motion vector is selected. This fusion strategy aims to use spectral information to guide more accurate motion estimation. Finally, the final motion vector information is organized into an enhanced motion vector field. For each pixel, its horizontal and vertical motion vector components are stored. This process generates an enhanced motion vector field containing three channels: visible light, near infrared, and short-wave infrared.

[0129] Preferably, step S23 includes the following steps:

[0130] Step S231: Calculate the spectral characteristic response graph of the time series spectral stream to obtain a spectral characteristic response graph set;

[0131] Step S232: constructing a spectral similarity weight map according to the spectral feature response atlas to obtain a spectral similarity weight atlas;

[0132] Step S233: fusing the filtered motion vector map with the spectral weight according to the spectral similarity weight atlas to obtain a spectral weighted motion vector map;

[0133] Step S234: performing time domain motion vector smoothing and accumulation on the spectrally weighted motion vector map to obtain a time domain smoothed motion vector map;

[0134] Step S235: performing motion vector enhancement according to the time-domain smoothed motion vector image and the filtered motion vector image to obtain a spectrally enhanced motion vector.

[0135] In the embodiment of the present invention, for the time series spectral stream generated in step S15, the spectral response of the stream and the predefined fire characteristics is calculated pixel by pixel. First, the radiation brightness value of each pixel in the three channels of visible light, near infrared and short-wave infrared is extracted to form a three-dimensional spectral feature vector. Standard spectral response models of three fire characteristics, namely, early smoldering, open flame and overheating, are established. The early smoldering model is set to have a higher reflectivity in the near infrared band than in the visible light and short-wave infrared; the open flame model is set to have the highest radiation intensity in the visible light band, followed by the near infrared, and the short-wave infrared intensity is relatively low; the overheating model is set to have a radiation intensity in the short-wave infrared band significantly higher than that in the visible light and near infrared. Then, the cosine similarity between the spectral feature vector of each pixel and the three standard spectral response models is calculated. The cosine similarity range is from -1 to 1, and the closer the value is to 1, the higher the similarity. For each pixel, its similarity value with the early smoldering, open flame and overheating models is calculated respectively. Finally, the similarity values ​​of each pixel and the three fire characteristics are stored separately to generate three spectral characteristic response maps, corresponding to the early smoldering response map, the open flame response map and the overheating response map. The pixel value range of each response map is -1 to 1.

[0136] For the spectral feature response atlas generated in step S231, namely, the early smoldering response map, the open flame response map and the overheating response map, a spectral similarity weight map is constructed. First, three weight mapping functions are set, corresponding to the three fire characteristics of early smoldering, open flame and overheating. For early smoldering, when the pixel value of its response map is greater than 0.6, the weight mapping function outputs 1.2; when the pixel value is less than 0.3, it outputs 0.8; when it is between 0.3 and 0.6, it outputs linear interpolation. For open flame, when the pixel value of its response map is greater than 0.8, the weight mapping function outputs 1.5; when it is less than 0.2, it outputs 0.7; and the intermediate value is linearly interpolated. For overheating, when the pixel value of its response map is greater than 0.7, the weight mapping function outputs 1.3; when it is less than 0.4, it outputs 0.9; and the intermediate value is linearly interpolated. Then, the pixel values ​​of each spectral feature response map are substituted into the corresponding weight mapping function, and the spectral similarity weight value of each pixel is calculated. Finally, the weight values ​​of each pixel for early smoldering, open flame, and overheating are stored separately to generate three spectral similarity weight maps, corresponding to the early smoldering weight map, open flame weight map, and overheating weight map. The pixel value range of each weight map is set according to the mapping function.

[0137] For the filtered motion vector map generated in step S22 and the spectral similarity weight atlas generated in step S232, the spectral weight and motion vector are fused. First, for each motion vector (including horizontal and vertical components) in the filtered motion vector map, the weight value of the corresponding pixel is read from the early smoldering weight map, the open flame weight map and the overheating weight map. Then, a weighted average fusion strategy is adopted. The filtered motion vector is multiplied by the three weight values ​​respectively to obtain three weighted motion vectors. The final spectral weighted motion vector is obtained by averaging the weights of these three weighted motion vectors. For example, assuming that the early smoldering weight is w1, the open flame weight is w2, the overheating weight is w3, and the filtered motion vector is V, then the spectral weighted motion vector V'=(w1*V+w2*V+w3*V) / (w1+w2+w3). If the three weight values ​​are all zero, the spectral weighted motion vector keeps the filtered motion vector unchanged. Finally, the motion vector after each pixel fusion is stored to generate a spectral weighted motion vector map. The vector map contains two components: horizontal and vertical.

[0138] For the spectrally weighted motion vector map sequence generated in step S233, the temporal motion vector is smoothed and accumulated. The Kalman filter algorithm is used to perform temporal smoothing on the motion vector of each pixel. First, a uniform motion model is established, assuming that the motion state (position and speed) of the pixel changes linearly over time. The state vector contains the x-coordinate, y-coordinate, x-direction speed and y-direction speed of the pixel. The observation vector contains the spectrally weighted motion vector of the pixel in the current frame. Then, the state estimate and covariance matrix of the Kalman filter are initialized. For each frame of image, the state of the current frame is predicted based on the state estimate and motion model of the previous frame. Then, the state estimate is updated based on the spectrally weighted motion vector of the current frame. The process noise covariance matrix and the observation noise covariance matrix of the Kalman filter are set according to experience. Through Kalman filtering, random noise in the motion vector can be effectively smoothed and continuous motion information can be accumulated. Finally, the motion vector after filtering of each frame is stored to generate a temporal smoothed motion vector map sequence. The vector map contains two components, horizontal and vertical.

[0139] The final motion vector enhancement is performed for the time domain smoothed motion vector map generated in step S234 and the filtered motion vector map generated in step S22. First, the amplitude of the time domain smoothed motion vector map and the filtered motion vector map is calculated. For each pixel, its amplitude value in the time domain smoothed motion vector and the filtered motion vector is calculated respectively. Then, an enhancement threshold is set, for example, 0.05 pixels. If the amplitude of the time domain smoothed motion vector of a certain pixel is greater than the threshold, and its direction is consistent with the direction of the filtered motion vector (the direction angle is less than 45 degrees), the filtered motion vector is enhanced. The enhancement method is to amplify the amplitude of the filtered motion vector by 1.2 times, and the direction remains unchanged. If the amplitude of the time domain smoothed motion vector is less than the threshold, or the direction is inconsistent with the direction of the filtered motion vector, the spectral enhancement motion vector keeps the filtered motion vector unchanged. Finally, the enhanced motion vector of each pixel is stored to generate a spectral enhancement motion vector map. The vector map contains two components, horizontal and vertical.

[0140] Preferably, step S3 comprises the following steps:

[0141] Step S31: extracting an infrared image sequence from the time-series spectral stream, and performing a reverse geometric transformation using an enhanced motion vector field to obtain an aligned infrared image sequence;

[0142] Step S32: performing thermal image preprocessing on the aligned infrared image sequence to obtain a preprocessed thermal image sequence;

[0143] Step S33: extracting thermal texture features from the preprocessed thermal image sequence to obtain a thermal texture feature map;

[0144] Step S34: performing a thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map;

[0145] Step S35: locating and characterizing thermal texture anomalies of the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map.

[0146] As an example of the present invention, refer to Figure 3 As shown, in this example, step S3 includes:

[0147] Step S31: extracting an infrared image sequence from the time-series spectral stream, and performing a reverse geometric transformation using an enhanced motion vector field to obtain an aligned infrared image sequence;

[0148] In the embodiment of the present invention, for the time series spectral stream generated in step S15, an image sequence of the short-wave infrared channel is extracted as the initial infrared image sequence. The short-wave infrared band is more sensitive to the thermal radiation generated by the early fire. Then, for the enhanced motion vector field generated in step S25, the initial infrared image sequence is subjected to an inverse geometric transformation. Specifically, for each frame in the initial infrared image sequence, the pixel position is reversely offset according to the motion vector of the corresponding pixel in the enhanced motion vector field. For example, if the enhanced motion vector field indicates that a pixel moves from the previous frame to the current frame by 2 pixels to the right and 1 pixel downward, then in the inverse geometric transformation, the pixel value of the pixel position in the current frame is assigned to the position of the previous frame by 2 pixels to the left and 1 pixel upward. The inverse geometric transformation uses a bilinear interpolation method to fill the pixel values ​​to reduce image distortion. This operation is intended to eliminate the image deformation introduced by the previous motion amplification process and to realign the infrared image sequence with the original scene in space. This process generates an infrared image sequence that is spatially aligned with the original scene.

[0149] Step S32: performing thermal image preprocessing on the aligned infrared image sequence to obtain a preprocessed thermal image sequence;

[0150] In an embodiment of the present invention, a thermal image preprocessing operation is performed on the aligned infrared image sequence generated in step S31. First, a Gaussian filtering method is used to suppress noise in the infrared image. A Gaussian filter kernel of size 5×5 is selected, and the standard deviation is set to 1.5, and each frame of the infrared image is smoothed to reduce random noise interference. Secondly, the contrast of the thermal image is enhanced using a histogram equalization method. For each frame of infrared image, the histogram of its pixel grayscale value is calculated, and the cumulative distribution function is calculated. Then, the pixel grayscale value is remapped according to the cumulative distribution function so that the grayscale distribution of the image is more uniform, thereby enhancing the contrast of the image and making the temperature difference more obvious. Finally, the pixel grayscale value of the infrared image is linearly mapped to a range of 0 to 255, and normalized to facilitate subsequent texture feature calculations. This process generates a preprocessed thermal image sequence after noise suppression, contrast enhancement and normalization.

[0151] Step S33: extracting thermal texture features from the preprocessed thermal image sequence to obtain a thermal texture feature map;

[0152] In an embodiment of the present invention, thermal texture features are extracted for the preprocessed thermal image sequence generated in step S32. The local binary pattern (LBP) operator is used to extract the local texture information of the image. Specifically, for each frame of thermal image, a circular neighborhood with a radius of 3 pixels is selected with each pixel as the center, and 8 sampling points are evenly selected in the neighborhood. The grayscale value of the neighborhood sampling point is compared with the grayscale value of the center pixel. If the grayscale value of the neighborhood sampling point is greater than or equal to the grayscale value of the center pixel, it is marked as 1, otherwise it is marked as 0. The comparison results of the 8 sampling points are arranged in sequence to form an 8-bit binary number, which is the LBP value of the center pixel. The LBP value of each pixel of the entire image is calculated to generate an LBP texture image. The LBP value ranges from 0 to 255, and different LBP values ​​correspond to different local texture patterns. This process generates a sequence of LBP thermal texture feature maps.

[0153] Step S34: performing a thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map;

[0154] In an embodiment of the present invention, a thermal texture dynamic change analysis is performed on the LBP thermal texture feature map sequence generated in step S33. First, the LBP histogram of the LBP thermal texture feature map of two adjacent frames is calculated. For each frame of LBP image, the frequency of occurrence of LBP values ​​0 to 255 is counted to obtain a 256-dimensional LBP histogram. Then, the Bhattacharyya distance between the LBP histograms of two adjacent frames is calculated. The Bhattacharyya distance is used to measure the similarity between two probability distributions. Its value range is 0 to 1. The smaller the value, the more similar the two histograms are, and the smaller the texture change. The calculation formula is: Bhattacharyya distance = the cumulative sum of 1-sqrt(H1[i]*H2[i]), where H1 and H2 represent the LBP histograms of two adjacent frames, respectively. Finally, the calculated Bhattacharyya distance value is used as the thermal texture change value of the current frame to generate a thermal texture change map. The pixel value range of the thermal texture change map is 0 to 1, and the larger the value, the more significant the texture change. This process generates a sequence of thermal texture change maps.

[0155] Step S35: locating and characterizing thermal texture anomalies of the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map;

[0156] In an embodiment of the present invention, thermal texture anomaly positioning and characterization are performed for the thermal texture change map sequence generated in step S34 and the preprocessed thermal image sequence generated in step S32. First, a thermal texture change threshold is set, for example, 0.2. Pixels in the thermal texture change map whose pixel values ​​are greater than the threshold are marked as abnormal points, and a binary abnormal mask is generated. Then, a connected component analysis is performed on the binary abnormal mask, adjacent abnormal pixels are divided into the same abnormal area, and a unique label is assigned to each abnormal area. Next, a pixel set corresponding to each abnormal area is extracted from the preprocessed thermal image sequence, and the average temperature, maximum temperature, minimum temperature and temperature standard deviation of each abnormal area are calculated. At the same time, the LBP histogram of each abnormal area is calculated, and the mean, variance, skewness and kurtosis of the LBP histogram are extracted as texture features. Finally, the abnormal areas are classified according to predefined rules. For example, if the average temperature of an abnormal area is more than 5 degrees Celsius higher than the ambient temperature and the LBP histogram variance is large, it is marked as a potential hot spot; if the texture change value of an abnormal area is continuously high, but the temperature rise is not obvious, it is marked as a potential thermal air disturbance. This process generates a thermal texture anomaly map containing the location, temperature characteristics and texture characteristics of the abnormal area, and marks different types of anomalies.

[0157] Preferably, step S34 includes the following steps:

[0158] Step S341: Calculate the texture difference feature map on the thermal texture feature map to obtain the texture difference feature map;

[0159] Step S342: Establishing a motion amplitude characteristic map for the enhanced motion vector field to obtain a motion amplitude characteristic map;

[0160] Step S343: fusing the texture difference feature map and the motion amplitude feature map to obtain a fused feature score map;

[0161] Step S344: performing time series anomaly detection on the fused feature score graph to obtain an anomaly score graph;

[0162] Step S345: Generate a thermal texture change map according to the abnormal score map to obtain a thermal texture change map.

[0163] In an embodiment of the present invention, for the LBP thermal texture feature map sequence generated in step S33, the difference in texture features between adjacent frames is calculated. The specific operation is to calculate the absolute difference in the LBP values ​​of the corresponding pixels for two adjacent frames in the LBP thermal texture feature map sequence. For example, assuming that the LBP value of the pixel point (x, y) of the previous frame is LBP1(x, y), and the LBP value of the pixel point (x, y) of the next frame is LBP2(x, y), then the texture difference value of the pixel point is |LBP1(x, y)-LBP2(x, y)|. The texture difference values ​​of all pixels in the entire image are calculated to generate a texture difference feature map. The pixel value range of the texture difference feature map is 0 to 255, and the larger the value, the more significant the texture change near the pixel point. This process generates a sequence of texture difference feature maps.

[0164] For the enhanced motion vector field generated in step S25, the motion amplitude of each pixel is calculated. The specific operation is to extract the corresponding motion vector for each pixel in the enhanced motion vector field, and the motion vector contains two components, horizontal and vertical. The amplitude value of the motion vector is calculated using the Pythagorean theorem, that is, the motion amplitude = sqrt (horizontal component ^2 + vertical component ^2). The motion amplitude values ​​of all pixels in the entire image are calculated to generate a motion amplitude feature map. The pixel value range of the motion amplitude feature map is from 0 to the length of the image diagonal. The larger the value, the larger the motion amplitude of the pixel. This process generates a sequence of motion amplitude feature maps.

[0165] For the texture difference feature map sequence generated in step S341 and the motion amplitude feature map sequence generated in step S342, the texture difference and the motion amplitude are fused. The specific operation is to perform weighted fusion on the texture difference value and the motion amplitude value of the corresponding pixel points for the texture difference feature map and the motion amplitude feature map at the same moment. Set the weight of the texture difference to 0.6 and the weight of the motion amplitude to 0.4. The fused feature score is: fusion feature score = 0.6*texture difference value + 0.4*motion amplitude value. Calculate the fusion feature scores of all pixels in the entire image to generate a fusion feature score map. The pixel value range of the fusion feature score map is determined according to the range of the texture difference value and the motion amplitude value. The larger the value, the more significant the texture change and the larger the motion amplitude are near the pixel point. This process generates a sequence of fusion feature score maps.

[0166] For the fusion feature score map sequence generated in step S343, time series anomaly detection is performed. The specific operation is to extract the fusion feature score value of each pixel in the fusion feature score map sequence in the past 5 frames to form a time series. An anomaly detection method based on a sliding window is adopted. The sliding window size is set to 5 frames, and the mean and standard deviation of the fusion feature score values ​​of the past 5 frames of the current frame pixel are calculated. Then, the difference between the fusion feature score value of the current frame and the mean score value of the past 5 frames is calculated, and the difference is divided by the standard deviation of the score of the past 5 frames to obtain a Z-score value. The larger the Z-score value, the farther the fusion feature score of the current frame deviates from the average level of the past period of time, and the more likely it is an anomaly. Set the anomaly threshold to 2.0. If the Z-score value of the current frame is greater than 2.0, it is considered that the pixel point is abnormal. The Z-score values ​​of all pixels in the entire image are stored to generate an anomaly score map. The pixel value of the anomaly score map indicates the possibility of a thermal texture dynamic anomaly at the pixel point. This process generates an anomaly score map sequence.

[0167] For the abnormal score map sequence generated in step S344, the final thermal texture change map is generated. The specific operation is to directly output the abnormal score map of the current frame as the thermal texture change map. The pixel value of the thermal texture change map is the abnormal score of the pixel point. The higher the score, the more significant the dynamic change of the thermal texture in the area, and the more likely it is a sign of an early fire. The abnormal score map can be normalized, for example, the score value can be mapped to a grayscale range of 0 to 255 to facilitate subsequent visualization and analysis. This process generates the final thermal texture change map.

[0168] Preferably, step S35 includes the following steps:

[0169] Step S351: generating a binary anomaly mask for the thermal texture change map to obtain a binary anomaly mask;

[0170] Step S352: Connect the binary anomaly masks by component labeling to obtain a connected component labeling graph;

[0171] Step S353: extracting abnormal regions from the preprocessed thermal image sequence according to the connected component label graph to obtain an abnormal region image block set;

[0172] Step S354: Calculate the regional temperature statistical characteristics of the abnormal region image block set to obtain a regional temperature characteristic table;

[0173] Step S355: Calculate the regional texture features of the abnormal region image block set to obtain a regional texture feature table;

[0174] Step S356: classifying and marking abnormal patterns according to the regional temperature feature table and the regional texture feature table to obtain a marked abnormal region map;

[0175] Step S357: Generate a thermal texture anomaly map based on the marked abnormal area map, the regional temperature feature table, and the regional texture feature table to obtain a thermal texture anomaly map.

[0176] In an embodiment of the present invention, a binary anomaly mask is generated for the thermal texture change map generated in step S345. The specific operation is to set an anomaly threshold, which is set to 0.7 based on historical data statistical analysis or expert experience. Traverse each pixel point in the thermal texture change map. If the thermal texture change value of the pixel point is greater than or equal to 0.7, the value of the corresponding position of the pixel point in the binary anomaly mask is set to 255; otherwise, it is set to 0. This operation marks the areas with significant changes in the thermal texture change map. The binary anomaly mask is a single-channel image with pixel values ​​of only 0 or 255. This process generates a binary anomaly mask.

[0177] For the binary anomaly mask generated in step S351, the connected components are marked. The specific operation is to traverse each pixel in the binary anomaly mask using the eight-neighborhood connection method. If the current pixel value is 255, and there are pixels with a pixel value of 255 in its eight neighborhoods, they are marked as the same connected component and assigned the same tag value. For isolated abnormal pixels that are not adjacent to any other abnormal pixels, a unique tag value is also assigned. The tag value increases from 1. The connected component tag map is a single-channel image, and different pixel values ​​represent different abnormal areas. This process generates a connected component tag map.

[0178] For the connected component label graph generated in step S352 and the preprocessed thermal image sequence generated in step S32, the abnormal area is extracted. The specific operation is to traverse each label value (representing an abnormal area) in the connected component label graph. For each label value, in the current frame of the preprocessed thermal image sequence, all pixels whose pixel values ​​are equal to the label value are extracted, and these pixels constitute an abnormal area. The image blocks corresponding to the pixels of each abnormal area in the preprocessed thermal image are cut out to form an independent image block. The size and shape of each image block depends on the size and shape of the abnormal area. This process generates a set of image blocks containing multiple abnormal areas.

[0179] For the abnormal area image block set generated in step S353, the temperature statistical characteristics of each abnormal area are calculated. The specific operation is to traverse each abnormal area image block and calculate the average, maximum, minimum and standard deviation of the temperature values ​​of all pixels in the image block. The temperature value is obtained by converting the pixel value of the infrared image, and the conversion formula is determined according to the camera calibration parameters. For example, the average temperature calculation method is to add the temperature values ​​of all pixels in the area and then divide it by the total number of pixels. The maximum and minimum values ​​directly take the maximum and minimum values ​​of the pixel temperatures in the area. The standard deviation reflects the degree of discreteness of the temperature distribution in the area. The average temperature, maximum temperature, minimum temperature and standard deviation of each abnormal area are recorded in a table, each row of the table corresponds to an abnormal area, and each column corresponds to a temperature statistical feature. This process generates a regional temperature feature table.

[0180] For the abnormal region image block set generated in step S353, the texture features of each abnormal region are calculated. The specific operation is to traverse each abnormal region image block and calculate the local binary pattern (LBP) histogram of the image block. The calculation method of the LBP histogram is similar to that of step S341, and the frequency of occurrence of each LBP value in the image block is counted. Then, the statistical features of the LBP histogram are extracted, including the mean, variance, skewness and kurtosis. The mean reflects the center position of the LBP value distribution, the variance reflects the discrete degree of the LBP value distribution, the skewness reflects the symmetry of the LBP value distribution, and the kurtosis reflects the sharpness of the LBP value distribution. The statistical features of the LBP histogram of each abnormal region are recorded in a table, each row of the table corresponds to an abnormal region, and each column corresponds to a texture statistical feature. This process generates a regional texture feature table.

[0181] For the regional temperature feature table generated in step S354 and the regional texture feature table generated in step S355, the abnormal patterns are classified and marked. The specific operation is that for each abnormal area, the corresponding temperature feature and texture feature are combined into a feature vector. Use a pre-trained classification model (for example, a support vector machine or a random forest) to classify the feature vector to determine which predefined abnormal pattern the abnormality belongs to, such as overheating, early smoke or flame. The training data of the classification model comes from historical fire cases or simulated fire experiment data. According to the classification results, a corresponding label is assigned to each abnormal area, such as "overheating", "smoke" or "flame". Then, in the connected component labeling map, the pixels belonging to the same abnormal area are marked with corresponding colors or assigned corresponding label values. This process generates a labeled abnormal area map.

[0182] The final thermal texture anomaly atlas is generated for the marked abnormal area map generated in step S356, the regional temperature feature table generated in step S354, and the regional texture feature table generated in step S355. The specific operation is to superimpose the marked abnormal area on the preprocessed thermal image, and use different colors or bounding boxes to mark different types of abnormal areas. For example, the overheated area is marked in red, the smoke area is marked in blue, and the flame area is marked in yellow. Add a legend to the side or bottom of the atlas to explain the abnormal types represented by different colors or marks. At the same time, the detailed information of each abnormal area is displayed in the atlas. For example, when the mouse hovers over an abnormal area, the average temperature, maximum temperature, LBP histogram mean and other information of the area pop up. The temperature features and texture features of the abnormal area can be added to the side of the atlas in the form of a table. This process generates a thermal texture anomaly atlas containing the location, type and detailed feature information of the abnormal area.

[0183] Preferably, step S4 comprises the following steps:

[0184] Step S41: aligning multimodal feature data of the time series spectral stream, the enhanced motion vector field, and the thermal texture anomaly map to obtain an aligned multimodal feature set;

[0185] Step S42: extracting multimodal deep features from the aligned multimodal feature set to obtain multimodal deep features;

[0186] Step S43: performing feature-level fusion on the multimodal deep features and performing feature interactive learning to obtain a fused feature vector;

[0187] Step S44: predict and generate risk confidence for the fused feature vector to obtain a risk confidence heat map.

[0188] In an embodiment of the present invention, an alignment operation of multimodal feature data is performed for the time series spectral stream generated in step S15, the enhanced motion vector field generated in step S25, and the thermal texture anomaly map generated in step S357. First, the spatial resolution and pixel coordinate system of the visible light channel image of the time series spectral stream are used as the benchmark, and the spatial resolution of the enhanced motion vector field and the thermal texture anomaly map is unified to the benchmark. For the enhanced motion vector field, the dimension of the motion vector field is adjusted to be consistent with the visible light image by a bilinear interpolation method. For the thermal texture anomaly map, if its resolution is different from that of the visible light image, the nearest neighbor interpolation method is used to adjust its resolution. Secondly, ensure that the data of different modes are synchronized in time. Assuming that the frame rate of the time series spectral stream is 10 frames / second, and the enhanced motion vector field and the thermal texture anomaly map are also generated at the same frame rate, no additional time synchronization operation is required. If the frame rates of the data of different modalities are different, a time interpolation or extraction operation is required to align them in time. Finally, the aligned multimodal data is combined into a multi-channel data set. For each image at each time point, it contains image data of three spectral channels: visible light, near infrared, and short-wave infrared, as well as the corresponding enhanced motion vector field (including horizontal and vertical components) and thermal texture anomaly map (including anomaly category labels). This process generates an aligned multimodal feature set.

[0189] For the aligned multimodal feature set generated in step S41, the depth features of different modalities are extracted respectively. First, for the time series spectral stream, a pre-trained three-dimensional convolutional neural network (3D-CNN), such as ResNet3D-18, is used to extract spatiotemporal spectral features. The network is pre-trained on the ImageNet video dataset and then fine-tuned on this inspection dataset. The multispectral image at each time point is input into the 3D-CNN, and the output of the last convolution layer is extracted as the spectral depth feature. Secondly, for the enhanced motion vector field, a two-dimensional convolutional neural network (2D-CNN), such as VGG16, is used to extract spatial motion features. The horizontal and vertical components of the enhanced motion vector field are input as two input channels into the 2D-CNN, and the output of the last pooling layer is extracted as the motion depth feature. The network is also pre-trained and fine-tuned on the ImageNet dataset. Finally, for the thermal texture anomaly map, another 2D-CNN, such as MobileNetV2, is used to extract the thermal texture depth feature. The thermal texture anomaly map is used as input, and the output of the last convolution layer is extracted as the thermal texture depth feature. This process generates spectral depth features, motion depth features, and thermal texture depth features.

[0190] Feature-level fusion and interactive learning are performed for the spectral depth features, motion depth features, and thermal texture depth features generated in step S42. A fusion strategy based on the attention mechanism is adopted. First, the depth features of the three modalities are input into three independent attention modules respectively. Each attention module contains a fully connected layer and a Softmax activation function, which is used to learn the importance weight of the modal feature. Then, the deep features of each modality are multiplied by their corresponding attention weights to obtain a weighted feature representation. Next, the weighted three modal features are concatenated to obtain a fused feature vector. In order to learn the interactive information between different modal features, several fully connected layers are added after the concatenated feature vector, and the Dropout regularization method is used to prevent overfitting. Through end-to-end training, the network learns the correlation between different modal features, and assigns different weights to different modal features, highlighting the features that are more important for fire hazard judgment. This process generates a fused feature vector.

[0191] For the fused feature vector generated in step S43, the fire risk confidence is predicted. The specific operation is to connect a fully connected layer at the end of the fused feature vector, which contains an output unit, and use the Sigmoid activation function to map the fused feature vector to a scalar value between 0 and 1, which is the confidence of the fire risk. The closer the confidence value is to 1, the higher the possibility of fire hazards in the area. Then, the risk confidence value of each pixel is mapped to a heat map. The heat map is coded in pseudo-color, for example, the confidence value 0 is mapped to blue, the confidence value 1 is mapped to red, and the intermediate values ​​are linearly interpolated. The risk confidence value of each pixel is filled into the corresponding position of the heat map to generate a risk confidence heat map. This process generates a risk confidence heat map.

[0192] Preferably, step S5 comprises the following steps:

[0193] Step S51: collecting and preprocessing context information of the inspection area according to the sensor network deployment plan to obtain a context information data set;

[0194] Step S52: extracting risk areas from the risk confidence heat map to obtain risk area data; performing spatial information matching on the risk area data and the context information data set, and correlating the risk areas to obtain correlated risk area information;

[0195] Step S53: weighting the situational factors according to the associated risk area information to obtain situational factor weighted data; performing risk correction according to the situational factor weighted data to obtain situational corrected risk confidence;

[0196] Step S54: performing warning level classification according to the scenario-corrected risk confidence level to obtain warning level classification information; dynamically adjusting the threshold of the warning level classification information according to the scenario information data set to obtain warning level information;

[0197] Step S55: Generate and push an intelligent warning report according to the warning level information, the associated risk area information and the time series spectrum flow to obtain an intelligent warning report.

[0198] In the embodiment of the present invention, according to the sensor network deployment scheme formulated in step S11, environmental parameter data is collected in real time from the temperature sensor, humidity sensor and smoke sensor deployed in the inspection area. The data collection frequency is once per minute. The collected raw data is preprocessed, firstly, data cleaning is performed to remove abnormal data that exceeds the sensor range or is obviously wrong, such as data with a temperature value exceeding 100 degrees Celsius or a negative humidity value. Secondly, data verification is performed, and the data collected three times in a row are smoothed using a sliding average filtering method, and the window size is 3. Then the data is formatted, and the collected temperature, humidity and smoke concentration values ​​are converted to floating point types, and corresponding timestamp information is added. At the same time, the pre-stored static scene semantic information of the inspection area is loaded, including the CAD Vector Drawings drawings of the inspection area, the equipment layout information (including equipment name, model, location coordinates) and the distribution map of flammable and explosive items (including item name, quantity, storage location). The CAD drawings are converted into regional information represented by polygons, and spatially associated with the equipment layout and the distribution information of flammable and explosive items to determine the area where each device and flammable and explosive item is located. In addition, the historical fire alarm records of the inspection area in the past year are retrieved from the historical database, including the alarm time, alarm type and alarm location. This process generates a context information dataset containing real-time environmental parameters, static scene semantic information and historical alarm records.

[0199] For the risk confidence heat map generated in step S44, risk areas are extracted. The risk threshold is set to 0.6, and the pixels with pixel values ​​greater than or equal to 0.6 in the risk confidence heat map are marked as high-risk pixels. Then, the eight-neighborhood connected component analysis algorithm is used to connect adjacent high-risk pixels into a risk area. The minimum circumscribed rectangle is calculated for each independent risk area, and the coordinates, width and height of its center point are recorded. This process generates risk area data containing multiple risk areas. Next, each extracted risk area is spatially matched with the context information data set generated in step S51. Using the center point coordinates of the risk area, it is determined whether the risk area is located in a predefined area (for example, a flammable and explosive storage area, an electrical equipment area). If the spatial distance between the risk area and a certain device or flammable and explosive material is less than 1 meter, the risk area is considered to be associated with the device or flammable and explosive material. At the same time, the real-time temperature, humidity and smoke concentration values ​​corresponding to the risk area, as well as the historical alarm records of the area, are obtained. This process generates associated risk area information, including the location of each risk area, risk confidence, and related environmental parameters, scene semantic information, and historical alarm records.

[0200] For the associated risk area information generated in step S52, the situational factors are weighted. Weight values ​​are set for different situational factors, and the weight value ranges from 0 to 1, indicating the degree of influence of the factor on the fire risk. For example, if the risk area is located in the flammable and explosive storage area, the weight of the flammable and explosive in the area is set to 0.8; if it is located in the electrical equipment area, the weight of the electrical equipment is set to 0.7; if the current ambient temperature is higher than 35 degrees Celsius, the high temperature weight is set to 0.6; if the humidity is lower than 30%, the low humidity weight is set to 0.5; if a fire alarm has occurred in the area in the past year, the historical alarm weight is set to 0.9. This process generates situational factor weighted data containing each risk area and its corresponding situational factor weight. Then, the original risk confidence is corrected according to the situational factor weighted data. For each risk area, its situational correction factor is calculated, and the calculation formula is: situational correction factor = 1 + (flammable and explosive weight + electrical equipment weight + high temperature weight + low humidity weight + historical alarm weight) / 5. Multiply the raw risk confidence by the scenario correction factor to get the scenario-corrected risk confidence. The scenario-corrected risk confidence value may be greater than 1. This process generates the scenario-corrected risk confidence.

[0201] According to the situation-corrected risk confidence generated in step S53, the warning level is divided. Four warning levels are set: low, medium, high and emergency. The thresholds for the division of warning levels are as follows: if the situation-corrected risk confidence is less than 0.7, it is a low risk; if it is greater than or equal to 0.7 and less than 0.85, it is a medium risk; if it is greater than or equal to 0.85 and less than 1.0, it is a high risk; if it is greater than or equal to 1.0, it is an emergency risk. This process generates warning level division information including each risk area and its corresponding initial warning level. Then, according to the situation information data set generated in step S51, the threshold of the warning level division is dynamically adjusted. If the average temperature of the current inspection area is higher than 30 degrees Celsius, the thresholds of all warning levels are reduced by 0.05; if the wind speed in the current area is greater than 5 meters / second, the thresholds of high risk and emergency risk are reduced by 0.1. The dynamically adjusted thresholds are used for the final warning level judgment. This process generates warning level information including each risk area and its final warning level.

[0202] According to the warning level information generated in step S54, the associated risk area information generated in step S52, and the time series spectrum stream generated in step S15, an intelligent warning report is generated. First, according to the warning level information, the risk area for which a warning report needs to be generated is determined. A warning report is generated only for risk areas with medium, high or emergency warning levels. Then, for each risk area for which a warning report needs to be generated, a red border Highlight is used on the visible light image of the time series spectrum stream to display the location of the risk area and mark its warning level. According to the equipment or flammable and explosive information related to the risk area recorded in the associated risk area information, as well as the current environmental parameters and historical alarm records, the possible causes of fire hazards in the risk area are analyzed, such as "equipment overheating risk" and "flammable leakage risk". According to the risk level and potential causes, corresponding disposal suggestions are given, such as "it is recommended to strengthen monitoring", "it is recommended to send someone to check immediately", and "it is recommended to start an emergency plan". The location of the risk area, warning level, potential cause analysis and disposal suggestions are integrated into a structured report, and the report format is JSON or XML. Finally, the generated intelligent warning report is pushed to the preset fire safety management personnel and patrol personnel via SMS and email. This process generates an intelligent early warning report.

[0203] Preferably, the present invention further provides a fire safety inspection system based on machine vision recognition, which is used to execute the fire safety inspection method based on machine vision recognition as described above. The fire safety inspection system based on machine vision recognition includes:

[0204] The multi-spectral dynamic perception module is used to deploy inspection equipment in the inspection area and obtain the camera deployment plan and the sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain the time-series spectral flow;

[0205] The micro-motion amplification module is used to perform multi-spectral channel motion estimation on the time-series spectral stream and perform motion vector noise suppression to obtain a filtered motion vector map; perform motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; apply a motion amplification algorithm to the time-series spectral stream according to the spectrally enhanced motion vector, and perform an enhanced motion vector field construction to obtain an enhanced motion vector field;

[0206] The thermal texture anomaly analysis module is used to extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence based on the thermal texture change map to obtain a thermal texture anomaly map;

[0207] The multimodal feature fusion module is used to perform multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain the fused feature vector; the risk confidence level of the fused feature vector is predicted and generated to obtain the risk confidence heat map;

[0208] The situational awareness and early warning module is used to collect and preprocess situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness and early warning based on the associated risk area information and the situational information data set to obtain an intelligent early warning report.

[0209] Therefore, the embodiments should be regarded as illustrative and non-restrictive from all points, and the scope of the present invention is limited by the appended claims rather than the above description, and it is therefore intended that all changes falling within the meaning and range of equivalent elements of the application documents are included in the present invention.

[0210] The above description is only a specific embodiment of the present invention, so that those skilled in the art can understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but should conform to the widest scope consistent with the principles and novel features invented herein.

Claims

1. A fire safety inspection method based on machine vision recognition, characterized in that: The following steps are involved: Step S1: Deploy inspection equipment in the inspection area to obtain a camera deployment plan and a sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain a time-series spectral stream; Step S2: performing multi-spectral channel motion estimation on the time series spectral stream and performing motion vector noise suppression to obtain a filtered motion vector map; performing motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; Applying motion amplification algorithm to the temporal spectral stream according to the spectral enhancement motion vector, and constructing the enhanced motion vector field to obtain the enhanced motion vector field; Step S3: extracting thermal texture features from the time series spectrum stream to obtain a thermal texture feature map; Performing thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; According to the thermal texture change map, the thermal texture anomaly is located and characterized for the preprocessed thermal image sequence to obtain a thermal texture anomaly map; Step S4: performing multimodal feature fusion on the time series spectral stream, enhanced motion vector field and thermal texture anomaly map to obtain a fused feature vector; Predict and generate risk confidence for the fused feature vector to obtain a risk confidence heat map; Step S5: collecting and preprocessing the situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; Extract risk area information from the risk confidence heat map to obtain associated risk area information; Based on the associated risk area information and situational information data set, situational awareness warning is carried out to obtain an intelligent warning report.

2. The fire safety inspection method based on machine vision recognition according to claim 1 is characterized in that: Step S1 includes the following steps: Step S11: deploying a multispectral camera in the inspection area and deploying a sensor network to obtain a camera deployment plan and a sensor network deployment plan; Step S12: performing high-speed acquisition of multispectral image sequences according to the camera deployment scheme to obtain an original multispectral image stream; Step S13: performing multispectral image preprocessing on the original multispectral image stream to obtain a corrected multispectral image sequence; Step S14: performing multispectral image channel alignment and fusion on the corrected multispectral image sequence to obtain a multispectral image stack; Step S15: integrating the time-series spectral information of the multispectral image stack to obtain a time-series spectral stream.

3. The fire safety inspection method based on machine vision recognition according to claim 1 is characterized in that: Step S2 includes the following steps: Step S21: performing multi-spectral channel motion estimation on the time series spectral stream to obtain a channel motion vector diagram; Step S22: performing motion vector noise suppression and screening on the channel motion vector map to obtain a filtered motion vector map; Step S23: performing motion enhancement based on spectral characteristics on the filtered motion vector map according to the time-series spectral stream to obtain a spectrally enhanced motion vector; Step S24: using the spectral enhancement motion vector as guidance information, applying a motion amplification algorithm to the temporal spectral stream to obtain an amplified image sequence; Step S25: constructing an enhanced motion vector field for the amplified image sequence according to the spectral enhanced motion vector to obtain an enhanced motion vector field.

4. The fire safety inspection method based on machine vision recognition according to claim 3 is characterized in that: Step S23 includes the following steps: Step S231: Calculate the spectral characteristic response graph of the time series spectral stream to obtain a spectral characteristic response graph set; Step S232: constructing a spectral similarity weight map according to the spectral feature response atlas to obtain a spectral similarity weight atlas; Step S233: fusing the filtered motion vector map with the spectral weight according to the spectral similarity weight atlas to obtain a spectral weighted motion vector map; Step S234: performing time domain motion vector smoothing and accumulation on the spectrally weighted motion vector map to obtain a time domain smoothed motion vector map; Step S235: performing motion vector enhancement according to the time-domain smoothed motion vector image and the filtered motion vector image to obtain a spectrally enhanced motion vector.

5. The fire safety inspection method based on machine vision recognition according to claim 1 is characterized in that: Step S3 includes the following steps: Step S31: extracting an infrared image sequence from the time-series spectral stream, and performing a reverse geometric transformation using an enhanced motion vector field to obtain an aligned infrared image sequence; Step S32: performing thermal image preprocessing on the aligned infrared image sequence to obtain a preprocessed thermal image sequence; Step S33: extracting thermal texture features from the preprocessed thermal image sequence to obtain a thermal texture feature map; Step S34: performing a thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; Step S35: locating and characterizing thermal texture anomalies of the preprocessed thermal image sequence according to the thermal texture change map to obtain a thermal texture anomaly map.

6. The fire safety inspection method based on machine vision recognition according to claim 5 is characterized in that: Step S34 includes the following steps: Step S341: Calculate the texture difference feature map on the thermal texture feature map to obtain the texture difference feature map; Step S342: Establishing a motion amplitude characteristic map for the enhanced motion vector field to obtain a motion amplitude characteristic map; Step S343: fusing the texture difference feature map and the motion amplitude feature map to obtain a fused feature score map; Step S344: performing time series anomaly detection on the fused feature score graph to obtain an anomaly score graph; Step S345: Generate a thermal texture change map according to the abnormal score map to obtain a thermal texture change map.

7. The fire safety inspection method based on machine vision recognition according to claim 5 is characterized in that: Step S35 includes the following steps: Step S351: generating a binary anomaly mask for the thermal texture change map to obtain a binary anomaly mask; Step S352: Connect the binary anomaly masks by component labeling to obtain a connected component labeling graph; Step S353: extracting abnormal regions from the preprocessed thermal image sequence according to the connected component label graph to obtain an abnormal region image block set; Step S354: Calculate the regional temperature statistical characteristics of the abnormal region image block set to obtain a regional temperature characteristic table; Step S355: Calculate the regional texture features of the abnormal region image block set to obtain a regional texture feature table; Step S356: classifying and marking abnormal patterns according to the regional temperature feature table and the regional texture feature table to obtain a marked abnormal region map; Step S357: Generate a thermal texture anomaly map based on the marked abnormal area map, the regional temperature feature table, and the regional texture feature table to obtain a thermal texture anomaly map.

8. The fire safety inspection method based on machine vision recognition according to claim 1 is characterized in that: Step S4 includes the following steps: Step S41: aligning multimodal feature data of the time series spectral stream, the enhanced motion vector field, and the thermal texture anomaly map to obtain an aligned multimodal feature set; Step S42: extracting multimodal deep features from the aligned multimodal feature set to obtain multimodal deep features; Step S43: performing feature-level fusion on the multimodal deep features and performing feature interactive learning to obtain a fused feature vector; Step S44: predict and generate risk confidence for the fused feature vector to obtain a risk confidence heat map.

9. The fire safety inspection method based on machine vision recognition according to claim 1 is characterized in that: Step S5 includes the following steps: Step S51: collecting and preprocessing context information of the inspection area according to the sensor network deployment plan to obtain a context information data set; Step S52: extracting risk areas from the risk confidence heat map to obtain risk area data; performing spatial information matching on the risk area data and the context information data set, and correlating the risk areas to obtain correlated risk area information; Step S53: weighting the situational factors according to the associated risk area information to obtain situational factor weighted data; performing risk correction according to the situational factor weighted data to obtain situational corrected risk confidence; Step S54: performing warning level classification according to the scenario-corrected risk confidence level to obtain warning level classification information; dynamically adjusting the threshold of the warning level classification information according to the scenario information data set to obtain warning level information; Step S55: Generate and push an intelligent warning report according to the warning level information, the associated risk area information and the time series spectrum flow to obtain an intelligent warning report.

10. A fire safety inspection system based on machine vision recognition, characterized in that: Used to execute the fire safety inspection method based on machine vision recognition as claimed in claim 1, the fire safety inspection system based on machine vision recognition comprises: The multi-spectral dynamic perception module is used to deploy inspection equipment in the inspection area and obtain the camera deployment plan and the sensor network deployment plan; perform multi-spectral dynamic perception according to the camera deployment plan to obtain the time-series spectral flow; The micro-motion amplification module is used to perform multi-spectral channel motion estimation on the time-series spectral stream and perform motion vector noise suppression to obtain a filtered motion vector map; perform motion enhancement based on spectral characteristics on the filtered motion vector map to obtain a spectrally enhanced motion vector; apply a motion amplification algorithm to the time-series spectral stream according to the spectrally enhanced motion vector, and perform an enhanced motion vector field construction to obtain an enhanced motion vector field; The thermal texture anomaly analysis module is used to extract thermal texture features from the time series spectral stream to obtain a thermal texture feature map; perform thermal texture dynamic change analysis on the thermal texture feature map to obtain a thermal texture change map; locate and characterize thermal texture anomalies on the preprocessed thermal image sequence based on the thermal texture change map to obtain a thermal texture anomaly map; The multimodal feature fusion module is used to perform multimodal feature fusion on the time series spectral flow, enhanced motion vector field and thermal texture anomaly map to obtain the fused feature vector; the risk confidence level of the fused feature vector is predicted and generated to obtain the risk confidence heat map; The situational awareness and early warning module is used to collect and preprocess situational information of the inspection area according to the sensor network deployment plan to obtain a situational information data set; extract risk area information from the risk confidence heat map to obtain associated risk area information; perform situational awareness and early warning based on the associated risk area information and the situational information data set to obtain an intelligent early warning report.

Images