Fire behavior identification method, system and equipment suitable for power grid inspection and medium

Abstract

The invention discloses a fire behavior identification method, system and device suitable for power grid inspection and a medium, and belongs to the technical field of mountain fire monitoring, and the method comprises the steps: obtaining multi-mode video data of a to-be-monitored region, and obtaining a synchronized multi-mode video frame according to the multi-mode video data; determining a baseline model of preliminary training convergence based on the synchronized multi-modal video frame and a preset forest fire recognition model; obtaining an optimized baseline model by obtaining unlabeled video data and combining the baseline model; and performing fire behavior identification based on the optimized baseline model and the synchronized multi-mode video frame to obtain a fire behavior identification result. The method has the beneficial effects that the accuracy and reliability of fire identification are improved, continuous and stable detection under the actual background condition across day and night is realized, and accurate and timely fire early warning guarantee is provided for power grid inspection and forest fire prevention.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of mountain fire monitoring, in particular to a fire identification method, system, device and medium suitable for power grid inspection. BACKGROUND

[0002] Early identification of forest fires is a key technical problem in forestry safety and power grid operation. Once a forest fire occurs, it will not only destroy a large number of forest resources and their ecosystems, but also threaten the safety of life and property of surrounding residents. In order to prevent the spread of the fire and organize timely fire fighting, efficient and reliable automatic fire identification technology is urgently needed. Traditional methods mainly rely on manual operation (such as observation from a lookout tower) or simple sensors (such as smoke alarms), which have the disadvantages of limited monitoring range, poor real-time performance, and large influence of human factors. In recent years, unmanned aerial vehicle inspection combined with deep learning image recognition has become a new solution for mountain fire monitoring: an unmanned aerial vehicle equipped with a camera patrols in the air to take pictures of the forest area, and a convolutional neural network model is used to automatically detect whether there is a fire in the picture. This solution expands the monitoring coverage and improves the response speed, but still faces many technical bottlenecks in practical application.

[0003] The existing fire identification technology faces many technical challenges in practical application. First, the traditional convolutional neural network focuses on the spatial feature extraction of static images and ignores the time dimension information. Under the interference of forest environment light conditions, light and dark changes, and tree shaking, it is difficult to distinguish between short-time floating smoke and static clouds, and false positives are prone to occur. In the early stage of a fire, smoke is often the main form, and the shape of the smoke is variable and easily confused with clouds, resulting in a high false positive rate of the model. At the same time, different fire scene differences are significant, and the infrared image features of forest fire and industrial spark, daytime and night are quite different. When the training sample coverage is insufficient, the robustness of a single model obviously decreases, and small fire target detection is particularly difficult. The initial flame or remote fire source only occupies a very small area of the image, and is often missed due to background noise. Second, mountain fire identification requires both reducing false positives and reducing false positives. Too many false positives can lead to frequent false alarms, consuming rescue resources and reducing system credibility. The source of false positives in a natural environment is numerous: sunlight reflection, high-temperature objects, vehicle lights, and cloud smoke can all be misidentified as fire features. Traditional model output threshold settings are fixed, and lack the ability to control environmental changes, making it difficult to balance false positives and missed calls. Although reinforcement learning methods provide a new way to reduce identification errors through a reward mechanism, it still needs to be further improved in terms of model structure and multi-source information verification to fully control the error rate. In addition, the existing technology lacks adaptability in multiple scenes, especially in the power grid inspection scene. The power transmission line passes through the forest area, and the arc spark caused by equipment failure may become a cause of forest fire. Compared with the forest environment, the image background in the power grid inspection scene is complex, and the high-light metal structures such as towers and wires in the picture interfere with each other. Fire is often manifested as equipment smoke or line spark, and the scale is small and the duration is short. Direct application of forest fire identification models can easily result in false positives and missed calls, such as misjudging high-voltage line arc light as flame, or low recognition rate of residual fire in the shadow of the equipment. SUMMARY

[0004] In view of the above problems, the present application is proposed.

[0005] Therefore, the technical problem solved by the present application is: how to address the deficiencies of existing fire identification technology, such as insufficient single modal information, lack of training data, and poor environmental adaptability, through multi-modal video data fusion, deep reinforcement learning training optimization, and self-supervised learning mechanism, to solve the problems of low fire identification accuracy and high false positive rate in the power grid inspection environment.

[0006] To solve the above technical problems, the present application provides the following technical solutions: a fire identification method suitable for power grid inspection, comprising,

[0007] The multi-modal video data of the to-be-monitored area is acquired, and synchronized multi-modal video frames are obtained according to the multi-modal video data; a baseline model preliminarily trained and converged is determined based on the synchronized multi-modal video frames and a preset forest fire identification model; an optimized baseline model is obtained by acquiring unlabeled video data and combining the baseline model; and fire condition identification is performed based on the optimized baseline model and the synchronized multi-modal video frames to obtain a fire condition identification result.

[0008] As a preferred scheme of the fire condition identification method suitable for power grid inspection, the multi-modal video data of the to-be-monitored area is acquired, and synchronized multi-modal video frames are obtained according to the multi-modal video data, which includes: acquiring a video data stream of the to-be-monitored area; performing time synchronization processing on the video data stream; and performing preprocessing on the synchronized video data stream to obtain synchronized multi-modal video frames.

[0009] As a preferred scheme of the fire condition identification method suitable for power grid inspection, the baseline model preliminarily trained and converged is determined based on the synchronized multi-modal video frames and a preset forest fire identification model, which includes: constructing a training data set based on the synchronized multi-modal video frames; taking the forest fire identification model as an agent and taking the video frames in the training data set as environment states; designing a reward mechanism to assign a reward value according to the matching condition of the identification result of the forest fire identification model and the true label; and iteratively optimizing the parameters of the forest fire identification model until the performance converges to obtain the baseline model preliminarily trained and converged. The preferred technical scheme has the beneficial effect that by taking the forest fire identification model as an agent and designing a reward mechanism to assign a reward value according to the matching condition of the identification result and the true label, the model learning of correct identification strategies can be effectively guided, the accuracy of fire condition identification can be improved, and the false positives and false negatives can be reduced.

[0010] As a preferred scheme of the fire condition identification method suitable for power grid inspection, the optimized baseline model is obtained by acquiring unlabeled video data and combining the baseline model, which includes: acquiring unlabeled video data; generating a predicted label and a confidence by using the baseline model to infer the unlabeled video data; determining a pseudo label according to the predicted label and the confidence; merging the sample with the pseudo label and the training data set, and optimizing the baseline model to obtain the optimized baseline model. The preferred technical scheme has the beneficial effect that by using the baseline model to infer the unlabeled video data to generate a pseudo label and using a confidence screening mechanism to ensure the quality of the pseudo label, the scale of the training data set can be effectively expanded, the problem of insufficient labeled data in fire condition identification can be solved, and the generalization ability of the model can be improved.

[0011] As a preferred scheme of the fire identification method suitable for power grid inspection, the fire identification based on the optimized baseline model and the synchronized multi-modal video frame comprises: extracting spatial features and time features of the synchronized multi-modal video frame through the optimized baseline model and performing feature fusion to generate comprehensive feature representation; and obtaining a fire identification result according to the comprehensive feature representation.

[0012] As a preferred scheme of the fire identification method suitable for power grid inspection, the extracting spatial features and time features of the synchronized multi-modal video frame through the optimized baseline model and performing feature fusion to generate comprehensive feature representation comprises: extracting a feature map of the multi-modal video frame; connecting and fusing the feature map of the multi-modal video frame in the middle layer by channel to obtain a fused feature map; performing weight distribution on the fused feature map, and combining the feature map weighted by attention with a time variation mode of a continuous frame to obtain the comprehensive feature representation.

[0013] As a preferred scheme of the fire identification method suitable for power grid inspection, the determining a pseudo label according to the predicted label and the confidence comprises: setting a preset confidence threshold; performing frame screening based on the confidence and the preset confidence threshold to obtain screened samples; and taking the predicted label of a frame as a pseudo label to obtain samples with pseudo labels.

[0014] Another object of the present application is to provide a fire identification system suitable for power grid inspection.

[0015] To solve the above technical problems, the present application provides the following technical scheme: a fire identification system suitable for power grid inspection, comprising: a video frame acquisition module configured to acquire multi-modal video data of a region to be monitored and obtain synchronized multi-modal video frames based on the multi-modal video data; a baseline model training module configured to determine a baseline model preliminarily trained to converge based on the synchronized multi-modal video frames and a preset forest fire identification model; a baseline model optimization module configured to obtain an optimized baseline model by acquiring unlabeled video data and combining the baseline model; and a fire identification module configured to perform fire identification based on the optimized baseline model and the synchronized multi-modal video frames to obtain a fire identification result.

[0016] The application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and wherein the processor implements steps of the fire identification method suitable for power grid inspection when executing the computer program.

[0017] The application provides a computer readable storage medium, which stores a computer program, wherein the computer program implements steps of the fire identification method suitable for power grid inspection when executed by a processor.

[0018] The application has the following beneficial effects: the application realizes multi-modal information acquisition of fire by simultaneously collecting visible light and infrared video data and performing time synchronization processing, identifies open fire and smoke morphology by visible light during the day, and captures temperature anomalies by infrared at night or in shielding conditions, thereby effectively making up for the blind area of a single sensor; the fire identification model is taken as an intelligent agent and a reward mechanism is designed for reinforcement learning training, so that the model is endowed with self-improvement ability, the dependence on massive manual labeling in pure supervised learning is avoided, and the shortcomings of slow convergence and unstable effect of traditional methods are overcome; the network structure combining a convolutional neural network and a long short-term memory network is adopted and an attention mechanism is introduced, so that spatial features can be extracted and time change patterns can be captured, the importance of features in each region is adjusted according to current input, the model focuses on key regions such as flames and smoke columns, the probability of misjudging transient smoke clouds as fire and the missed detection rate of weak fire sources are reduced; the pseudo-label self-supervised training strategy is introduced and a confidence threshold screening mechanism is adopted, so that the value of unlabeled data is fully tapped, the training data scale is expanded, and the generalization ability of the model to new environments is improved; not only the accuracy and reliability of fire identification are improved, but also continuous and stable detection under cross-day and night and actual background conditions is realized, thereby providing accurate and timely fire warning protection for power grid inspection and forest fire prevention. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings. Among them:

[0020] Figure 1 The application provides a fire identification method suitable for power grid inspection.

[0021] Figure 2 The application provides a fire identification method suitable for power grid inspection. DETAILED DESCRIPTION

[0022] In order to make the above objectives, features and advantages of the present application more apparent, the specific embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0023] In the following description, numerous specific details are set forth in order to provide a thorough understanding of the present application. However, the present application can be practiced without the specific details. In other instances, well-known methods have not been described in detail in order not to unnecessarily obscure aspects of the present application.

[0024] Secondly, the "one embodiment" or "embodiment" referred to herein means that the specific features, structures or characteristics can be included in at least one implementation of the present application. "In one embodiment" appearing in different places in the specification does not mean the same embodiment, nor is it an independent or alternative embodiment mutually exclusive with other embodiments.

[0025] Embodiment 1, with reference to Figure 1 For one embodiment of the present application, the embodiment provides a fire identification method suitable for power grid inspection, comprising:

[0026] S100: Obtain multi-modal video data of a region to be monitored, and obtain synchronized multi-modal video frames according to the multi-modal video data.

[0027] S200: Determine a preliminary training converged baseline model based on the synchronized multi-modal video frames and a preset forest fire identification model.

[0028] S300: Obtain an optimized baseline model by acquiring unlabeled video data and combining the baseline model.

[0029] S400: Perform fire identification based on the optimized baseline model and the synchronized multi-modal video frames to obtain a fire identification result.

[0030] It should be noted that in the fire identification of power grid inspection, the traditional fire identification method mainly relies on single modal visual information, which is easily affected by factors such as light changes, weather conditions, background interference and the like in power grid line environment and forest environment, resulting in low identification accuracy; at the same time, the existing fire identification model is usually trained in a supervised learning manner and seriously depends on a large amount of manually annotated training data, but in the actual power grid inspection scene, it is costly and time-consuming to obtain sufficient labeled data, especially the fire samples are relatively rare, resulting in insufficient model training data; in addition, there are various interference factors in the power grid inspection environment, such as dust smoke raised by cars, equipment reflection, cloud and mist shielding and the like, these instantaneous interferences have similarities with the real fire in visual features, and the traditional method is difficult to effectively distinguish and is easy to produce false alarms; and the existing identification method lacks effective use of time sequence information, and cannot fully capture the characteristics of fire development, such as the sustained combustion characteristics of flame and the drifting trajectory of smoke, affecting the reliability of identification.

[0031] Therefore, in view of the problems of insufficient single modal information, training data scarcity, complex interference factors and insufficient use of time characteristics in the fire identification of power grid inspection, through the steps of S100-S400, a multi-modal fusion fire identification model is constructed to realize the cooperative processing of infrared thermal imaging and visible light images, obtain more rich and complementary feature information; the model parameters are optimized through deep reinforcement learning to improve the adaptability of the model to the actual scene; unlabeled data is used for self-supervised learning to expand the training samples and improve the generalization performance of the model; at the same time, based on time sequence feature fusion and attention mechanism, the fire characteristics are accurately captured and the interference factors are effectively suppressed, so as to realize the accurate identification and reliable early warning of fire in power grid inspection.

[0032] Embodiment 2, refer to Figure 1 For an embodiment of the present application, a fire identification method suitable for power grid inspection is provided based on the previous embodiment, comprising:

[0033] In the embodiment of the present application, the multi-modal video data of the to-be-monitored region is obtained in step S100, and the synchronized multi-modal video frame is obtained according to the multi-modal video data, comprising the following steps A1-A3:

[0034] A1: Obtain the video data stream of the to-be-monitored region.

[0035] A2: Perform time synchronization processing on the video data stream.

[0036] A3: Preprocess the synchronized video data stream to obtain the synchronized multi-modal video frame.

[0037] Specifically, the video data stream of the to-be-monitored region is obtained in step A1, and the specific operation can be:

[0038]

[0038] acquire a visible light video data stream of the to-be-monitored region by using at least one visible light camera.

[0039] acquire an infrared thermal imaging video data stream of the to-be-monitored region by using at least one infrared thermal imaging camera.

[0040] It should be noted that the time synchronization processing in step A2 ensures the consistency of the visible light video data stream and the infrared thermal imaging video data stream in the time dimension by time synchronizing video frames from different sources, and eliminates the time offset caused by the difference in device response time.

[0041] Specifically, the pre-processing of the synchronized video data stream in step A3 can be:

[0042] Based on the visible light video data stream and the infrared thermal imaging video data stream, the model receives two synchronous image sequence inputs, including a visible light image and an infrared thermal image at the corresponding time.

[0043] The visible light image and the infrared thermal image are processed by the pre-processing convolutional layer respectively to obtain preliminary feature representations.

[0044] It should be noted that, due to the difference in data distribution between the infrared thermal image and the visible light image, a double convolution branch is used to extract respective features to retain the unique information in each modality, such as the temperature gradient in the infrared thermal image and the color texture in the visible light image.

[0045] In an optional embodiment, in step S100, multi-modal video data of the to-be-monitored region is acquired, and synchronized multi-modal video frames are obtained according to the multi-modal video data. In addition, a method of multi-sensor fusion acquisition can be used to integrate multiple visible light cameras and infrared thermal imaging cameras with different perspectives to form a monitoring network covering a wider range. At the same time, a GPS time synchronization mechanism is introduced to ensure the accurate time alignment of each video stream. An image registration algorithm is used to correct the perspective deviation and geometric distortion between different cameras, and exposure control and gain adjustment are used to optimize image quality, so that clear and stable video data can be obtained under different lighting conditions and weather environments, and high-quality multi-modal input data is provided for the subsequent fire identification model.

[0046] In another optional embodiment, the multi-modal video data of the to-be-monitored area is acquired in step S100, and the synchronized multi-modal video frames are obtained according to the multi-modal video data. The real-time preprocessing of the video data can also be performed by the edge computing device. The preliminary image enhancement, noise suppression and format standardization processing are completed at the data acquisition end, so as to reduce the data transmission burden. At the same time, a data quality evaluation mechanism is established to monitor the quality indicators such as definition, contrast and signal-to-noise ratio of the video stream in real time, automatically remove the frame data that does not meet the quality standard, and process the frame loss problem caused by network jitter or device failure by using the time sequence interpolation and motion compensation technology, so as to ensure the continuity and integrity of the multi-modal video data and provide a stable and reliable data source for the fire identification algorithm.

[0047] It should be noted that, by acquiring the multi-modal video data of the to-be-monitored area and performing time synchronization and preprocessing, the present application realizes the cooperative acquisition and unified processing of visible light and infrared thermal imaging data. The complementary visual information is acquired through the dual-camera, the time synchronization processing eliminates data offset, and the preprocessing retains the unique information of each modality. Compared with the fire identification method in the prior art which relies on single modal visual information, the present application solves the problem of low recognition accuracy of the traditional method in the actual environment through multi-modal data fusion. In particular, the infrared thermal imaging data can effectively detect temperature anomalies, and the visible light data can provide rich texture and shape information. The combination of the two modalities can maintain stable detection performance under different lighting conditions and weather environments, not only improving the accuracy and reliability of fire identification, but also providing high-quality multi-dimensional feature input for subsequent deep learning model training and optimization, effectively improving the overall performance of fire identification in power grid inspection.

[0048] In the embodiment of the present application, the baseline model preliminarily trained is determined based on the synchronized multi-modal video frames and the preset forest fire identification model in step S200, including the following steps B1-B4:

[0049] B1: constructing a training data set based on the synchronized multi-modal video frames.

[0050] B2: taking the forest fire identification model as an agent and taking the video frames in the training data set as the environment state.

[0051] B3: designing a reward mechanism and assigning a reward value according to the matching condition of the identification result of the forest fire identification model and the real label.

[0052] B4: iteratively optimizing the parameters of the forest fire identification model until the performance converges, to obtain the baseline model preliminarily trained.

[0053] Specifically, the training data set is constructed in step B1, and the specific operation can be:

[0054] Collect a number of training video data from different scenarios, including forest environment and power grid line environment under fire and non-fire samples.

[0055] Manually label the training video data with fire or non-fire labels.

[0056] Further, in step B2, the wildfire recognition model is used as an agent, and the video frames in the training data set are used as environmental states. The specific operation can be:

[0057] The process of inputting the training video data frame by frame into the wildfire recognition model is regarded as the interaction process between the agent and the environment.

[0058] A corresponding Q table is established for each training video data, which is used to record the action decision and reward value of the wildfire recognition model on the training video data.

[0059] Further, in step B3, the reward mechanism is designed, and the specific operation can be:

[0060] When the fire identification result output by the wildfire recognition model matches the true label, a positive reward is given.

[0061] When the fire identification result is wrong, a negative reward is given to encourage the wildfire recognition model to make decisions to improve the recognition accuracy.

[0062] Specifically, in step B4, the parameters of the wildfire recognition model are iteratively optimized, which can include the following steps B41-B45:

[0063] B41: Input each training video sequence into the current wildfire recognition model in turn. For each frame, according to the matching situation of the recognition result output by the wildfire recognition model and the true label, record the state, action and immediate reward value at this moment in the corresponding Q table.

[0064] B42: When a video sequence is completed, calculate the cumulative reward of the video sequence.

[0065] B43: Repeat steps B41 to B42 to traverse all training videos, and take the sum of the cumulative rewards of all Q tables as the performance indicator of the current strategy.

[0066] B44: Use reinforcement learning algorithm to adjust the parameters of the wildfire recognition model to maximize the total reward. The policy iteration process is repeated, and the wildfire recognition model is constantly updated with all training videos until the total reward of the wildfire recognition model on the training set converges and no longer improves significantly.

[0067] B45: Test the trained forest fire recognition model on an independent verification video set, and use the forest fire recognition model that meets the predetermined requirements of recognition accuracy and false positive rate as the baseline model of preliminary training convergence for standby.

[0068] In an optional embodiment, in step S200, the baseline model of preliminary training convergence is determined based on the synchronized multi-modal video frames and the preset forest fire recognition model. The training data set can also be expanded through data enhancement techniques, such as random rotation, scaling, flipping, and color transformation, to generate more diverse training samples and improve the generalization ability of the model. At the same time, a curriculum learning strategy is introduced to arrange the training samples in order from simple to complex, allowing the model to first learn obvious fire features and then gradually adapt to complex boundary conditions. A multi-task learning framework is also used to train related tasks such as fire detection, smoke recognition, and temperature estimation, and the overall performance is improved through knowledge sharing between tasks. Finally, a transfer learning method is used to utilize a pre-trained feature extractor on a large-scale natural image data set to accelerate model convergence and improve recognition accuracy.

[0069] In another optional embodiment, in step S200, the baseline model of preliminary training convergence is determined based on the synchronized multi-modal video frames and the preset forest fire recognition model. The training process can also be optimized through a learning rate adjustment mechanism, which adjusts the learning rate according to the training progress. A larger learning rate is used in the early stages of training to quickly converge, and a smaller learning rate is used in the later stages of training for fine tuning. At the same time, an early stopping mechanism is used to prevent overfitting by monitoring the performance indicators on the validation set and automatically stopping training when there is no improvement for several consecutive rounds of training. An ensemble learning method is also introduced to train multiple baseline models with different initialization parameters or network structures, and the final recognition accuracy is improved through voting or weighted fusion to ensure the stability and reliability of the baseline model.

[0070] It should be noted that, in the present application, a training data set is constructed based on synchronized multi-modal video frames, and a deep reinforcement learning method is used to train and optimize the forest fire recognition model as an agent. A reward mechanism is established to guide the model to learn the correct recognition strategy, and the model parameters are iteratively optimized until the performance converges. Compared with the model training method in the prior art, which relies solely on supervised learning, the present application solves the problem of decision sequence and environmental adaptation that traditional methods cannot handle. In particular, the training method of agent interaction with the environment enables the model to learn more robust feature representations. The state, action, and reward information recorded in the Q table helps the model understand the optimal decision strategy in different scenarios, not only improving the model's adaptability to power grid inspection environments, but also effectively reducing false positives and false negatives through positive and negative reward mechanisms, ensuring that the baseline model maintains stable recognition performance in different scenarios such as forest environments and power line environments, and laying a solid foundation for subsequent self-supervised learning optimization.

[0071] In the embodiments of the present application, the step S300 of obtaining the unannotated video data to obtain the optimized baseline model in combination with the baseline model comprises the following steps C1-C4:

[0072] C1: Obtain unannotated video data.

[0073] C2: Use the baseline model to infer the unannotated video data to generate predicted labels and confidence.

[0074] C3: Determine pseudo labels according to the predicted labels and confidence.

[0075] C4: Merge the samples with pseudo labels with the training data set, and optimize the baseline model to obtain the optimized baseline model.

[0076] Specifically, the step C1 of obtaining unannotated video data can specifically include collecting unmanned aerial vehicle inspection video data from actual deployment scenarios, wherein the unmanned aerial vehicle inspection video data is not manually annotated with fire or not, collecting large-scale video data from actual deployment environments, covering various weather, terrain, and equipment conditions, and dividing the video data into appropriate length segments for processing.

[0077] Specifically, the step C2 of using the baseline model to infer the unannotated video data can specifically operate as follows:

[0078] Use the baseline model trained to converge to process the unannotated video data one by one.

[0079] Generate predicted labels and confidence for each frame in the unannotated video data.

[0080] Specifically, the step C3 of determining pseudo labels according to the predicted labels and confidence comprises the following steps C31-C33:

[0081] C31: Set a preset confidence threshold.

[0082] C32: Frame screening based on the confidence and the preset confidence threshold to obtain screened samples.

[0083] C33: Use the predicted labels of the frames as pseudo labels to obtain samples with pseudo labels.

[0084] Illustratively, the step C32 of screening high-confidence samples can specifically operate as follows:

[0085] Select frames higher than the preset confidence threshold as positive samples.

[0086] Select frames lower than the preset confidence threshold as negative samples.

[0087] Frames in the intermediate uncertain region are temporarily not used.

[0088] It should be noted that, in order to ensure the quality of pseudo labels, only the predictions that the model is confident about are retained, for example, the threshold value can be set to 0.9 / 0.1 for the first self-training to screen out high-confidence positive and negative samples; for rare scenarios such as the model frequently outputting medium probability uncertainty, it is temporarily not used to avoid introducing noise labels.

[0089] Specifically, in step C4, the samples with pseudo labels are merged with the training data set, and the baseline model is optimized. The specific operation can be:

[0090] The samples with pseudo labels are merged with the original manually annotated data to form an expanded training set.

[0091] The parameters of the baseline model are fine-tuned in a supervised learning manner, the pseudo labels are fixed as true values to allow the model to fit, and the adaptability of the model to previously unseen scenarios is improved.

[0092] After updating the model, the confidence threshold is increased, new pseudo labels are generated for the remaining unused data, and the model is trained again. Through multiple iterations, most of the unlabeled data is absorbed and utilized by the baseline model.

[0093] In an optional embodiment, in step S300, the unlabeled video data is obtained, and the optimized baseline model is obtained by combining the baseline model. The most informative unlabeled samples can also be selected for pseudo label generation through an active learning strategy. An uncertainty sampling and diversity sampling combined method is used to preferentially select samples with high model prediction uncertainty but high representativeness. Consistency regularization technology is introduced to require the model to maintain consistent prediction results for the enhanced samples. A self-training and co-training combined strategy is used to utilize multiple models with different architectures to provide pseudo labels for each other, improve the reliability of the pseudo labels through collaborative learning between models, and finally establish a threshold adjustment mechanism to adjust the confidence threshold according to the improvement of the model performance.

[0094] In another optional embodiment, in step S300, the unlabeled video data is obtained, and the optimized baseline model is obtained by combining the baseline model. The pseudo label quality can also be optimized through temporal consistency constraints. The temporal relationship between consecutive frames in a video sequence is used to smooth the prediction results, reduce noise and inconsistency in the pseudo labels, and a confidence weighted loss function is used to assign different weights according to the confidence of the pseudo labels. High-confidence samples obtain greater training weights. An adversarial training mechanism is introduced to improve the robustness of the model, so that the model can better generalize to unseen scenarios. Finally, a pseudo label quality evaluation system is established to monitor the accuracy of the pseudo labels through methods such as manual sampling verification and cross-validation to ensure the effectiveness of the self-supervised learning process.

[0095] It should be explained that the present application obtains a large amount of unlabeled actual inspection video data, generates pseudo labels by reasoning using a baseline model that has been preliminarily trained to converge, and ensures the quality of the pseudo labels by using a high confidence threshold screening mechanism to merge the samples with pseudo labels and the original training data for model optimization. Compared with the prior art that relies only on limited labeled data for supervised learning, the present application solves the problem of the scarcity of labeled data in fire identification through a self-supervised learning mechanism. In particular, the confidence threshold screening can effectively avoid the negative impact of low-quality pseudo labels on the performance of the model, and the iterative optimization strategy enables the model to continuously learn from new data and improve itself, thereby not only expanding the scale and diversity of the training data but also improving the adaptability of the model to various scenes in the actual deployment environment, ensuring stable and reliable identification performance in different weather conditions, terrain environments and equipment conditions, and effectively improving the accuracy and practicality of fire identification in power grid inspection.

[0096] It should be explained that the optimized baseline model first extracts multi-scale image spatial features through a shallow convolution, and then fuses the feature maps of the visible light video data stream and the infrared thermal imaging video data stream in the middle, and then strengthens the feature response of the suspected fire source region through an attention module, and finally outputs the fire identification result through a fully connected layer.

[0097] In the embodiment of the present application, the fire identification based on the optimized baseline model and the synchronized multi-modal video frames in step S400 includes the following steps D1-D2:

[0098] D1: Extracting spatial features and temporal features of the synchronized multi-modal video frames through the optimized baseline model and performing feature fusion to generate a comprehensive feature representation.

[0099] D2: Obtaining a fire identification result according to the comprehensive feature representation.

[0100] Specifically, in step D1, the spatial features and temporal features of the synchronized multi-modal video frames are extracted through the optimized baseline model and feature fusion is performed to generate a comprehensive feature representation, including the following steps D11-D13:

[0101] D11: Extracting feature maps of the multi-modal video frames.

[0102] D12: Connecting and fusing the feature maps of the multi-modal video frames in the middle layer by channel to obtain a fused feature map.

[0103] D13: Weight allocation is performed on the fused feature map, and the feature map weighted by attention is combined with the time variation pattern of the continuous frames to obtain a comprehensive feature representation.

[0104] Specifically, the feature map of the multi-modal video frame in step D11 can be extracted by the following specific operation:

[0105] The feature maps of the infrared thermal imaging video data stream and the visible light video data stream are extracted by a double-path convolutional neural network, and are connected and fused in the middle layer, and are then jointly judged by an attention mechanism and a subsequent layer.

[0106] It should be noted that the double-path convolutional neural network in step D11 refers to the extraction of multi-scale image spatial features by shallow convolution, wherein the convolutional neural network is composed of a plurality of light-weight convolution units, each unit including a depth separable convolution and a subsequent batch normalization and activation function; the depth separable convolution decomposes the convolution calculation into a per-channel spatial convolution and a channel combination convolution, greatly reducing the parameter quantity and multiplication operation times, and is suitable for embedded implementation; the convolutional neural network uses multi-scale convolution kernels (such as 3x3 combined with 5x5 convolution) to extract features at different scales, so as to detect large-scale open flames and small-size flames.

[0107] Specifically, the feature map of the multi-modal video frame in step D12 can be connected and fused in the middle layer by the following specific operation:

[0108] The feature map of the infrared thermal imaging video data stream and the feature map of the visible light video data stream are fused and spliced in the channel dimension to obtain a fused feature map.

[0109] The fused feature map is input into a subsequent convolutional neural network for deep feature extraction to obtain a feature tensor.

[0110] It should be noted that under the action of continuous convolution layers, the fused feature map gradually refines high-level semantic information, such as the edge shape and motion pattern of the flame, the texture and drifting trajectory of the smoke, and the structure of the background forest or equipment, etc. These deep features will provide a basis for the subsequent attention mechanism and time module.

[0111] Specifically, the fused feature map in step D13 can be weighted and distributed, and the feature map weighted by attention and the time variation pattern of the continuous frame can be combined to obtain a comprehensive feature representation by the following specific operation:

[0112] The feature tensor extracted by the convolutional neural network is input into a spatial attention module and a channel attention module at the same time to obtain a weighted feature tensor.

[0113] The weighted feature tensor, i.e. the current frame feature weighted by attention and the hidden state of the previous frame, is input into a long short-term memory network layer, and the long short-term memory network layer uses multiple gate units to control the information flow, and calculates the time features such as the flicker frequency of the firelight and the drifting direction of the smoke through continuous iteration to obtain a comprehensive feature representation.

[0114] It should be noted that the spatial attention module automatically assigns importance coefficients to each position on the feature map by learning a weight mask of the same size as the feature map, which actually reflects which areas in the current frame may contain fire information, for example, the flame / smoke area has a large weight, and the forest sky background has a small weight; the channel attention module assigns weights to the channel dimension (different feature modes) of the feature tensor, highlighting the fire-related feature modes (such as high temperature, high-contrast edges, etc.), and weakening the modes unrelated to the fire; the two attention mechanisms work together to output a weighted feature tensor, in which the fire-related signals are enhanced and the noise interference is suppressed; through this mechanism, the model can adapt to the changes of the focus point in different scenarios: in a forest environment, attention may focus on smoke columns and fire points, and in a power grid environment, more attention is paid to sparks or smoke at the top of equipment, thereby realizing adaptive focusing on key features and improving discrimination accuracy.

[0115] It should be noted that the long short-term memory network layer uses multiple gate units to control the information flow, including a forgetting gate to determine which information to retain from the previous time, an input gate to write new information in combination with the current frame features, and an output gate to generate an updated hidden state vector; this is extremely important for distinguishing between real fires and transient disturbances - for example, real flames often burn and spread, while dust raised by a passing car is transient; with time memory, the model can make judgments based on the persistence of features over a period of time, and the hidden vector output by the long short-term memory network layer integrates spatial features of the current frame and recent temporal evolution information.

[0116] For example, in step D2, the fire identification result is obtained according to the integrated feature representation, and the specific operation can be:

[0117] The fire identification result is output by the fully connected layer, and is a binary quantity: 0 indicates that no fire is detected, and 1 indicates that a fire is detected.

[0118] In an optional implementation, in step S400, fire identification is performed based on the optimized baseline model and the synchronized multi-modal video frames, and the spatial feature extraction capability can also be enhanced by a multi-scale feature pyramid network, different resolution feature levels are constructed to capture multi-level information from fine-grained texture to coarse-grained semantics, deformable convolution is used to adapt to irregular shape changes of flames and smoke, residual connection and dense connection mechanisms are introduced to deepen the network depth, improve the feature expression capability, and through the cascade combination of feature channel attention and spatial attention, more fine feature selection and weight distribution are realized to ensure that the model can accurately locate and identify fire features of various scales and morphologies.

[0119] In another optional implementation, the fire identification in step S400 based on the optimized baseline model and the synchronized multi-modal video frame can also be directly processed by a spatio-temporal convolutional neural network to extract spatial and temporal features, a 3D convolution kernel is used for feature learning in the spatio-temporal dimension to capture the pattern of fire development, a self-attention mechanism is introduced to model long-range spatio-temporal dependencies and improve the understanding of the fire scene, a multi-task learning framework is used for fire detection, flame segmentation and smoke tracking at the same time, the mutual promotion between tasks is used to improve the overall recognition performance, and finally an ensemble learning method is used to fuse the prediction results of multiple models to improve the accuracy and stability of the final recognition through voting or weighted averaging.

[0120] It should be noted that the present application inputs the synchronized multi-modal video frame into the optimized baseline model, uses a two-way convolutional neural network to extract infrared and visible light features respectively, strengthens the key area feature response through the attention mechanism, captures the time sequence dynamic feature through the long short-term memory network, and finally outputs an accurate fire identification result. Compared with the recognition method in the prior art which simply relies on image features or single modal information, the present application solves the problem of low recognition accuracy and easy interference in the actual environment by combining multi-modal feature fusion and spatio-temporal features, especially through the attention mechanism to focus on the suspected fire source area and the long short-term memory network to effectively distinguish the persistence feature of the real fire and the instantaneous interference signal, which not only improves the accuracy and reliability of the fire identification, but also effectively reduces the false alarm and missed alarm phenomenon, ensures that the fire identification in the power grid inspection can provide timely and accurate early warning information in the face of various scenes, and provides reliable technical support for the safe operation of the power grid.

[0121] In summary, the present application realizes multi-modal information acquisition of fire by simultaneously collecting visible light and infrared video data and performing time synchronization processing, uses visible light to identify open fire and smoke shape during the day, and captures temperature anomalies with the help of infrared at night or in shielding conditions, effectively making up for the blind area of a single sensor; by taking the forest fire recognition model as an agent and designing a reward mechanism for reinforcement learning training, the model is given the ability to self-improve, avoiding the dependence of pure supervised learning on massive manual annotation and overcoming the shortcomings of slow convergence and unstable effect of traditional methods; by adopting a network structure combining convolutional neural network and long short-term memory network and introducing an attention mechanism, spatial features can be extracted and temporal change patterns can be captured, the importance of features in each region is adjusted according to the current input, the model focuses on key regions such as flames and smoke columns, and the probability of misjudging transient smoke clouds as fire and the missed detection rate of weak fire sources are reduced; by introducing a pseudo-label self-supervised training strategy and using a confidence threshold screening mechanism, the value of unlabeled data is fully tapped, the training data size is expanded, and the model's generalization ability to new environments is improved; not only the accuracy and reliability of fire recognition are improved, but also continuous and stable detection under cross-day and night and actual background conditions is realized, providing accurate and timely fire warning protection for power grid inspection and forest fire prevention.

[0122] Embodiment 3 is an embodiment of the present application, which provides a fire recognition system suitable for power grid inspection, comprising: a video frame acquisition module, configured to acquire multi-modal video data of a to-be-monitored region, and obtain synchronized multi-modal video frames according to the multi-modal video data; a baseline model training module, configured to determine a baseline model preliminarily trained and converged based on the synchronized multi-modal video frames and a preset forest fire recognition model; a baseline model optimization module, configured to obtain an optimized baseline model by acquiring unlabeled video data and combining the baseline model; and a fire recognition module, configured to perform fire recognition based on the optimized baseline model and the synchronized multi-modal video frames, and obtain a fire recognition result.

[0123] Embodiment 4 is an embodiment of the present application, which is different from the previous three embodiments in that: Figure 2As shown, the functions if implemented in the form of software functional units and sold or used as independent products, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part of the prior art or the part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0124] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered a list of executable instructions for implementing logic functions, and can be specifically embodied in any computer-readable medium for use by an instruction execution system, apparatus, or device, such as a computer-based system, a system including a processor, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions, or in conjunction with these instructions execution systems, apparatuses, or devices. For the purpose of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport programs for use by an instruction execution system, apparatus, or device, or in conjunction with these instruction execution systems, apparatuses, or devices.

[0125] More specific examples (a non-exhaustive list) of the computer-readable medium include the following: an electrical connection having one or more wires (electrical devices), a portable computer diskette (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium can even be paper or other suitable medium on which the program can be printed, as the program can be electronically obtained, for example, by optical scanning of the paper or other medium, followed by electronic editing, interpretation, or necessary processing, and then stored in a computer memory if necessary. Other suitable media can also be used.

[0126] It should be understood that portions of the present application can be implemented in hardware, software, firmware, or combinations thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, implementation can be with any or a combination of the following technologies, which are all well known in the art: a discrete logic circuit having logic gates for implementing logic functions upon an application of data signals, an application specific integrated circuit having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0127] It should be noted that the above examples are only used to illustrate the technical solutions of the present application but not limit the present application. Although the present application is described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present application can be modified or replaced equivalently without departing from the spirit and scope of the present application, which should be covered in the scope of claims of the present application.

Claims

1. A fire identification method suitable for power grid inspection, characterized in that: The method comprises the steps of: acquiring multi-modal video data of a region to be monitored, and obtaining synchronized multi-modal video frames according to the multi-modal video data; determining a baseline model preliminarily trained and converged based on the synchronized multi-modal video frames and a preset forest fire recognition model; obtaining an optimized baseline model by acquiring unlabeled video data and combining the baseline model; performing fire condition recognition based on the optimized baseline model and the synchronized multi-modal video frames to obtain a fire condition recognition result.

2. The fire identification method for power grid inspection of claim 1, wherein: The method comprises the steps of: acquiring a video data stream of a region to be monitored; performing time synchronization processing on the video data stream; performing preprocessing on the synchronized video data stream to obtain synchronized multi-modal video frames.

3. The fire identification method for power grid inspection of claim 2, wherein: The method comprises the steps of: constructing a training data set based on the synchronized multi-modal video frames; taking the forest fire recognition model as an intelligent agent and taking the video frames in the training data set as environment states; designing a reward mechanism and assigning a reward value according to the matching condition of the recognition result of the forest fire recognition model and a real label; iteratively optimizing the parameters of the forest fire recognition model until the performance converges to obtain a baseline model preliminarily trained and converged.

4. The fire identification method for power grid inspection of claim 3, wherein: The method comprises the steps of: acquiring unlabeled video data; generating a predicted label and a confidence by using the baseline model to infer the unlabeled video data; determining a pseudo label according to the predicted label and the confidence; merging the sample with the pseudo label and the training data set, and optimizing the baseline model to obtain an optimized baseline model.

5. The fire identification method for power grid inspection according to claim 4, characterized in that: The method comprises the steps of: extracting spatial features and temporal features of the synchronized multi-modal video frames by the optimized baseline model and performing feature fusion to generate a comprehensive feature representation; obtaining a fire condition recognition result according to the comprehensive feature representation.

6. The fire identification method for power grid inspection according to claim 5, characterized in that: The method comprises the steps of: extracting a feature map of the multi-modal video frames; connecting and fusing the feature map of the multi-modal video frames in the middle layer by channel to obtain a fused feature map; performing weight distribution on the fused feature map, combining the feature map weighted by attention with the time variation mode of continuous frames to obtain a comprehensive feature representation.

7. The fire identification method for power grid inspection according to claim 6, characterized in that: The method comprises the steps of: setting a preset confidence threshold; performing frame screening based on the confidence and the preset confidence threshold to obtain screened samples; taking the predicted label of the frame as a pseudo label to obtain a sample with a pseudo label.

8. A fire identification system for grid patrol, applying a fire identification method for grid patrol according to any one of claims 1-7, characterized in that, The method comprises the steps of: a video frame acquisition module for acquiring multi-modal video data of a region to be monitored and obtaining synchronized multi-modal video frames according to the multi-modal video data; The baseline model training module is configured to determine a preliminary training converged baseline model based on the synchronized multi-modal video frames and a preset forest fire identification model. The baseline model optimization module is configured to obtain an optimized baseline model by obtaining unlabeled video data and combining the baseline model. The fire situation identification module is configured to identify a fire situation based on the optimized baseline model and the synchronized multi-modal video frames to obtain a fire situation identification result. 9.A computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the computer device is configured to perform the method according to any one of claims 1-8 when the computer program is executed by the processor. The computer program is executed by the processor to implement the steps of the fire situation identification method for power grid inspection according to any one of claims 1 to 7.

10. A computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the fire situation identification method for power grid inspection according to any one of claims 1 to 7.

Images