A multi-model fusion transmission line smoke and fire recognition method and system

Abstract

The present invention relates to the technical field of fireworks recognition, specifically to a multi-model fusion transmission line fireworks recognition method and system, comprising the following steps: S1, establishing a stable communication connection with the monitoring equipment on the transmission line, real-time acquisition of the full amount of image data of the transmission line channel monitoring in the area, covering the surrounding scenes of the transmission line under different time periods, weather and terrain conditions; S2, pre-processing the collected image data; S3, using the pre-trained yolov5 fireworks recognition small model to recognize the pre-processed full amount of image data. The small model yolov5, with its fast computing power, performs preliminary screening of the full amount of transmission line image data, quickly locates the fireworks hidden danger area, greatly improves the recognition efficiency, and flexibly uses the fine-tuning strategy in transfer learning to fix some of the underlying network layer parameters, and only adjusts the parameters close to the output layer, which not only accelerates the convergence speed of the model on the new data set, but also improves the generalization ability of the model.

Description

Technical Field

[0001] The present invention relates to the technical field of fireworks recognition, and in particular to a multi-model fusion transmission line fireworks recognition method and system. Background Art

[0002] Transmission lines are often built in sparsely populated areas such as mountainous areas, forests, and wilderness. Once a wildfire occurs in this area, it is very easy to cause a large-scale power outage. Smoke is an early signal of a fire. In a mountain forest with a complex background, it is easier to observe than a flame. Smoke is an early signal of a fire. In a mountain forest with a complex background, it is easier to observe than a flame. Therefore, by detecting smoke to help detect wildfires in time, the expansion of accidents can be effectively avoided. In order to deal with such situations, image recognition is performed on the video image content collected by the monitoring equipment installed on the transmission line, and images with fireworks are distinguished and alarms are issued.

[0003] However, in areas with a humid climate and a large proportion of mountains, hills and water areas, there are more water reflections, clouds and mist in this type of terrain. In addition, the night lighting factors can easily be mistakenly identified as fireworks by traditional deep learning fireworks models, resulting in a large number of false fireworks alarms, which greatly increases the workload of team members. Traditional deep learning fireworks recognition models have a large number of false alarms, but the computing power requirements are relatively low. The multimodal large model only needs a small number of samples to achieve more accurate fireworks recognition, and the missed data is greatly reduced, but the computing power requirements are high, which is not conducive to the current fireworks recognition in the area. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a multi-model fusion transmission line smoke and fire identification method and system to solve the problems raised in the above background technology.

[0005] To achieve the above objectives, the present invention is implemented through the following technical solutions: a multi-model fusion transmission line smoke and fire recognition method, comprising the following steps: S1. Establish a stable communication connection with the monitoring equipment on the transmission line, collect the full amount of image data of the transmission line channel monitoring in the area in real time, and cover the surrounding scenes of the transmission line under different time periods, weather and terrain conditions; S2, preprocessing the collected image data; S3, using the previously trained yolov5 fireworks recognition model to recognize the pre-processed full image data; S4. Manually label the fireworks hazard data output by the yolov5 model, and integrate the meteorological data provided by the Meteorological Bureau into the labeled information, including but not limited to fireworks location information, real-time weather information, humidity information, and wind data, to form a complete labeled data set sample library; S5. Use the labeled data as training samples to carry out fine-tuning training of the cogvlm2 multimodal large model, and obtain a large model dedicated to local fireworks in the area; S6. Use the trained local fireworks-specific large model to identify the fireworks hazard data in the yolov5 fireworks model recognition results, and output the final model recognition result, which is the final output result of the platform.

[0006] Preferably, in step S2, preprocessing the collected image includes the following steps: S21, denoising the collected image, the calculation formula is: , where After bilateral filtering, the coordinate The pixel value at is the set of pixels within the filter window, is the normalization factor, is the Gaussian kernel in the spatial domain, is the Gaussian kernel of the threshold, For the image at coordinates The pixel value of S22, performing enhancement operation on the denoised image, the calculation formula is: , where The original image Gray value, is the first Gray value, The gray value in the original image is The number of pixels, is the total number of pixels in the image; S23, normalize the image and map the pixel values ​​of the image to the interval [0,1], and the calculation formula is: , where is the original image at coordinates The pixel value at and are the minimum and maximum pixel values ​​of the original image, respectively. is the normalized pixel value.

[0007] Preferably, the yolov5 fireworks recognition model is based on a convolutional neural network, and image features are extracted through a series of convolutional layers, pooling layers and fully connected layers. Its loss function adopts a cross entropy loss function, and the calculation formula is: , where is the sample size, is the number of categories, For sample Belongs to category The real label, Predict samples for the model Belongs to category probability.

[0008] Preferably, in step S6, the processed fireworks hazard data is input into the trained local fireworks dedicated large model for secondary recognition, and the attention mechanism is used to perform weighted fusion of different modal information, and the calculation formula is: , where is the query vector, is the key vector, is a value vector, To calculate and The obtained attention weights are is the dimension of the vector, To calculate and The resulting vector of values.

[0009] A multi-model fusion transmission line fireworks recognition system, applied to any one of the multi-model fusion transmission line fireworks recognition methods described above, comprising: The data acquisition and preprocessing module is responsible for communicating with the monitoring equipment on the transmission lines in the area, acquiring the full amount of image data of the transmission line channel monitoring in real time, and integrating the image denoising, enhancement and normalization algorithm modules to perform Gaussian filtering denoising, histogram equalization enhancement and normalization processing on the acquired image data in turn, providing high-quality data for subsequent recognition tasks; The small model recognition module has a built-in pre-trained yolov5 fireworks recognition small model. It receives pre-processed image data, uses convolutional neural networks to extract features and classify and recognize them, quickly screens out data suspected of fireworks hazards, and outputs fireworks hazard data with preliminary recognition result coordinates; The data labeling and management module is used to manually label the fireworks hazard data output by the small model, evaluate the consistency and accuracy indicators of the labeled data, and store, classify, retrieve and version the labeled data; The large model training and optimization module is based on the cogvlm2 multimodal large model. It uses the labeled data for fine-tuning training, adopts the stochastic gradient descent algorithm to update the model parameters, trains to obtain a large model dedicated to local fireworks, regularly collects new sample data, and retrains and optimizes the large model. The secondary recognition and result output module receives the fireworks hazard data output by the small model, processes its coordinates, expands the recognition area, and then inputs it into the local fireworks-specific large model for secondary recognition. Based on the large model recognition results and weather information, the final fireworks recognition results are output according to preset rules; The communication and interface module is used to connect to the data interface of the Meteorological Bureau, obtain meteorological data in real time, and send the final fireworks recognition results to the transmission line operation and maintenance management platform and the emergency command center to achieve data sharing and collaborative work.

[0010] Preferably, the small model recognition module and the large model training and optimization module both adopt a distributed computing architecture to improve model processing speed and training efficiency.

[0011] Preferably, the data labeling and management module has data backup and recovery functions to prevent loss of labeling data due to hardware failure or data loss.

[0012] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned multi-model fusion transmission line fireworks recognition methods.

[0013] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned multi-model fusion transmission line fireworks recognition methods are implemented.

[0014] The present invention provides a multi-model fusion transmission line smoke and fire identification method and system. It has the following beneficial effects: The present invention uses the small model yolov5 with its fast computing power to perform preliminary screening of the full amount of transmission line image data, quickly locate the fire and smoke hazard areas, and greatly improve the recognition efficiency. The large model dedicated to local fireworks, after fusing multimodal information and performing weighted fusion with the help of the attention mechanism, significantly improves the recognition accuracy of fireworks in complex scenes.

[0015] The present invention regularly collects actual fireworks events and misjudgment and missed judgment data, continuously expands and updates the sample set, so that the model can continuously learn new features and patterns, ensuring that the model can maintain good performance in different scenarios. At the same time, when the model is updated, the fine-tuning strategy in transfer learning is flexibly used to fix some underlying network layer parameters and only adjust the parameters close to the output layer. This not only speeds up the convergence speed of the model on the new data set, but also improves the generalization ability of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 It is a schematic diagram of the fireworks identification process of the present invention; Figure 2 The figure is a schematic diagram of the process of generating a large model dedicated to local fireworks according to the present invention. DETAILED DESCRIPTION

[0017] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0018] like Figure 1-Figure 2 As shown, an embodiment of the present invention provides a multi-model fusion transmission line smoke and fire recognition method, comprising the following steps: S1. Establish a stable communication connection with the monitoring equipment on the transmission line, collect the full amount of image data of the transmission line channel monitoring in the area in real time, and cover the surrounding scenes of the transmission line under different time periods, weather and terrain conditions; S2, preprocessing the collected image data; S3, using the previously trained yolov5 fireworks recognition model to recognize the pre-processed full image data; S4. Manually label the fireworks hazard data output by the yolov5 model, and only label the fireworks data that are likely to cause hidden dangers in the transmission line. Real and invalid fireworks alarms, such as chimneys, open areas far from the wires, factories, and houses, do not need to be labeled. In the labeled information, the meteorological data provided by the Meteorological Bureau is integrated, including but not limited to fireworks location information, real-time weather information, humidity information, and wind data, to form a complete labeled data set sample library; S5. Use the labeled data as training samples to carry out fine-tuning training of the cogvlm2 multimodal large model, and obtain a large model dedicated to local fireworks in the area; S6. Use the trained local fireworks-specific large model to identify the fireworks hazard data in the yolov5 fireworks model recognition results. The recognition method is: first, process the coordinates of the fireworks hazard location, process the coordinates of the model recognition results, expand the recognition area, increase the width and height of the recognition rectangular area by 50 pixels each, and expand it to the image boundary point if it is less than 50, to improve the recognition and understanding capabilities of the large model, and finally input it into the local fireworks-specific large model in the area for recognition. According to the large model recognition results and combined with the weather information in the output results, the specific parameters are shown in Table 1, and the final model recognition result is output, which is the final output result of the platform.

[0019] Table 1 Result output reference table ; Compared with the traditional deep learning fireworks recognition model, the fusion of large and small models reduces the number of warning images by an average of 267,000 per day.

[0020] In this embodiment, in step S2, preprocessing the collected image includes the following steps: S21, denoising the collected image, the calculation formula is: , where After bilateral filtering, the coordinate The pixel value at is the set of pixels within the filter window, is the normalization factor, is the Gaussian kernel in the spatial domain, is the Gaussian kernel of the threshold, For the image at coordinates The pixel value of S22, performing enhancement operation on the denoised image, the calculation formula is: , where The original image Gray value, is the first Gray value, The gray value in the original image is The number of pixels, is the total number of pixels in the image; S23, normalize the image and map the pixel values ​​of the image to the interval [0,1], and the calculation formula is: , where is the original image at coordinates The pixel value at and are the minimum and maximum pixel values ​​of the original image, respectively. is the normalized pixel value.

[0021] In this embodiment, the yolov5 fireworks recognition model is based on a convolutional neural network, and image features are extracted through a series of convolutional layers, pooling layers, and fully connected layers. Its loss function adopts a cross entropy loss function, and the calculation formula is: , where is the sample size, is the number of categories, For sample Belongs to category The real label, Predict samples for the model Belongs to category probability.

[0022] In this embodiment, in step S6, the processed fireworks hazard data is input into the trained local fireworks dedicated large model for secondary recognition, and the attention mechanism is used to perform weighted fusion of different modal information. The calculation formula is: , where is the query vector, is the key vector, is a value vector, To calculate and The obtained attention weights are is the dimension of the vector, To calculate and The resulting vector of values.

[0023] Specifically, by comprehensively optimizing the collected image data, the image quality was effectively improved, laying a solid foundation for subsequent accurate identification. The small model yolov5, with its fast computing power, conducted preliminary screening of the full amount of transmission line image data and quickly located areas with hidden dangers of fireworks and fire, greatly improving the recognition efficiency. The large model dedicated to local fireworks, after fusing multimodal information and performing weighted fusion with the help of the attention mechanism, significantly improved the recognition accuracy of fireworks in complex scenes.

[0024] A multi-model fusion transmission line fireworks recognition system, applied to any one of the multi-model fusion transmission line fireworks recognition methods described above, comprising: The data acquisition and preprocessing module is responsible for communicating with the monitoring equipment on the transmission lines in the area, acquiring the full amount of image data of the transmission line channel monitoring in real time, and integrating the image denoising, enhancement and normalization algorithm modules to perform Gaussian filtering denoising, histogram equalization enhancement and normalization processing on the acquired image data in turn, providing high-quality data for subsequent recognition tasks; The small model recognition module has a built-in pre-trained yolov5 fireworks recognition small model. It receives pre-processed image data, uses convolutional neural networks to extract features and classify and recognize them, quickly screens out data suspected of fireworks hazards, and outputs fireworks hazard data with preliminary recognition result coordinates; The data labeling and management module is used to manually label the fireworks hazard data output by the small model, evaluate the consistency and accuracy indicators of the labeled data, and store, classify, retrieve and version the labeled data; The large model training and optimization module is based on the cogvlm2 multimodal large model. It uses the labeled data for fine-tuning training, adopts the stochastic gradient descent algorithm to update the model parameters, trains to obtain a large model dedicated to local fireworks, regularly collects new sample data, and retrains and optimizes the large model. The secondary recognition and result output module receives the fireworks hazard data output by the small model, processes its coordinates, expands the recognition area, and then inputs it into the local fireworks-specific large model for secondary recognition. Based on the large model recognition results and weather information, the final fireworks recognition results are output according to preset rules; The communication and interface module is used to connect to the data interface of the Meteorological Bureau, obtain meteorological data in real time, and send the final fireworks recognition results to the transmission line operation and maintenance management platform and the emergency command center to achieve data sharing and collaborative work.

[0025] In this embodiment, the small model recognition module and the large model training and optimization module both adopt a distributed computing architecture to improve model processing speed and training efficiency.

[0026] In this embodiment, the data labeling and management module has data backup and recovery functions to prevent the loss of labeling data due to hardware failure or data loss.

[0027] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of any one of the above-mentioned multi-model fusion transmission line fireworks recognition methods.

[0028] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any one of the above-mentioned multi-model fusion transmission line fireworks recognition methods are implemented.

[0029] Specifically, by regularly collecting actual fireworks events and data on misjudgments and missed judgments, and continuously expanding and updating the sample set, the model can continuously learn new features and patterns, ensuring that the model can maintain good performance in different scenarios. At the same time, when updating the model, the fine-tuning strategy in transfer learning is flexibly used to fix some of the underlying network layer parameters and only adjust the parameters close to the output layer. This not only speeds up the convergence of the model on the new data set, but also improves the generalization ability of the model.

[0030] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A multi-model fusion transmission line fireworks recognition method, characterized in that: The following steps are involved: S1. Establish a stable communication connection with the monitoring equipment on the transmission line, collect the full amount of image data of the transmission line channel monitoring in the area in real time, and cover the surrounding scenes of the transmission line under different time periods, weather and terrain conditions; S2, preprocessing the collected image data; S3, using the previously trained yolov5 fireworks recognition model to recognize the pre-processed full image data; S4. Manually label the fireworks hazard data output by the yolov5 model, and integrate the meteorological data provided by the Meteorological Bureau into the labeled information, including but not limited to fireworks location information, real-time weather information, humidity information, and wind data, to form a complete labeled data set sample library; S5. Use the labeled data as training samples to carry out fine-tuning training of the cogvlm2 multimodal large model, and obtain a large model dedicated to local fireworks in the area; S6. Use the trained local fireworks-specific large model to identify the fireworks hidden danger data in the yolov5 fireworks model recognition result, and output the final model recognition result, which is the final output result of the platform; In step S2, preprocessing the collected image includes the following steps: S21, denoising the collected image, the calculation formula is: , where After bilateral filtering, the coordinate The pixel value at is the set of pixels within the filter window, is the normalization factor, is the Gaussian kernel in the spatial domain, is the Gaussian kernel of the threshold, For the image at coordinates The pixel value of S22, performing enhancement operation on the denoised image, the calculation formula is: , where The original image Gray value, is the first Gray value, The gray value in the original image is The number of pixels, is the total number of pixels in the image; S23, normalize the image and map the pixel values ​​of the image to the interval [0,1], and the calculation formula is: , where is the original image at coordinates The pixel value at and are the minimum and maximum pixel values ​​of the original image, respectively. is the normalized pixel value; In step S6, the processed fireworks hazard data is input into the trained local fireworks dedicated large model for secondary recognition, and the attention mechanism is used to perform weighted fusion of different modal information. The calculation formula is: , where is the query vector, is the key vector, is a value vector, To calculate and The obtained attention weights are is the dimension of the vector, To calculate and The resulting vector of values.

2. A multi-model fusion transmission line smoke and fire identification method according to claim 1, characterized in that: The yolov5 fireworks recognition model is based on a convolutional neural network. It extracts image features through a series of convolutional layers, pooling layers, and fully connected layers. Its loss function uses a cross entropy loss function, and the calculation formula is: , where is the sample size, is the number of categories, For sample Belongs to category The real label, Predict samples for the model Belongs to category probability.

3. A multi-model fusion transmission line fireworks recognition system, applied to a multi-model fusion transmission line fireworks recognition method according to any one of claims 1-2, characterized in that: include: The data acquisition and preprocessing module is responsible for communicating with the monitoring equipment on the transmission lines in the area, acquiring the full amount of image data of the transmission line channel monitoring in real time, and integrating the image denoising, enhancement and normalization algorithm modules to perform Gaussian filtering denoising, histogram equalization enhancement and normalization processing on the acquired image data in turn, providing high-quality data for subsequent recognition tasks; The small model recognition module has a built-in pre-trained yolov5 fireworks recognition small model. It receives pre-processed image data, uses convolutional neural networks to extract features and classify and recognize them, quickly screens out data suspected of fireworks hazards, and outputs fireworks hazard data with preliminary recognition result coordinates; The data labeling and management module is used to manually label the fireworks hazard data output by the small model, evaluate the consistency and accuracy indicators of the labeled data, and store, classify, retrieve and version the labeled data; The large model training and optimization module is based on the cogvlm2 multimodal large model. It uses the labeled data for fine-tuning training, adopts the stochastic gradient descent algorithm to update the model parameters, trains to obtain a large model dedicated to local fireworks, regularly collects new sample data, and retrains and optimizes the large model. The secondary recognition and result output module receives the fireworks hazard data output by the small model, processes its coordinates, expands the recognition area, and then inputs it into the local fireworks-specific large model for secondary recognition. Based on the large model recognition results and weather information, the final fireworks recognition results are output according to preset rules; The communication and interface module is used to connect to the data interface of the Meteorological Bureau, obtain meteorological data in real time, and send the final fireworks recognition results to the transmission line operation and maintenance management platform and the emergency command center to achieve data sharing and collaborative work.

4. The multi-model fusion transmission line fire and smoke recognition system according to claim 3 is characterized in that: The small model recognition module and the large model training and optimization module both adopt a distributed computing architecture to improve model processing speed and training efficiency.

5. The multi-model fusion transmission line fire and smoke recognition system according to claim 3 is characterized in that: The data labeling and management module has data backup and recovery functions to prevent loss of labeling data due to hardware failure or data loss.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-model fusion transmission line fireworks recognition method described in any one of claims 1-2 are implemented.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the multi-model fusion transmission line fireworks recognition method described in any one of claims 1-2 are implemented.

Images