Leakage gas detection method, device and electronic equipment based on multi-data mixing

Abstract

The embodiments of the present application provide a method, device and electronic device for detecting leaked gas based on multi-data mixing, which relates to the field of gas detection technology. The method includes: obtaining multiple compressed images obtained by compressing multiple first infrared original images respectively, predicting the location of potential leaked gas based on the multiple compressed images as candidate locations; extracting sub-images at the candidate locations from multiple second infrared original images respectively as infrared sub-images, predicting the first classification result of the candidate location based on all infrared sub-images, and the first classification result is used to indicate that there is leaked gas at the candidate location, or there is no leaked gas at the candidate location. Through the embodiments of the present application, it is possible to quickly and accurately detect leaked gas.

Description

Technical Field

[0001] The present application relates to the field of gas detection technology, and in particular to a method, device and electronic equipment for detecting leaked gas based on multi-data mixing. Background Art

[0002] In industrial production, transportation and daily life, gas leakage may cause serious safety accidents or environmental pollution, posing a threat to people's lives and causing property losses. Through gas leak detection, potential gas leakage sources can be discovered and dealt with in a timely manner, thereby effectively preventing these safety accidents from happening.

[0003] In the related art, gas leakage detection methods mainly include two categories: the first type of gas leakage detection method is based on traditional image processing and machine learning, and its process is: first, use an infrared camera to shoot the current scene to obtain video imaging data, then enhance the video image, reduce noise, enhance the gas cloud characteristics, and then perform differential calculation based on adjacent video frames to obtain the moving target area, and finally classify the moving target area, determine the classification result as a gas cloud and alarm, but this type of method has poor detection effect in complex environments such as complex scenes, low temperature differences, and weak ambient light, resulting in low detection accuracy; the second type of gas leakage detection solution is to introduce a deep neural network with strong processing capabilities to simultaneously process the infrared image and visible light image of the current scene, and its process is: first, send the infrared image to the leakage gas target detection model to obtain the preliminary suspected location of the leakage gas, and then send the visible light image to the scene understanding model, and according to the output scene area segmentation map, the above leakage The initial suspicious gas positions are screened and the suspicious gas positions in the specific area are removed. For example, removing the suspicious gas positions in the sky can reduce the false alarms caused by clouds, so as to obtain the highly suspicious positions of the leaking gas; then the infrared channel time-series multi-frame images are sent to the gas sequence classification model, and the highly suspicious positions of the leaking gas are further screened to obtain the final leaking gas alarm positions. This type of method directly uses the original data collected by the camera or directly processes the image data obtained after the original data is compressed. If the original data collected by the camera is used, the model performance can be optimized, the processing capability for complex scenes can be improved, and the alarm positions of the leaking gas in the scenes can be accurately obtained. However, the use of raw data has the problems of large storage capacity and slow transmission speed, which will increase the product cost while extending the iteration cycle of the model, thereby reducing the detection efficiency; if the image data obtained after the original data is compressed is directly processed, the product transmission cost can be reduced, but it will lead to the problem of low detection accuracy.

[0004] It can be seen that when the relevant technology detects leaked gas, there is a problem of low accuracy or low detection efficiency. How to detect leaked gas quickly and accurately has become a technical problem that needs to be solved urgently in the relevant field. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device and electronic device for detecting leaked gas based on multi-data mixing, so as to realize rapid and accurate detection of leaked gas. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present application provides a method for detecting gas leakage based on multi-data mixing, the method comprising:

[0007] Acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene;

[0008] Predicting a location of potential gas leakage according to the plurality of compressed images as a candidate location;

[0009] Extracting sub-images at the candidate positions from a plurality of second infrared original images respectively as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images;

[0010] A first classification result of the candidate position is predicted based on all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or that there is no gas leak at the candidate position.

[0011] In a possible implementation, the predicting the first classification result of the candidate position according to all the infrared sub-images includes:

[0012] Extracting an image feature of at least one of the infrared sub-images as an infrared spatial feature; and extracting a sequence feature of an image sequence formed by arranging all the infrared sub-images in chronological order of shooting time as an infrared sequence feature;

[0013] A first classification result of the candidate position is predicted based on the infrared spatial features and the infrared sequence features.

[0014] In a possible implementation manner, extracting the image feature of at least one of the infrared sub-images as the infrared spatial feature includes:

[0015] Predicting gas scores of all the infrared sub-images respectively, wherein the gas scores are positively correlated with the number of features of the leaking gas contained in the images;

[0016] Determine, from all the infrared sub-images, an image whose gas score meets a preset screening condition as a target infrared sub-image;

[0017] The image features of the target infrared sub-image are extracted as infrared spatial features.

[0018] In a possible implementation, the method further includes:

[0019] Extracting sub-images at the candidate positions from a plurality of visible light original images respectively as visible light sub-images, wherein the visible light original images are uncompressed visible light images obtained by photographing the target scene, and the shooting time span of the plurality of second infrared original images is the same as the shooting time span of the plurality of visible light original images;

[0020] Predicting a second classification result of the candidate position according to all the visible light sub-images, the second classification result being used to indicate that a moving object exists at the candidate position, or that a moving object does not exist at the candidate position;

[0021] If the second classification result indicates that there is no moving object, and the first classification result indicates that there is leaked gas, it is determined that there is leaked gas at the candidate position.

[0022] In a possible implementation, the predicting the second classification result of the candidate position according to all the visible light sub-images includes:

[0023] Extracting image features of the target visible light sub-image as visible light spatial features; and extracting sequence features of an image sequence formed by sorting all the visible light sub-images in chronological order of shooting time as visible light sequence features;

[0024] Predicting a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature;

[0025] The target visible light sub-image is a visible light sub-image that is farthest from the beginning and the end in the image sequence.

[0026] In a possible implementation, the method further includes:

[0027] Acquire an uncompressed infrared image obtained by photographing the target scene within a first time period from the time to be detected as a first infrared original image;

[0028] An uncompressed infrared image obtained by photographing the target scene within a second time period from the time to be detected is obtained as a second infrared original image, wherein the second time period is greater than the first time period.

[0029] In a second aspect, an embodiment of the present application provides a gas leakage detection device based on multi-data mixing, the device comprising:

[0030] A first acquisition module, used to acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene;

[0031] A position prediction module, used for predicting the position of potential gas leakage according to the plurality of compressed images as a candidate position;

[0032] A first extraction module is used to extract sub-images at the candidate positions from a plurality of second infrared original images as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a true subset of the shooting time span of the plurality of second infrared original images;

[0033] The first classification module is used to predict the first classification result of the candidate position according to all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or there is no gas leak at the candidate position.

[0034] In a possible implementation, the first classification module includes:

[0035] The first classification submodule is used to extract the image features of at least one of the infrared sub-images as infrared spatial features; and to extract the sequence features of the image sequence formed by sorting all the infrared sub-images in chronological order of shooting time as infrared sequence features;

[0036] The second classification submodule is used to predict a first classification result of the candidate position according to the infrared spatial feature and the infrared sequence feature.

[0037] In a possible implementation, the first classification submodule includes:

[0038] A first classification unit, used for respectively predicting gas scores of all the infrared sub-images, wherein the gas scores are positively correlated with the number of features of the leaked gas contained in the image;

[0039] A second classification unit is used to determine, from all the infrared sub-images, an image whose gas score meets a preset screening condition as a target infrared sub-image;

[0040] The third classification unit is used to extract image features of the target infrared sub-image as infrared spatial features.

[0041] In a possible implementation, the device further includes:

[0042] a second extraction module, configured to extract sub-images at the candidate positions from a plurality of visible light original images, respectively, as visible light sub-images, wherein the visible light original images are uncompressed visible light images obtained by photographing the target scene, and the photographing time span of the plurality of second infrared original images is the same as the photographing time span of the plurality of visible light original images;

[0043] A second classification module, used for predicting a second classification result of the candidate position according to all the visible light sub-images, wherein the second classification result is used for indicating that a moving object exists at the candidate position, or that a moving object does not exist at the candidate position;

[0044] The result determination module is configured to determine that there is a gas leak at the candidate position if the second classification result indicates that there is no moving object and the first classification result indicates that there is a gas leak.

[0045] In a possible implementation manner, the second classification module includes:

[0046] a fourth classification unit, configured to extract image features of the target visible light sub-image as visible light spatial features; and extract sequence features of an image sequence formed by sorting all the visible light sub-images in chronological order of shooting time as visible light sequence features;

[0047] a fifth classification unit, configured to predict a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature;

[0048] The target visible light sub-image is a visible light sub-image that is farthest from the beginning and the end in the image sequence.

[0049] In a possible implementation, the device further includes:

[0050] A second acquisition module is used to acquire an uncompressed infrared image obtained by photographing the target scene within a first time period from the time to be detected as a first infrared original image;

[0051] The third acquisition module is used to acquire an uncompressed infrared image obtained by shooting the target scene within a second time period from the time to be detected as a second infrared original image, wherein the second time period is greater than the first time period.

[0052] In a third aspect, an embodiment of the present application provides an electronic device, including:

[0053] Memory, used to store computer programs;

[0054] The processor is used to implement any of the above-mentioned gas leakage detection methods based on multi-data mixing when executing the program stored in the memory.

[0055] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, characterized in that a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the above-mentioned gas leakage detection methods based on multi-data mixing is implemented.

[0056] In a fifth aspect, an embodiment of the present application provides an embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned gas leakage detection methods based on multi-data mixing.

[0057] Beneficial effects of the embodiments of the present application:

[0058] The leaking gas detection method based on multi-data mixing provided in the embodiment of the present application first predicts the candidate positions of the potential leaking gas based on the compressed infrared original image of the target scene, then extracts the infrared sub-images at each candidate position from each infrared original image, and finally predicts whether there is a leaking gas at the candidate position based on each infrared sub-image. Through the method of the embodiment of the present application, since the data volume of the compressed infrared image is small and the processing speed is relatively fast, each candidate position can be quickly obtained by determining the candidate position based on the compressed infrared image; and since the information included in the infrared original image is more comprehensive than the information included in the compressed infrared original image, after the candidate positions of the potential leaking gas are obtained based on the compressed infrared original image, by extracting the sub-images of each candidate position in the infrared original image, it is possible to accurately determine whether there is a gas leak at each candidate position based on each sub-image, thereby realizing rapid and accurate detection of leaking gas.

[0059] In addition, predicting the candidate locations of potential gas leakage based on the compressed infrared original image of the target scene can be achieved based on a pre-trained model. During the process of training the model, the input of the model is also the compressed infrared original image. Since the cost of compressed images is relatively low, if the current scene is relatively complex, the model can be iterated in a relatively short time to improve the scene adaptability of the model; and, predicting the first classification result of the candidate location based on all infrared sub-images can also be achieved based on a pre-trained model. During the process of training the model, the input of the model is also the infrared sub-image. Since each infrared sub-image is a sub-image at the candidate location in the infrared original image, and the candidate location is usually small, the processing task of the model is relatively simple, so that the model does not need to be frequently iterated and updated, thereby achieving fast and accurate gas leakage detection effects while not increasing or increasing the control cost very little.

[0060] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0062] Figure 1 A first flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0063] Figure 2 A processing flow chart of a leak gas sequence detection model provided in an embodiment of the present application;

[0064] Figure 3 A second flow chart of the gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0065] Figure 4 A third flow chart of the gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0066] Figure 5 A processing flow chart of the leakage gas sequence classification model provided in an embodiment of the present application;

[0067] Figure 6 A fourth flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0068] Figure 7 A fifth flow chart of the gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0069] Figure 8 A processing flow chart of a visible light target analysis model provided in an embodiment of the present application;

[0070] Fig. 9 An example flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application;

[0071] Fig.10 A schematic diagram of the structure of a gas leakage detection device based on multi-data mixing provided in an embodiment of the present application;

[0072] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0073] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0074] First, the professional terms in the embodiments of the present application are explained:

[0075] VOCs (Volatile Organic Compounds): Volatile organic compounds, usually divided into several categories such as non-methane hydrocarbons, oxygen-containing organic compounds, halogenated hydrocarbons, nitrogen-containing organic compounds, and sulfur-containing organic compounds;

[0076] Infrared camera: Captures and images light waves in a specific band in the environment to generate a single-channel image.

[0077] Dual-light camera: The camera has two different movements, which can respectively image the visible light band and the infrared light band.

[0078] RAW data: Raw data, in the embodiment of the present application, refers to the raw data obtained by the infrared camera movement capturing ambient light, usually a single pixel value is represented by 16 bits wide.

[0079] Long-term exposure of VOCs gas to the air will cause great harm to the human body and cause environmental pollution. Therefore, natural gas, as a clean fossil energy, whose main component is gaseous low-molecular hydrocarbons, plays an important role in the daily life of residents. However, due to the strong flammability of natural gas, if it leaks during use or transportation, it is very easy to cause fire or even explosion, posing a huge threat to the lives and property of residents. Therefore, in order to prevent fires and explosions and protect the environment, gas leakage detection is usually carried out on the production and transportation of VOCs gas and natural gas industries.

[0080] Since both VOCs gas and natural gas are invisible in the visible light band, but have obvious imaging effects in the mid-infrared band. Based on this, there are many gas leakage detection algorithm solutions in the relevant technology, and the main processing flow is as follows: first, use an infrared camera to detect the leakage scene and obtain infrared image imaging data; then use the relevant data preprocessing method to process the above data to reduce noise and enhance gas characteristics; then based on the data with enhanced gas characteristics, use the corresponding machine learning or deep learning method to process to obtain the leakage gas location warning; finally, according to the imaging results of the visible light camera, the above leakage gas warning is filtered to obtain the final leakage gas location alarm.

[0081] The above schemes can achieve the purpose of intelligent monitoring of gas leakage events in the scene to a certain extent, but there are also some problems, as follows: 1. Most schemes use traditional manual feature methods with low complexity, and the overall monitoring effect is relatively limited, which is difficult to apply in complex scenes; 2. A few schemes use cutting-edge deep learning methods to achieve gas leakage monitoring. This type of scheme uses multi-scene data to train deep learning models, thereby improving its processing capabilities for complex scenes. The training of deep learning models usually requires a large amount of data covering each processing scene. Some schemes directly collect 16-bit wide camera RAW data of each scene. Although they can achieve the optimization of model performance, the collection of RAW data has the problems of large storage capacity and slow transmission speed, which will greatly increase product costs and extend the iteration cycle of the model, making it difficult to respond to short-term needs of the project site in a timely manner; some schemes will compress RAW data on the camera side and convert it into 8-bit wide video data, and then transmit it to achieve large-scale data collection and model training. However, when compressing 16-bit wide RAW data into 8-bit wide video data, it will cause a large information loss, which will blur the characteristics of rarefied gas and cause the model's alarm capability to decline for this type of gas.

[0082] In addition, infrared cameras are unstable when imaging. Even if a certain part of the image has not actually changed, the pixel values ​​corresponding to adjacent video frames may be quite different. In the actual processing scene, there will be a large number of non-gas cloud moving objects, such as cars, people, pipe shaking, leaves shaking, clouds floating in the sky, and camera shaking caused by wind, which may cause the differential algorithm to generate a large number of suspicious targets, which is not conducive to the accurate detection of gas cloud targets.

[0083] In order to improve the accuracy and efficiency of leaked gas detection, the first aspect of the embodiment of the present application first provides a leaked gas detection method based on multi-data mixing, which can be applied to electronic devices. In specific applications, the electronic device can be a server or a terminal device, both of which are within the protection scope of the present application.

[0084] See also Figure 1 , Figure 1 A first flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application, the method comprises the following steps:

[0085] Step S1, obtaining a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images;

[0086] The first infrared original image is an uncompressed infrared image obtained by photographing the target scene;

[0087] Step S2, predicting the potential gas leakage location based on the multiple compressed images as a candidate location;

[0088] Step S3, extracting sub-images at the candidate positions from the plurality of second infrared original images respectively as infrared sub-images;

[0089] The second infrared original image is an uncompressed infrared image obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images;

[0090] Step S4, predicting and obtaining a first classification result of the candidate position based on all infrared sub-images, wherein the first classification result is used to indicate that there is a gas leak at the candidate position, or that there is no gas leak at the candidate position.

[0091] Through the method of the embodiment of the present application, since the data volume of the compressed infrared image is small and the processing speed is relatively fast, each candidate position can be quickly obtained by determining the candidate position based on the compressed infrared image; and since the information included in the infrared original image is more comprehensive than the information included in the compressed infrared original image, after the candidate positions of the potential gas leakage are obtained based on the compressed infrared original image, by extracting the sub-image of each candidate position in the infrared original image, it can be accurately determined whether there is a gas leakage at each candidate position based on each sub-image, thereby achieving fast and accurate detection of leaking gas.

[0092] In addition, predicting the candidate position of potential gas leakage according to the compressed infrared original image of the target scene can be realized based on a pre-trained model. In the process of training the model, the input of the model is also the compressed infrared original image. In this way, since the cost of compressed images is relatively low, if the current scene is relatively complex, the model can be iterated in a relatively short time to improve the scene adaptability of the model; and predicting the first classification result of the candidate position according to all infrared sub-images can also be realized based on a pre-trained model. In the process of training the model, the input of the model is also the infrared sub-image. Since each infrared sub-image is a sub-image at the candidate position in the infrared original image, and the candidate position is usually small, the processing task of the model is relatively simple, so the model does not need to be frequently iterated and updated, thereby achieving a fast and accurate gas leakage detection effect while the control cost does not increase or increases very little. The specific description of the model for predicting the candidate position of potential gas leakage and the model for predicting the first classification result of the candidate position is shown below and will not be repeated here.

[0093] The above steps S1 to S4 are described below:

[0094] In step S1, the first infrared original image may be an uncompressed original infrared image obtained by photographing the target scene with an infrared camera, or may be an uncompressed original infrared image obtained by photographing the target scene with a bi-optical camera.

[0095] The bit width of the first infrared raw image captured by the infrared camera core is usually 16 bits, which means that each pixel has a depth of 16 bits and can represent 65536 different gray levels.

[0096] Compressing the first infrared original image to obtain multiple compressed images means de-framing the first infrared original image and compressing it into images with a bit width less than 16, for example, de-framing and compressing the first infrared original image into multiple compressed images with a bit width of 8 bits.

[0097] In step S2, the location of potential gas leakage is predicted based on multiple compressed images, which can be achieved by a deep model trained by machine learning or a traditional model trained by other machine learning algorithms.

[0098] The deep model for determining the location of potential gas leakage obtained through machine learning training is referred to as the gas leakage sequence detection model below. In the gas leakage sequence detection model, the compressed infrared image, i.e., the compressed image, is used as input for prediction, and the output is the location of the potential gas leakage.

[0099] like Figure 2 The figure shows a processing flow chart of the leakage gas sequence detection model provided in an embodiment of the present application. The leakage gas sequence detection model mainly includes a multi-frame image feature extraction module, a target rough generation module and a target selection module.

[0100] After compressing the original infrared image obtained by the infrared channel to obtain a compressed image, the multi-frame image feature extraction module receives multiple compressed images at different times as input, and extracts the gas feature information in each compressed image (hereinafter referred to as the image sequence) by capturing the full spatial information and time sequence information in each compressed image to obtain the feature extraction result. The feature extraction result is then input into the target rough generation module to output the rough position of the foreground target in the image (corresponding to the foreground target box in the figure). The leakage gas sequence detection model includes multiple target selection modules, each of which is composed of multiple convolutional layers and fully connected layers. Since the target coarse generation module outputs only the rough position of the foreground target, multiple target selection modules continuously refine the rough position of the foreground target on this basis to obtain the accurate position of the potential leakage gas. For example, the foreground target frame is input into the target selection module 1, and the gas screening frame 1 is output. Then the gas screening frame 1 is input into the target selection module 2, and the gas screening frame 2 is output. Then the gas screening frame 2 is input into the target selection module 3, and the gas screening frame 3 is output. Finally, the gas screening frame 3 is used as the output of the leakage gas sequence detection model to obtain the position of the potential leakage gas, that is, the suspected position of the leakage gas.

[0101] Among them, the multi-frame image feature extraction module can use CovNext network, or deep convolutional networks such as ResNet, DenseNet, etc., which are all possible, and the embodiments of this application do not limit this. The target rough generation module can use RPN network, or other networks, which are also not limited in this application.

[0102] Combination Figure 2 It can be seen that in the process of training the leaked gas sequence detection model, the modules of the leaked gas sequence detection model can be trained using sample compressed images with gas position true value labels, and the leaked gas sequence detection model is optimized based on the loss by calculating the loss function to obtain a trained leaked gas sequence detection model. The loss function includes but is not limited to cross entropy loss and mean square error loss.

[0103] The gas leakage sequence detection model can capture the full spatial information and temporal information in the target scene at the same time by adopting a sequence input method. In addition, since the prediction is based on compressed images, the amount of data is small and the location of each potential gas leakage can be quickly determined.

[0104] However, the processing capacity of the leaking gas sequence detection model is limited, and the input image sequence contains limited temporal information, usually 1 to 2 seconds in length. This may lead to false alarms of some targets that are similar to the imaging features of leaking gas, such as false alarms of gas caused by shadow changes. In the process of reviewing the positions of each potential leaking gas in subsequent steps S3 and S4, image frames with a longer time span may be selected.

[0105] In step S3, the second infrared original image is also uncompressed and obtained by shooting the target scene with an infrared camera or a bi-focal camera. The shooting time span of the plurality of first infrared original images is less than the shooting time span of the plurality of second infrared original images, and the plurality of first infrared original images are partial infrared images in the plurality of second infrared images, that is, the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images. Exemplarily, 10 consecutive image frames are obtained by shooting the target scene within 5 seconds from the current time, and they are numbered as image frame 1 to image frame 10 respectively, then the first infrared original image is at least one of any consecutive or multiple consecutively numbered image frames from image frame 1 to image frame 10, such as image frame 1 to image frame 5, or image frame 2 to image frame 4.

[0106] Based on the above, in order to facilitate the acquisition of the first infrared original image and the second infrared original image, in a possible implementation, after the original infrared image of the target scene is captured, each infrared original image will be stored. In this way, when performing gas leakage detection, the uncompressed infrared image of the target scene captured within the first time period before the detection moment can be directly obtained from the stored infrared original images as the first infrared original image, and the uncompressed infrared image of the target scene captured within the second time period before the detection moment can be obtained as the second infrared original image, wherein the second time period is greater than the first time period. Since the second time period is greater than the first time period, the first infrared original image can be considered to be a subset of the second infrared original image, which can ensure that the first classification results of all candidate positions can be predicted, thereby improving the accuracy of gas leakage detection.

[0107] Extracting a sub-image at a candidate position from multiple second infrared original images refers to extracting an image corresponding to the candidate position in the second infrared original image according to the candidate position. The extraction can be performed according to image processing software, or by a deep model for image extraction obtained through machine learning training. The embodiments of the present application are not limited to this.

[0108] Then, in step S4, the classification result of each candidate position is predicted. The classification result is predicted based on each infrared sub-image, which can be achieved by a deep model trained by machine learning for determining sequence classification, or by a traditional model trained by other machine learning algorithms.

[0109] In order to improve the accuracy of the classification result, in the process of predicting the classification result of the candidate position according to all the infrared sub-images, the spatial features and temporal features of the infrared sub-images can be extracted and prediction can be performed based on the spatial features and temporal features.

[0110] Based on this, Figure 3 The figure shows a second flow chart of the leakage gas detection method based on multi-data mixing provided in an embodiment of the present application, wherein the above step S4 specifically includes:

[0111] Step S41, extracting image features of at least one infrared sub-image as infrared spatial features; and extracting sequence features of an image sequence formed by arranging all infrared sub-images in chronological order of shooting time as infrared sequence features;

[0112] Among them, the infrared sequence features reflect the temporal characteristics of all infrared sub-images.

[0113] Step S42: predicting a first classification result of the candidate position according to the infrared spatial features and the infrared sequence features.

[0114] The method of the embodiment of the present application is selected to predict the classification results of the candidate positions by extracting the spatial features and temporal features of the infrared sub-image. Since the infrared sub-image and the spatial features and the temporal features can more comprehensively reflect the changes in the candidate positions, the accuracy of the classification results is improved.

[0115] It is understandable that, since gas leakage is dynamically changing, the number of features of leaked gas contained in infrared sub-images at different times is also different. In order to ensure that no false detection occurs, in a possible implementation, before spatial feature extraction, the number of image features in each infrared sub-image can be determined first, and each infrared sub-image can be scored according to the number of image features. The higher the score, the more image features are contained in the infrared sub-image, and then the infrared sub-image with the highest score is selected for spatial feature extraction.

[0116] Based on this, in a possible implementation manner, Figure 4 The third flow chart of the gas leakage detection method based on multi-data mixing provided in the embodiment of the present application is shown. The above step S41 may specifically include:

[0117] Step S411, predicting the gas scores of all infrared sub-images respectively;

[0118] The gas score is positively correlated with the number of features of the leaked gas contained in the image. The more features of the leaked gas contained in the image, the higher the gas score of the image.

[0119] Step S412, determining an image whose gas score meets a preset screening condition from all infrared sub-images as a target infrared sub-image;

[0120] The preset screening condition may refer to a score higher than a preset threshold or a highest score. The preset threshold is determined based on actual experience and is not limited in the present embodiment.

[0121] Step S413: extracting image features of the target infrared sub-image as infrared spatial features.

[0122] By selecting the method of the embodiment of the present application and determining the target infrared sub-image for spatial feature extraction based on the gas score of each infrared sub-image, it can be ensured that the extracted spatial features can comprehensively reflect all gas features in the target scene and avoid missing the location of leaking gas.

[0123] In the case where both the extraction of the original infrared image and the classification prediction of the candidate position are implemented by a deep model trained by machine learning, in order to reduce hardware costs and simplify the operation process, in a possible implementation, the deep model used for image extraction and the deep model used for classification prediction can be fused to obtain a leakage gas sequence classification model. In the leakage gas sequence classification model, the candidate position and the second original infrared image are used as input for prediction, and the classification result of the candidate position is output.

[0124] like Figure 5 The figure shows a processing flow chart of the leakage gas sequence classification model provided in an embodiment of the present application. The leakage gas sequence classification model includes a raw data local extraction module, a multi-frame raw data feature extraction module and a category classification module. The candidate position (i.e., the suspected position of the leakage gas) obtained in the above step S2 is input into the raw data local extraction module. The raw data local extraction module extracts the image at the candidate position in the infrared raw image corresponding to the compressed image where the candidate position is located according to the candidate position, and obtains the local multi-frame sequence raw data, i.e., multiple infrared sub-images.

[0125] Then, a multi-frame raw data feature extraction module is used to extract features from the local multi-frame raw data to obtain multi-dimensional features of the candidate positions. Finally, the extracted multi-dimensional features of the candidate positions are input into the category classification module to obtain the classification results.

[0126] Among them, the process of feature extraction of local multi-frame raw data by the multi-frame raw data feature extraction module includes: first, preprocessing the local multi-frame raw data, including but not limited to pixel value normalization, compression and amplification, image increment, etc., to obtain processed enhanced data; then, using the spatial feature extraction submodule and the temporal feature extraction submodule to extract features from the above-mentioned enhanced data from the two dimensions of time and space, respectively, to obtain spatial features and temporal features, and then inputting the above-mentioned temporal features and spatial features into the feature fusion submodule for fusion to obtain multi-dimensional features of the candidate position.

[0127] When extracting spatial features, the spatial feature extraction submodule first searches in each original infrared image frame sequence to determine the key frame (i.e., the original infrared image key frame is screened), and then extracts features from the key frame to obtain the spatial features of the candidate position. The spatial feature extraction submodel can be obtained by fusion of the Inception_V2 classification network and the DarkNet18 network, or by fusion of other classification networks and feature extraction networks, which is not limited in the embodiments of the present application; the temporal feature extraction network can be obtained by fusion of the ResNet18 network and the pseudo-3D network, or by other networks used for temporal feature extraction, which is not limited in the embodiments of the present application.

[0128] Combination Figure 5 It can be seen that in the process of training the leakage gas sequence classification model, after obtaining the sequence classification result, the leakage gas sequence classification model can be optimized using the cross entropy loss to obtain a trained leakage gas sequence classification model.

[0129] It can be understood that since gas is invisible in a visible light camera, but other objects have obvious features in a visible light camera, after determining the candidate position according to the above step S2, it is possible to further determine whether a gas leak occurs at the candidate position based on changes in the candidate position in both the infrared original image and the visible light image.

[0130] Based on this, see Figure 6 , Figure 6 A fourth flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application, wherein the method further includes:

[0131] Step S5, extracting sub-images at the candidate positions from the multiple visible light original images respectively as visible light sub-images;

[0132] The visible light original image is an uncompressed visible light image obtained by photographing the target scene, and the shooting time span of the plurality of second infrared original images is the same as the shooting time span of the plurality of visible light original images.

[0133] Step S6, predicting a second classification result of the candidate position based on all visible light sub-images, the second classification result is used to indicate that there is a moving object at the candidate position, or that there is no moving object at the candidate position;

[0134] Step S7: If the second classification result indicates that there is no moving object and the first classification result indicates that there is leaking gas, it is determined that there is leaking gas at the candidate position.

[0135] By selecting the embodiment of the present application and combining the visible light image with the infrared image to predict the classification result of the candidate position, the problem of false detection caused by moving objects or moving light and shadow can be reduced, and the accuracy of gas leakage detection can be improved.

[0136] Similar to the aforementioned step S4, in the process of predicting the classification results of the candidate positions according to all visible light sub-images, the spatial features and temporal features of the visible light sub-images may be extracted to perform prediction based on the spatial features and temporal features.

[0137] Based on this, in a possible implementation manner, Figure 7 The fifth flow chart of the gas leakage detection method based on multi-data mixing provided in the embodiment of the present application is shown, and the above step S6 also includes:

[0138] Step S61, extracting image features of the target visible light sub-image as visible light spatial features; and extracting sequence features of an image sequence formed by sorting all visible light sub-images in chronological order of shooting time as visible light sequence features;

[0139] Since there are many object attributes in the visible light image, when determining the target visible light sub-image, the key frame can no longer be determined based on the number of image features. Instead, the visible light sub-images are arranged in chronological order and the visible light sub-image farthest from the beginning and the end is taken as the target visible light sub-image.

[0140] Step S62, predicting a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature;

[0141] By selecting an embodiment of the present application, the classification results of the candidate positions are predicted by extracting the spatial features and temporal features of the visible light sub-image. Since the visible light sub-image and the spatial features and the temporal features can more comprehensively reflect whether there are movement conditions of objects other than gas, the accuracy of the classification results can be further improved.

[0142] In the case where the above steps S5 to S7 are all implemented by a deep model trained by machine learning, a deep model for image extraction and classification prediction can be trained, which is referred to as a visible light target analysis model below. Different from the above leakage gas sequence classification model, the visible light target analysis model is trained based on visible light images, and the input of the visible light target analysis model is the candidate position and the visible light image, and the output is the classification result of the candidate position.

[0143] like Figure 8 The figure shows a processing flow chart of the visible light target analysis model provided in the embodiment of the present application. The model structure of the visible light target analysis model is the same as the model structure of the above-mentioned leakage gas sequence classification model, including a raw data local extraction module, a multi-frame raw data feature extraction module and a category classification module. However, the raw data local extraction module of the visible light target analysis model is used to extract the visible light image according to the candidate position obtained in the above-mentioned step S2, and the multi-frame raw data feature extraction module is used to extract features from the visible light image.

[0144] For details, see Figure 8 First, the candidate position and the visible light image are input into the raw data local extraction module. The raw data local extraction module extracts the image at the candidate position in the visible light image corresponding to the compressed image where the candidate position is located according to the candidate position, and obtains local multi-frame sequence raw data, that is, multiple visible light sub-images.

[0145] Then, a multi-frame raw data feature extraction module is used to extract features from the local multi-frame raw data to obtain multi-dimensional features of the candidate positions. Finally, the extracted multi-dimensional features of the candidate positions are input into the category classification module to obtain the classification results.

[0146] Among them, the process of feature extraction of local multi-frame raw data by the multi-frame raw data feature extraction module includes: first, preprocessing the visible light raw data, including but not limited to pixel value normalization, compression and amplification, image increment, etc., to obtain processed enhanced data; then, using the spatial feature extraction submodule and the temporal feature extraction submodule to extract features from the above-mentioned enhanced data from the two dimensions of time and space, respectively, to obtain spatial features and temporal features, and then inputting the above-mentioned temporal features and spatial features into the feature fusion submodule for fusion to obtain multi-dimensional features of the candidate position.

[0147] When extracting spatial features, the spatial feature extraction submodule first selects the image frames that are farthest from the beginning and the end in each original infrared image frame sequence as key frames, such as the middle frame (i.e., the visible light image key frame screening); then extracts features from the key frames to obtain the spatial features of the candidate positions. The spatial feature extraction submodel can be a DarkNet18 network or other feature extraction networks, which is not limited in the embodiments of the present application; the temporal feature extraction network can be a fusion of a ResNet18 network and a pseudo-3D network, or other networks used for temporal feature extraction, which is not limited in the embodiments of the present application.

[0148] Combination Figure 8 It can be seen that in the process of training the visible light target analysis model, after obtaining the sequence classification result, the visible light target analysis model can be optimized using the cross entropy loss to obtain a trained visible light target analysis model.

[0149] In order to facilitate relevant personnel to deal with the leaked gas in a timely manner, after obtaining the above classification results, an alarm can be issued based on the classification results. It can be understood that in order to achieve automatic alarm, in a possible implementation, a result processing analysis model can be pre-trained, and the result processing analysis model can integrate the output results of the above models to alarm the location where the leaked gas exists.

[0150] For infrared cameras or dual-light cameras without visible light analysis, the result processing and analysis model directly issues an alarm based on the classification results output by the leakage gas sequence classification model; for dual-light cameras with visible light analysis turned on, the result processing and analysis model issues an alarm based on the output results of the visible light target analysis model and the classification results output by the leakage gas sequence classification model.

[0151] In order to more clearly explain the leakage gas detection method based on multi-data mixing of the embodiment of the present application, the following is described in conjunction with a specific embodiment. Fig. 9 The figure shows an example flow chart of a gas leakage detection method based on multi-data mixing provided in an embodiment of the present application.

[0152] Taking the camera that shoots the target scene as a dual-light camera as an example, after the scene data of the target scene is captured, the data of the visible light channel (i.e., the visible light image) is cached in multiple frames, and the data of the infrared channel (i.e., the infrared original image) is cached in multiple frames. Then the infrared original image is compressed, and the compressed multiple image frames are input into the 8-bit wide leak gas sequence detection model to output the candidate position, and then the candidate position and the infrared original image are input into the 16-bit wide leak gas sequence classification model to obtain the first classification result, i.e., the leak gas candidate position, and the candidate position and the visible light image are input into the 16-bit wide visible light target analysis model for prediction to obtain the second classification result, and finally the result analysis is performed based on the first classification result and the second classification result, and the candidate position where the gas leak occurs (i.e., the leak gas alarm position) is alarmed.

[0153] In a possible implementation, the above-mentioned leakage gas sequence detection model, leakage gas sequence classification model and visible light target analysis model can be integrated into a leakage gas detection model. In this embodiment, the leakage gas detection model is pre-trained according to the following method:

[0154] Acquire a plurality of sample compressed images obtained by respectively compressing a plurality of first infrared sample images, wherein the first infrared sample images are uncompressed infrared images obtained by photographing a preset scene;

[0155] Predicting the potential gas leakage location based on multiple sample compression images as a sample candidate location;

[0156] Extracting sub-images at the sample candidate positions from a plurality of second infrared sample images respectively as sample infrared sub-images, wherein the second infrared sample images are uncompressed infrared images obtained by photographing the preset scene and marked with gas position true value labels, and the shooting time span of the plurality of first infrared sample images is a true subset of the shooting time span of the plurality of second infrared sample images;

[0157] Predicting a first classification result of the sample candidate position according to all the sample infrared sub-images, the first classification result is used to indicate that there is a gas leak at the candidate position, or that there is no gas leak at the candidate position;

[0158] The model parameters of the leakage gas detection model are adjusted according to the loss between the sample candidate position and the gas position and the loss between the first classification result and the actual classification result, and the step of predicting the position of potential leakage gas based on multiple sample compressed images as the sample candidate position is returned to execute until the loss between the first classification result and the actual classification result reaches a preset convergence condition.

[0159] Corresponding to the aforementioned gas leakage detection method based on multiple data mixing, the third aspect of the embodiment of the present application further provides a gas leakage detection device based on multiple data mixing, see Fig.10 , Fig.10 The structural schematic diagram of the leakage gas detection device based on multi-data mixing provided in the embodiment of the present application includes:

[0160] A first acquisition module 1001 is used to acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene;

[0161] A position prediction module 1002 is used to predict the position of potential gas leakage according to the multiple compressed images as a candidate position;

[0162] A first extraction module 1003 is used to extract sub-images at the candidate positions from a plurality of second infrared original images, respectively, as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images;

[0163] The first classification module 1004 is used to predict the first classification result of the candidate position according to all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or there is no gas leak at the candidate position.

[0164] By applying the above-mentioned embodiment, since the data volume of the compressed infrared image is small and the processing speed is relatively fast, each candidate position can be quickly obtained by determining the candidate position based on the compressed infrared image; and since the information included in the infrared original image is more comprehensive than the information included in the compressed infrared original image, after the candidate positions of the potential gas leakage are obtained based on the compressed infrared original image, by extracting the sub-image of each candidate position in the infrared original image, it can be accurately determined whether there is a gas leakage at each candidate position based on each sub-image, thereby realizing fast and accurate detection of the leaking gas.

[0165] In addition, predicting the candidate locations of potential gas leakage based on the compressed infrared original image of the target scene can be achieved based on a pre-trained model. During the process of training the model, the input of the model is also the compressed infrared original image. In this way, since the cost of compressing the image is relatively low, if the current scene is relatively complex, the model can be iterated in a relatively short time, thereby improving the scene adaptability of the model; and, predicting the first classification result of the candidate location based on all infrared sub-images can also be achieved based on a pre-trained model. During the process of training the model, the input of the model is also the infrared sub-image. Since each infrared sub-image is a sub-image at the candidate location in the infrared original image, and the candidate location is usually small, the processing task of the model is relatively simple, and the model does not need to be frequently iterated and updated, thereby achieving fast and accurate gas leakage detection effects while not increasing or increasing the control cost very little.

[0166] In a possible implementation, the first classification module 1004 includes:

[0167] The first classification submodule is used to extract the image features of at least one of the infrared sub-images as infrared spatial features; and to extract the sequence features of the image sequence formed by sorting all the infrared sub-images in chronological order of shooting time as infrared sequence features;

[0168] The second classification submodule is used to predict a first classification result of the candidate position according to the infrared spatial feature and the infrared sequence feature.

[0169] In a possible implementation, the first classification submodule includes:

[0170] A first classification unit, used for respectively predicting gas scores of all the infrared sub-images, wherein the gas scores are positively correlated with the number of features of the leaked gas contained in the image;

[0171] A second classification unit is used to determine, from all the infrared sub-images, an image whose gas score meets a preset screening condition as a target infrared sub-image;

[0172] The third classification unit is used to extract image features of the target infrared sub-image as infrared spatial features.

[0173] In a possible implementation, the device further includes:

[0174] a second extraction module, configured to extract sub-images at the candidate positions from a plurality of visible light original images, respectively, as visible light sub-images, wherein the visible light original images are uncompressed visible light images obtained by photographing the target scene, and the photographing time span of the plurality of second infrared original images is the same as the photographing time span of the plurality of visible light original images;

[0175] A second classification module, used for predicting a second classification result of the candidate position according to all the visible light sub-images, wherein the second classification result is used for indicating that a moving object exists at the candidate position, or that a moving object does not exist at the candidate position;

[0176] The result determination module is configured to determine that there is a gas leak at the candidate position if the second classification result indicates that there is no moving object and the first classification result indicates that there is a gas leak.

[0177] In a possible implementation manner, the second classification module includes:

[0178] a fourth classification unit, configured to extract image features of the target visible light sub-image as visible light spatial features; and extract sequence features of an image sequence formed by sorting all the visible light sub-images in chronological order of shooting time as visible light sequence features;

[0179] a fifth classification unit, configured to predict a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature;

[0180] The target visible light sub-image is a visible light sub-image that is farthest from the beginning and the end in the image sequence.

[0181] In a possible implementation, the device further includes:

[0182] A second acquisition module is used to acquire an uncompressed infrared image obtained by photographing the target scene within a first time period from the time to be detected as a first infrared original image;

[0183] The third acquisition module is used to acquire an uncompressed infrared image obtained by shooting the target scene within a second time period from the time to be detected as a second infrared original image, wherein the second time period is greater than the first time period.

[0184] The present application also provides an electronic device, such as Fig.11 As shown, including:

[0185] Memory 1101, used for storing computer programs;

[0186] The processor 1102 is used to implement the following steps when executing the program stored in the memory 1101:

[0187] Acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene;

[0188] Predicting a location of potential gas leakage according to the plurality of compressed images as a candidate location;

[0189] Extracting sub-images at the candidate positions from a plurality of second infrared original images respectively as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images;

[0190] A first classification result of the candidate position is predicted based on all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or that there is no gas leak at the candidate position.

[0191] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0192] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0193] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned gas leakage detection methods based on multi-data mixing are implemented.

[0194] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the gas leakage detection methods based on multi-data mixing in the above embodiments.

[0195] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0196] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0197] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0198] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A gas leakage detection method based on multi-data mixing, characterized in that: The method comprises: Acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene, and information included in the first infrared original images is more comprehensive than information included in the compressed images; Predicting a location of potential gas leakage according to the plurality of compressed images as a candidate location; Extracting sub-images at the candidate positions from a plurality of second infrared original images respectively as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a proper subset of the shooting time span of the plurality of second infrared original images; A first classification result of the candidate position is predicted based on all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or that there is no gas leak at the candidate position.

2. The method according to claim 1, characterized in that The first classification result of the candidate position is obtained by predicting all the infrared sub-images, including: Extracting an image feature of at least one of the infrared sub-images as an infrared spatial feature; and extracting a sequence feature of an image sequence formed by arranging all the infrared sub-images in chronological order of shooting time as an infrared sequence feature; A first classification result of the candidate position is predicted based on the infrared spatial features and the infrared sequence features.

3. The method according to claim 2, characterized in that The step of extracting the image feature of at least one infrared sub-image as an infrared spatial feature comprises: Predicting gas scores of all the infrared sub-images respectively, wherein the gas scores are positively correlated with the number of features of the leaking gas contained in the images; Determine, from all the infrared sub-images, an image whose gas score meets a preset screening condition as a target infrared sub-image; The image features of the target infrared sub-image are extracted as infrared spatial features.

4. The method according to claim 1, characterized in that The method further comprises: Extracting sub-images at the candidate positions from a plurality of visible light original images respectively as visible light sub-images, wherein the visible light original images are uncompressed visible light images obtained by photographing the target scene, and the shooting time span of the plurality of second infrared original images is the same as the shooting time span of the plurality of visible light original images; Predicting a second classification result of the candidate position according to all the visible light sub-images, the second classification result being used to indicate that a moving object exists at the candidate position, or that a moving object does not exist at the candidate position; If the second classification result indicates that there is no moving object, and the first classification result indicates that there is leaked gas, it is determined that there is leaked gas at the candidate position.

5. The method according to claim 4, characterized in that The predicting of the second classification result of the candidate position according to all the visible light sub-images includes: Extracting image features of the target visible light sub-image as visible light spatial features; and extracting sequence features of an image sequence formed by sorting all the visible light sub-images in chronological order of shooting time as visible light sequence features; Predicting a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature; The target visible light sub-image is a visible light sub-image that is farthest from the beginning and the end in the image sequence.

6. The method according to claim 1, characterized in that The method further comprises: Acquire an uncompressed infrared image obtained by photographing the target scene within a first time period from the time to be detected as a first infrared original image; An uncompressed infrared image obtained by photographing the target scene within a second time period from the time to be detected is obtained as a second infrared original image, wherein the second time period is greater than the first time period.

7. A gas leakage detection device based on multi-data mixing, characterized in that: The device comprises: A first acquisition module is used to acquire a plurality of compressed images obtained by respectively compressing a plurality of first infrared original images, wherein the first infrared original images are uncompressed infrared images obtained by photographing a target scene, and the information included in the first infrared original images is more comprehensive than the information included in the compressed images; A position prediction module, used for predicting the position of potential gas leakage according to the plurality of compressed images as a candidate position; A first extraction module is used to extract sub-images at the candidate positions from a plurality of second infrared original images as infrared sub-images, wherein the second infrared original images are uncompressed infrared images obtained by photographing the target scene, and the shooting time span of the plurality of first infrared original images is a true subset of the shooting time span of the plurality of second infrared original images; The first classification module is used to predict the first classification result of the candidate position according to all the infrared sub-images, and the first classification result is used to indicate that there is a gas leak at the candidate position, or there is no gas leak at the candidate position.

8. The device according to claim 7, characterized in that The first classification module comprises: The first classification submodule is used to extract the image features of at least one of the infrared sub-images as infrared spatial features; and to extract the sequence features of the image sequence formed by sorting all the infrared sub-images in chronological order of shooting time as infrared sequence features; A second classification submodule, used for predicting a first classification result of the candidate position according to the infrared spatial feature and the infrared sequence feature; The first classification submodule includes: A first classification unit, used for respectively predicting gas scores of all the infrared sub-images, wherein the gas scores are positively correlated with the number of features of the leaked gas contained in the image; A second classification unit is used to determine, from all the infrared sub-images, an image whose gas score meets a preset screening condition as a target infrared sub-image; A third classification unit is used to extract image features of the target infrared sub-image as infrared spatial features; The device also includes: a second extraction module, configured to extract sub-images at the candidate positions from a plurality of visible light original images, respectively, as visible light sub-images, wherein the visible light original images are uncompressed visible light images obtained by photographing the target scene, and the photographing time span of the plurality of second infrared original images is the same as the photographing time span of the plurality of visible light original images; A second classification module, used for predicting a second classification result of the candidate position according to all the visible light sub-images, wherein the second classification result is used for indicating that a moving object exists at the candidate position, or that a moving object does not exist at the candidate position; a result determination module, configured to determine that there is a gas leak at the candidate position if the second classification result indicates that there is no moving object and the first classification result indicates that there is a gas leak; The second classification module comprises: a fourth classification unit, configured to extract image features of the target visible light sub-image as visible light spatial features; and extract sequence features of an image sequence formed by sorting all the visible light sub-images in chronological order of shooting time as visible light sequence features; a fifth classification unit, configured to predict a second classification result of the candidate position according to the visible light spatial feature and the visible light sequence feature; Wherein, the target visible light sub-image is a visible light sub-image which is farthest from the beginning and the end in the image sequence; The device also includes: A second acquisition module is used to acquire an uncompressed infrared image obtained by photographing the target scene within a first time period from the time to be detected as a first infrared original image; The third acquisition module is used to acquire an uncompressed infrared image obtained by shooting the target scene within a second time period from the time to be detected as a second infrared original image, wherein the second time period is greater than the first time period.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images