Processing method and device for gas monitoring image, medium and equipment

Abstract

The invention discloses a processing method and device for a gas monitoring image, a medium and equipment, and the method comprises the steps: obtaining the multi-modal image data and monitoring data of gas monitoring equipment in a monitoring region based on a gas platform, generating a neural network through a pre-constructed feature, processing the multi-modal image data, obtaining a first feature of the gas monitoring equipment, and obtaining the first feature of the gas monitoring equipment; processing the monitoring data and the first feature by using a pre-constructed state recognition neural network to obtain a state recognition result of the gas monitoring equipment, the state recognition neural network comprising a multi-mode data fusion network and a classification network which are connected in sequence, unified interconnection and unified calculation of various monitoring data and dynamic sensing data which are distributed in various spatial positions in a large scale are achieved, unified data acquisition and processing are carried out on the data, the state recognition result of the gas monitoring equipment is automatically obtained, and a foundation is laid for intelligent gas monitoring management.

Description

Technical Field

[0001] The present application relates to the field of Internet of Things technology, and in particular to a method, device, medium and equipment for processing gas monitoring images. Background Art

[0002] With the acceleration of urbanization, gas, as an important part of urban energy, has become particularly important for safety monitoring and management. Traditional gas monitoring systems often rely on a single data source, such as temperature, pressure and other sensor data. Although these data can provide certain monitoring information, they often lack sufficient information to fully evaluate the operating status of gas equipment. In addition, traditional monitoring systems also have limitations in data processing and analysis, making it difficult to achieve large-scale, efficient monitoring and management. Summary of the invention

[0003] The main purpose of this application is to provide a method, device, medium and equipment for processing gas monitoring images, aiming to solve the above-mentioned technical problems caused by manual monitoring of gas equipment.

[0004] To achieve the above-mentioned objectives, the present application provides a method for processing gas monitoring images, comprising: obtaining multimodal image data and monitoring data of a gas monitoring device in a monitoring area based on a gas platform; processing the multimodal image data using a pre-constructed feature generation neural network to obtain a first feature of the gas monitoring device, wherein the feature generation neural network is based on a fusion generative adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; processing the monitoring data and the first feature using a pre-constructed state recognition neural network to obtain a state recognition result of the gas monitoring device, and the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence.

[0005] Optionally, before using a pre-constructed feature generation neural network to process the multimodal image data, the processing method for gas monitoring images also includes: processing the multimodal image data by scale transformation to obtain a first input image; normalizing and dimensionality adjusting the first input image to obtain a second input image; and standardizing the second input image to obtain the pre-processed multimodal image data.

[0006] Optionally, the use of a pre-constructed feature generation neural network to process the multimodal image data to obtain the first feature of the gas monitoring equipment includes: using the pre-constructed fusion generative adversarial network to process the multimodal image data to obtain a fused image containing multi-mode features, wherein the fusion generative adversarial network includes a generator, and the generator includes a fully connected module and a convolution module connected in sequence, wherein the convolution module includes a first batch normalization layer, a first upsampling module, a first one-dimensional convolution layer, a second batch normalization layer, a first activation layer, a second upsampling module, a second one-dimensional convolution layer, a third batch normalization layer, a second activation layer and a third one-dimensional convolution layer connected in sequence; using a pre-constructed feature extraction network to process the fused image to obtain multi-scale image features; using a pre-constructed multi-scale feature fusion network to process the multi-scale image features to obtain the first feature of the gas monitoring equipment after the multi-scale features are fused.

[0007] Optionally, the fused image is processed using a pre-built feature extraction network to obtain multi-scale image features, including:

[0008] The fused image is processed by using a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a pooling layer, a first fully connected layer, and a second fully connected layer that are connected in sequence to obtain multi-scale image features.

[0009] Optionally, the multi-scale image features are processed using a pre-constructed multi-scale feature fusion network to obtain a first feature of the gas monitoring device after multi-scale feature fusion, including:

[0010] The multi-scale image features are processed using a feature pyramid network to obtain a first feature of the gas monitoring equipment after multi-scale feature fusion.

[0011] Optionally, acquiring monitoring data includes:

[0012] Acquire real-time historical data and / or alarm data and / or trend data of pressure parameters, flow parameters and temperature parameters.

[0013] Optionally, the use of a prebuilt state recognition neural network to process the monitoring data and the first feature to obtain a state recognition result of the gas monitoring equipment includes: using a prebuilt multi-source data fusion network to process the monitoring data and the first feature to obtain a second feature of data fusion; using a prebuilt classification network to process the second feature to obtain a state recognition result of the gas monitoring equipment, wherein the gas monitoring equipment includes a flow monitor, a valve, a gas safety pipe network controller, a temperature and humidity sensor, a methanol flow meter, a gas pressure regulating valve and a fire station.

[0014] To achieve the above-mentioned purpose, the present application provides a gas monitoring image processing device, including: an acquisition module, used to acquire multimodal image data and monitoring data of a gas monitoring device in a monitoring area based on a gas platform; a processing module, used to process the multimodal image data using a pre-built feature generation neural network to obtain a first feature of the gas monitoring device, wherein the feature generation neural network is based on a fusion generative adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; an output module, used to process the monitoring data and the first feature using a pre-built state recognition neural network to obtain a state recognition result of the gas monitoring device, and the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence.

[0015] To achieve the above objectives, the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the processing method for gas monitoring images in any of the above embodiments is implemented.

[0016] To achieve the above objectives, the present application provides a terminal device, including: a memory and a processor; the processor executes a computer program stored in the memory to implement the processing method for gas monitoring images in any of the above embodiments.

[0017] The embodiments of the present application propose a method, device, medium and equipment for processing gas monitoring images. The method obtains multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area based on the gas platform; uses a pre-constructed feature generation neural network to process the multimodal image data to obtain a first feature of the gas monitoring equipment, wherein the feature generation neural network is composed of a fusion generative adversarial network, a feature extraction network and a multi-scale feature fusion network connected in sequence; uses a pre-constructed state recognition neural network to process the monitoring data and the first feature to obtain a state recognition result of the gas monitoring equipment, and the state recognition neural network includes a multi-modal data fusion network and a classification network connected in sequence, which realizes the unified interconnection and unified calculation of various monitoring data and dynamic sensor data dispersed in various spatial locations on a large scale, performs unified data collection and processing on the above data, and automatically obtains the state recognition result of the gas monitoring equipment, laying the foundation for intelligent gas monitoring management. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flowchart of a method for processing a gas monitoring image provided in an embodiment of the present application;

[0019] Figure 2 A structural block diagram of a gas monitoring image processing device provided in an embodiment of the present application;

[0020] Figure 3A schematic diagram of a storage medium structure provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application.

[0022] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0023] It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0024] In the prior art, the monitoring of equipment status is carried out manually. The gas meter is mainly held by hand and the cumulative cubic meters displayed on the meter are read. The gas leakage in the pipeline where the meter is located can be calculated by accumulating data. This is a traditional mode and a general mode in the field, that is, a manual collection mode, which is essentially an industrial manifestation of the manual monitoring operation mode. However, the main disadvantages of manual monitoring are: first, due to the limitations of manual operation steps, a large amount of intermediate data is unrealistically generated, such as the need for manual image information collection and image information recognition, and the purpose of gas monitoring cannot be achieved in one step; second, the gas monitoring system is mostly deployed in the field. The environment where the gas device is located is very poor, some of which are plateaus, caves or underground, or in places where there are few people. The work intensity of manual inspection is high and the efficiency is low, which easily leads to the extension of the detection cycle, so that real-time monitoring cannot be performed; third, as the scope of equipment deployment expands, the workload of manual inspection continues to increase.

[0025] In view of the problem of labor cost and operation efficiency in equipment status monitoring, a method for gas monitoring image processing is now needed to achieve real-time monitoring and obtain equipment status, thereby improving the work efficiency of gas monitoring.

[0026] This application provides a method for processing gas monitoring images. Figure 1 A flowchart of a method for processing a gas monitoring image provided in an embodiment of the present application, which method can be implemented by a terminal with edge computing capabilities, such as Figure 1 As shown, the method may include the following execution process:

[0027] S10, acquiring multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area based on the gas platform;

[0028] Among them, in the gas monitoring system based on , through the advanced monitoring equipment deployed at key nodes, multimodal image data and monitoring data of the gas monitoring area can be obtained in real time. These data include but are not limited to visible light images, infrared thermal imaging images, and monitoring data collected by sensors such as pressure, flow, and temperature. This information is crucial to understanding the operating status of gas equipment.

[0029] It should be noted that in addition to visible light images and infrared thermal imaging images, multimodal images can also be ultraviolet images, X-ray or gamma ray images, sonar images, laser scanning images, drone images, multispectral or hyperspectral images, etc. Among them, ultraviolet images can be obtained by detecting the specific fluorescence reactions of certain materials through ultraviolet cameras, which is very useful in detecting tiny cracks or specific types of leaks.

[0030] X-ray or gamma-ray images can penetrate objects and can be used to detect internal structures or hidden defects; sonar images can be used to create images of pipelines or other hidden structures to detect corrosion or blockages; laser scanning images can be captured using laser radar (LiDAR) technology, which can provide precise distance and three-dimensional information of the scene; drone images can be obtained from high altitudes and can be used to monitor large areas of gas facilities and pipeline networks; multispectral or hyperspectral images can capture multiple bands beyond the visible spectrum of the human eye, which can reveal the chemical composition or biological properties of materials.

[0031] In an embodiment of the present application, after step S10, the method for processing gas monitoring images may further include the following image preprocessing process:

[0032] S41, scaling the multimodal image data to obtain a first input image;

[0033] Among them, scale transformation is an important preprocessing step when processing multimodal image data. This step can involve adjusting images from different sources or different resolutions to a uniform size so that they can be processed by the neural network. For example, if the images captured by the monitoring equipment have different sizes, scale transformation will ensure that all images are adjusted to a size suitable for network input, such as 256x256 pixels. This step can use interpolation methods such as bilinear interpolation or nearest neighbor interpolation to avoid information loss due to inconsistent sizes.

[0034] S42, normalizing and dimensionally adjusting the first input image to obtain a second input image;

[0035] Specifically, normalization and dimensionality adjustment are two key steps in image preprocessing. Normalization can refer to scaling image data to a specific range, such as [0, 1] or [-1, 1], which helps the neural network converge faster and improves the stability of training. Dimensionality adjustment involves converting image data from three dimensions (height, width, number of channels) to the input format required by the neural network, for example, converting image data from HWC (height, width, number of channels) format to CHW (number of channels, height, width) format. The above process can avoid information loss due to inconsistent sizes, and channel alignment and removal of unnecessary data improve the performance of the algorithm.

[0036] S43: Standardize the second input image to obtain the pre-processed multimodal image data.

[0037] Standardization is another important step in image preprocessing, which involves using the statistical properties of image data, such as mean and standard deviation, to adjust the data distribution. This process can include subtracting the mean from each pixel value and dividing by the standard deviation, so that the data has zero mean and unit variance. This treatment helps improve the generalization ability of subsequent algorithms on the input data and reduces the correlation between different features, thereby improving the performance of the algorithm.

[0038] Through these preprocessing steps, multimodal image data are converted into a common format so that they can be effectively processed by neural networks for subsequent feature extraction and state recognition tasks. These steps are crucial to ensure that the model can extract useful information from various image data and are the basis for achieving accurate state recognition.

[0039] S20, using a pre-built feature generation neural network to process the multimodal image data to obtain a first feature of the gas monitoring device, wherein the feature generation neural network is composed of a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence;

[0040] In this step, in order to extract useful features from multimodal image data, the present invention provides a feature generation neural network, which first uses a fusion generative adversarial network (such as FusionGAN) to integrate multimodal image data to generate a fused image containing multiple modal information. Then, the feature extraction network processes the fused image to extract image features. Finally, the multi-scale feature fusion network effectively integrates feature maps of different scales to obtain a richer first feature set.

[0041] In an embodiment of the present invention, step S20 may include the following specific execution process:

[0042] S21, using the pre-constructed fusion generative adversarial network to process the multimodal image data to obtain a fusion image containing multi-model features, wherein the fusion generative adversarial network includes a generator, and the generator includes a fully connected module and a convolution module connected in sequence, wherein the convolution module includes a first batch normalization layer, a first upsampling module, a first one-dimensional convolution layer, a second batch normalization layer, a first activation layer, a second upsampling module, a second one-dimensional convolution layer, a third batch normalization layer, a second activation layer and a third one-dimensional convolution layer connected in sequence;

[0043] Exemplarily, the fully connected module includes a fully connected layer, and the convolution module includes a first batch normalization layer, an upsampling module, a first one-dimensional convolution layer, a second batch normalization layer, a first activation function, a second upsampling module, a second one-dimensional convolution layer, a third batch normalization layer, and a second activation function connected in sequence. Among them, the input nodes of the fully connected layer are 2 and the output nodes are 128 (the default input is 2 modes), the feature channels of the first batch normalization layer are 128; the scaling factor of the first upsampling module is 2; the input channels of the first one-dimensional convolution layer are 128, the output channels are 128, and the convolution kernel is 3; the feature channels of the second batch normalization layer are 128; the negative slope parameter of the first activation function is 0.2; the scaling factor of the second upsampling module is 2; the input channels of the second one-dimensional convolution layer are 128, the output channels are 64, and the convolution kernel is 3; the feature channels of the third batch normalization layer are 64; the negative slope parameter of the second activation function is 0.2; the input channels of the third one-dimensional convolution layer are 64, the output channels are 1, and the convolution kernel is 3. In this way, the generative network of the UNET structure can be used to accurately segment the region of interest in the image.

[0044] S22, using a pre-built feature extraction network to process the fused image to obtain multi-scale image features;

[0045] Specifically, the terminal can use a pre-built feature extraction network to process the fused image. This network can include multiple convolutional layers to extract key visual features in the image. For example, a convolutional neural network (CNN) can be used to automatically learn local features in an image and gradually build more abstract high-level features.

[0046] Exemplarily, the feature extraction network includes a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence, wherein the input channel of the first two-dimensional convolutional layer can be 1, the output channel is 648, and the convolution kernel is 3; the input channel of the second two-dimensional convolutional layer can be 64, the output channel is 128, and the convolution kernel is 3; the pooling window of the pooling layer is 2x2; the number of input nodes of the first fully connected layer is 128*6*6, and the number of output nodes is 512; the number of input nodes of the second fully connected layer is 512, and the number of output nodes is 128. The feature extraction network can automatically learn local features in the image.

[0047] S23. Use a pre-constructed multi-scale feature fusion network to process the multi-scale image features to obtain a first feature of the gas monitoring device after multi-scale feature fusion.

[0048] In this step, the terminal can use a multi-scale feature fusion network to process the extracted multi-scale image features. This network structure can be, for example, the Inception module in the Inception network, or a serial skip-layer connection structure, such as a feature pyramid network (FPN). These structures can extract features under different receptive fields and effectively fuse features at different levels. After the multi-scale feature fusion, the first feature of the gas monitoring equipment is obtained. These features contain rich information of the image, such as edges, textures, shapes, etc. In order to fuse these image features with the monitoring data, a fusion layer is designed, which can merge feature vectors using simple concatenation or weighted average.

[0049] In an embodiment of the present invention, the multi-scale feature fusion network is a feature pyramid network.

[0050] Exemplarily, the feature pyramid network includes an upsampling convolution layer and a lateral convolution layer, wherein the upsampling convolution layer includes: a first upsampling convolution layer to a third upsampling convolution layer, wherein the first upsampling convolution layer has 128 input channels, 64 output channels, a convolution kernel size of 2x2, and a step size of 2; the second upsampling convolution layer has 64 input channels, 32 output channels, a convolution kernel size of 2x2, and a step size of 2; the third upsampling convolution layer has 32 input channels, 1 output channel, a convolution kernel size of 2x2, and a step size of 2. The horizontal convolution layer includes the first horizontal convolution layer to the third horizontal convolution layer, where the first horizontal convolution layer has 128 input channels, 64 output channels, and a convolution kernel size of 1x1; the second horizontal convolution layer has 64 input channels, 32 output channels, and a convolution kernel size of 1x1; the third horizontal convolution layer has 32 input channels, 1 output channel, and a convolution kernel size of 1x1. The upsampling convolution layer can be used to upsample the high-level (semantic information-rich but low-resolution) feature maps so that they can be fused with the low-level (detail information-rich but semantic information-weak) feature maps. The horizontal convolution layer is used to adjust the number of channels of the feature map before fusion to ensure that feature maps of different levels can be effectively combined. This design enables the network to simultaneously utilize the high-level semantic information and low-level detail information of the image, thereby improving the ability to recognize objects of different scales. The feature pyramid network can upsample the high-level (rich in semantic information but low in resolution) feature maps to fuse them with the low-level (rich in detail information but weak in semantic information) feature maps, thereby improving the recognition ability of objects of different scales.

[0051] S30. Process the monitoring data and the first feature using a pre-built state recognition neural network to obtain a state recognition result of the gas monitoring device. The state recognition neural network includes a multi-mode data fusion network and a classification network connected in sequence.

[0052] Among them, the terminal inputs the fused features into the state recognition neural network. This network can include one or more fully connected layers to map the fused features to different operating states of the device. Through training, the network can learn patterns to distinguish between normal operating states and various potential failure modes. Ultimately, the state recognition network will output the operating state of the device, which can be a category label (such as "normal", "abnormal") or a continuous score indicating the severity of the device state.

[0053] In an embodiment of the present invention, step S30 may include the following specific execution steps:

[0054] S31, using a pre-built multi-source data fusion network to process the monitoring data and the first feature to obtain a second feature of data fusion;

[0055] S32. Use the pre-built classification network to process the second feature to obtain the state recognition result of the gas monitoring equipment, wherein the gas monitoring equipment includes a flow monitor, a valve, a gas safety pipe network controller, a temperature and humidity sensor, a methanol flow meter, a gas pressure regulating valve and a fire station.

[0056] In step S31 and step S32, the model data fusion network and the classification network are both fully connected neural networks. The terminal combines multi-scale features and monitoring data to perform final state recognition by executing steps S31-S32. In addition, by fusing the monitoring data with the first feature data, information-rich feature data can be obtained.

[0057] Optionally, after step S30, the method for processing gas monitoring images may further include the following execution process:

[0058] S50. Once the training is completed, the state recognition neural network will be deployed on the terminal of the gas monitoring platform to receive multimodal data from the monitoring equipment in real time. The state recognition neural network will automatically process this data and output the state recognition results of the equipment, which can provide immediate decision support.

[0059] S60. In addition, the system includes a feedback mechanism that allows operators to verify and adjust the state recognition results. This manual feedback will be used to further fine-tune the network, forming a continuous improvement cycle to ensure that the accuracy and reliability of state recognition improves over time.

[0060] Through these steps, the gas monitoring system based on can realize real-time, automatic and accurate status monitoring of gas equipment, thereby improving the safety and efficiency of gas supply.

[0061] The processing method for gas monitoring images proposed in the embodiment of the present application obtains multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area based on the gas platform; uses a pre-constructed feature generation neural network to process the multimodal image data to obtain a first feature of the gas monitoring equipment, wherein the feature generation neural network is composed of a fusion generative adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; uses a pre-constructed state recognition neural network to process the monitoring data and the first feature to obtain a state recognition result of the gas monitoring equipment, and the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence, which realizes the unified interconnection and unified calculation of various monitoring data and dynamic sensor data dispersed in various spatial locations on a large scale, performs unified data collection and processing on the above data, and automatically obtains the state recognition result of the gas monitoring equipment, laying the foundation for intelligent gas monitoring management.

[0062] Based on the above embodiments, the present application provides a gas monitoring image processing device, Figure 2 For the structure diagram of the gas monitoring image processing device provided in the embodiment of the present application, refer to Figure 2 The structure 100 of the gas monitoring image processing device may include an acquisition module 101, a processing module 102 and an output module 103, wherein the acquisition module 101 is used to acquire multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area based on the gas platform; the processing module 102 is used to use a pre-constructed feature generation neural network to process the multimodal image data to obtain a first feature of the gas monitoring equipment, specifically, the feature generation neural network is based on a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; the output module 103 is used to use a pre-constructed state recognition neural network to process the monitoring data and the first feature to obtain a state recognition result of the gas monitoring equipment, and the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence.

[0063] In an embodiment of the present application, the structure 100 of the gas monitoring image processing device may also include a preprocessing module 104, which may be used to scale transform the original visible light image and the original infrared light image to obtain a first input image; normalize and dimensionally adjust the first input image to obtain a second input image; and standardize the second input image to obtain the preprocessed multimodal image data.

[0064] In an embodiment of the present application, the processing module 102 can also be used to use the pre-constructed fusion generative adversarial network to process the multimodal image data to obtain a fused image containing multi-model features, wherein the fusion generative adversarial network includes a generator, the generator includes a fully connected module and a convolution module connected in sequence, the convolution module includes a first batch normalization layer, a first upsampling module, a first one-dimensional convolution layer, a second batch normalization layer, a first activation layer, a second upsampling module, a second one-dimensional convolution layer, a third batch normalization layer, a second activation layer and a third one-dimensional convolution layer connected in sequence; the fused image is processed using a pre-constructed feature extraction network to obtain multi-scale image features; the multi-scale image features are processed using a pre-constructed multi-scale feature fusion network to obtain the first feature of the gas monitoring equipment after the multi-scale features are fused.

[0065] In an embodiment of the present application, the processing module 102 can also be used to process the fused image using a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a pooling layer, a first fully connected layer, and a second fully connected layer connected in sequence to obtain multi-scale image features.

[0066] In an embodiment of the present application, the output module 103 may also be used to process the multi-scale image features using a feature pyramid network to obtain a first feature of the gas monitoring device after fusion of the multi-scale features.

[0067] In an embodiment of the present application, the acquisition module 101 may also be used to acquire real-time historical data and / or alarm data and / or trend data of pressure parameters, flow parameters and temperature parameters.

[0068] In an embodiment of the present application, the output module 103 can also be used to process the monitoring data and the first feature using a pre-built multi-source data fusion network to obtain a second feature of the data fusion; and process the second feature using a pre-built classification network to obtain a state recognition result of the gas monitoring equipment, wherein the gas monitoring equipment includes a flow monitor, a valve, a gas safety pipe network controller, a temperature and humidity sensor, a methanol flow meter, a gas pressure regulating valve, and a fire station.

[0069] Based on the above embodiments, the present application also provides a computer-readable storage medium, referring to Figure 3 , the computer-readable storage medium shown is a CD 50, on which a computer algorithm, i.e., an algorithm product, is stored. When the computer algorithm is executed by the processor, each step recorded in the above method implementation is implemented, for example, based on the gas platform, multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area are obtained; the multimodal image data is processed by a pre-constructed feature generation neural network to obtain a first feature of the gas monitoring equipment, wherein the feature generation neural network is composed of a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; the monitoring data and the first feature are processed by a pre-constructed state recognition neural network to obtain a state recognition result of the gas monitoring equipment, and the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence.

[0070] It should be noted that examples of computer-readable storage media may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0071] In addition, based on the above embodiment, the embodiment of the present application further provides a terminal device, Figure 4 A block diagram of an exemplary terminal device 60 suitable for implementing the embodiments of the present application is shown, and the terminal device 60 may be a computer system or a server. Figure 4The terminal device 60 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0072] refer to Figure 4 , the components of the terminal device 60 may include but are not limited to: one or more processors or processing units 601, a system memory 602, and a bus 603 connecting different system components (including the system memory 602 and the processing unit 601).

[0073] The terminal device 60 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the terminal device 60, including volatile and non-volatile media, removable and non-removable media.

[0074] The system memory 602 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 6021 and / or cache memory 6022. The terminal device 60 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the ROM 6023 may be used to read and write non-removable, non-volatile magnetic media ( Figure 4 is not shown in the Figure 4 As shown in the figure, a disk drive for reading and writing a removable non-volatile disk (e.g., a "floppy disk") and an optical disk drive for reading and writing a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) can be provided. In these cases, each drive can be connected to a bus 603 connecting different system components through one or more data medium interfaces. The system memory 602 may include at least one algorithm product, which has a set (e.g., at least one) algorithm module, which is configured to perform the functions of each embodiment of the present application.

[0075] An algorithm / utility 6025 having a set (at least one) of algorithm modules 6024 may be stored, for example, in system memory 602, and such algorithm modules 6024 include, but are not limited to, an operating system, one or more application algorithms, other algorithm modules, and algorithm data, each of which or some combination may include an implementation of a network environment. Algorithm modules 6024 generally perform the functions and / or methods of the embodiments described herein.

[0076] The terminal device 60 may also communicate with one or more external devices 604 (e.g., a keyboard, a pointing device, a display, etc.). Such communication may be performed via an input / output (I / O) interface 605. Furthermore, the terminal device 60 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 606. Figure 4 As shown, the network adapter 606 communicates with other modules (such as the processing unit 601, etc.) of the terminal device 60 via the bus 603 that connects different system components. Figure 4 Not shown, other hardware and / or software program modules may be used in conjunction with terminal device 60 .

[0077] The processing unit 601 executes various functional applications and data processing by running the algorithm stored in the system memory 602, for example, obtaining multimodal image data and monitoring data of the gas monitoring device in the monitoring area based on the gas platform; using a pre-built feature generation neural network to process the multimodal image data to obtain the first feature of the gas monitoring device, wherein the feature generation neural network is based on a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; using a pre-built state recognition neural network to process the monitoring data and the first feature to obtain the state recognition result of the gas monitoring device, the state recognition neural network includes a multimodal data fusion network and a classification network connected in sequence, and the specific implementation method of each step is not repeated here. It should be noted that although several units / modules or sub-units / sub-modules for the gas monitoring image processing device are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be concretized.

[0078] In the description of the present application, it should be noted that the terms "first", "second" and "third" are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0079] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0080] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0081] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0082] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0083] If the function is implemented in the form of a software program functional unit and sold or used as an independent product, it can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software program product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., various media that can store algorithm codes.

[0084] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

[0085] In addition, although the operations of the method of the present application are described in a specific order in the drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps.

[0086] The above are only preferred embodiments of the present application, and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A method for processing gas monitoring images, characterized in that: include: Based on the gas platform, obtain multimodal image data and monitoring data of gas monitoring equipment in the monitoring area; Processing the multimodal image data using a pre-constructed feature generation neural network to obtain a first feature of the gas monitoring device, wherein the feature generation neural network is composed of a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; The monitoring data and the first feature are processed by using a pre-built state recognition neural network to obtain a state recognition result of the gas monitoring device. The state recognition neural network includes a multi-mode data fusion network and a classification network connected in sequence.

2. The method for processing gas monitoring images according to claim 1, characterized in that: Before processing the multimodal image data using the pre-built feature generation neural network, the method for processing gas monitoring images further includes: The original visible light image and the original infrared light image are processed by scaling to obtain a first input image; Normalizing and dimensionally adjusting the first input image to obtain a second input image; The second input image is subjected to standardization processing to obtain the preprocessed multimodal image data.

3. The method for processing gas monitoring images according to claim 1, characterized in that: The method of using a pre-built feature generation neural network to process the multimodal image data to obtain a first feature of the gas monitoring device includes: Processing the multimodal image data using the pre-constructed fusion generative adversarial network to obtain a fusion image containing multi-modal features, wherein the fusion generative adversarial network includes a generator, the generator includes a fully connected module and a convolution module connected in sequence, and the convolution module includes a first batch normalization layer, a first upsampling module, a first one-dimensional convolution layer, a second batch normalization layer, a first activation layer, a second upsampling module, a second one-dimensional convolution layer, a third batch normalization layer, a second activation layer, and a third one-dimensional convolution layer connected in sequence; Processing the fused image using a pre-built feature extraction network to obtain multi-scale image features; The multi-scale image features are processed using a pre-constructed multi-scale feature fusion network to obtain a first feature of the gas monitoring device after multi-scale feature fusion.

4. The method for processing gas monitoring images according to claim 3, characterized in that: The method of processing the fused image using a pre-built feature extraction network to obtain multi-scale image features includes: The fused image is processed by using a first two-dimensional convolutional layer, a second two-dimensional convolutional layer, a pooling layer, a first fully connected layer, and a second fully connected layer that are connected in sequence to obtain multi-scale image features.

5. The method for processing gas monitoring images according to claim 3, characterized in that: The method of using a pre-built multi-scale feature fusion network to process the multi-scale image features to obtain a first feature of the gas monitoring device after multi-scale feature fusion includes: The multi-scale image features are processed using a feature pyramid network to obtain a first feature of the gas monitoring equipment after multi-scale feature fusion.

6. The method for processing gas monitoring images according to claim 1, characterized in that: Acquiring the monitoring data includes: Acquire real-time historical data and / or alarm data and / or trend data of pressure parameters, flow parameters and temperature parameters.

7. The method for processing gas monitoring images according to claim 1, characterized in that: The method of using a pre-built state recognition neural network to process the monitoring data and the first feature to obtain a state recognition result of the gas monitoring device includes: Processing the monitoring data and the first feature using a pre-built multi-source data fusion network to obtain a second feature of data fusion; The second feature is processed using a pre-built classification network to obtain a state recognition result of the gas monitoring equipment, wherein the gas monitoring equipment includes a flow monitor, a valve, a gas safety pipe network controller, a temperature and humidity sensor, a methanol flow meter, a gas pressure regulating valve and a fire station.

8. A gas monitoring image processing device, characterized in that: include: An acquisition module, used to acquire multimodal image data and monitoring data of the gas monitoring equipment in the monitoring area based on the gas platform; A processing module, used to process the multimodal image data using a pre-built feature generation neural network to obtain a first feature of the gas monitoring device, wherein the feature generation neural network is composed of a fusion generation adversarial network, a feature extraction network, and a multi-scale feature fusion network connected in sequence; The output module is used to process the monitoring data and the first feature using a pre-built state recognition neural network to obtain a state recognition result of the gas monitoring equipment. The state recognition neural network includes a multi-mode data fusion network and a classification network connected in sequence.

9. A computer-readable storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method for processing gas monitoring images according to any one of claims 1 to 7 is implemented.

10. A terminal device, comprising: A memory and a processor; characterized in that the processor executes a computer program stored in the memory to implement the method for processing gas monitoring images described in any one of claims 1-7.

Images