Urban water pollution detection method, device and equipment based on multi-modal deep learning and medium

Abstract

The invention discloses an urban water pollution detection method, device and equipment based on multi-modal deep learning and a medium, and the method comprises the steps: constructing topological graph data which takes a pollution monitoring sensor as a node and monitoring data as node data, taking the node as a topology center, processing the graph data through employing a first graph convolutional neural network, obtaining graph structure features, and obtaining a second graph structure feature; and calculating a similarity matrix of each monitored water area image, processing the similarity matrix and the graph structure features by using a second graph convolutional network to obtain pollution features, predicting a pollution detection result of each pollution monitoring sensor based on the pollution features, and obtaining a pollution detection result of each pollution monitoring sensor through bimodal prediction of the monitored water area image and the monitored data. In addition, the graph convolutional neural network is utilized to fully mine the local features of the neighborhood with the node as the topology center and the similarity relation between the nodes to be detected, the influence of other irrelevant features in the monitored water area image on the detection result can be reduced, and the detection precision is improved.

Description

Technical Field

[0001] The present invention relates to smart city technology, and in particular to a method, device, equipment and medium for detecting urban water pollution based on multimodal deep learning. Background Art

[0002] With the advancement of urbanization and the gradual development of the economy, the demand for water resources is also increasing, and the accompanying water pollution caused by industrial waste and residents' domestic garbage and waste has become an increasingly important issue. Especially in the field of residential water use, agricultural irrigation and breeding, and fine chemical industry water use, the requirements for water quality are increasing day by day. It is more and more important to detect the water quality of the water source area and predict water pollution events in a timely manner.

[0003] The existing solution is to place pollution monitoring sensors in the monitored waters and determine whether the monitored waters are polluted based on the concentration of the monitored substances detected by the pollution monitoring sensors. This method is greatly affected by accidental factors, resulting in inaccurate detection results. Summary of the invention

[0004] The present invention provides a method, device, equipment and medium for urban water pollution detection based on multimodal deep learning, so as to reduce the influence of other irrelevant features in the monitored water area image on the detection result and improve the detection accuracy.

[0005] In a first aspect, the present invention provides a method for detecting urban water pollution based on multimodal deep learning, comprising:

[0006] Acquire monitoring water area images of the water area that each pollution monitoring sensor is responsible for and monitoring data collected by the plurality of pollution monitoring sensors;

[0007] Constructing a topological graph data with the pollution monitoring sensors as nodes, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two of the pollution monitoring sensors as the connection lines between the nodes;

[0008] For each of the nodes, taking the node as the topological center, using a first graph convolutional neural network to process the graph data to obtain graph structural features;

[0009] Calculating a similarity matrix of each of the monitored water area images;

[0010] Using a second graph convolutional network to process the similarity matrix and the graph structure features to obtain pollution features;

[0011] A pollution detection result of each of the pollution monitoring sensors is predicted based on the pollution characteristics.

[0012] Optionally, constructing a topological graph data with the pollution monitoring sensor as a node, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two of the pollution monitoring sensors as the connection line between the nodes, including:

[0013] Taking the pollution monitoring sensors as nodes and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection lines between the nodes, a topological graph is constructed;

[0014] The monitoring data collected by the pollution monitoring sensor is assigned to each node in the topological graph to form graph data, which includes a monitoring data matrix composed of the monitoring data of each node and a diffusion matrix composed of the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors.

[0015] Optionally, the first graph convolutional neural network includes a first graph convolutional layer and a second graph convolutional layer. For each of the nodes, the first graph convolutional neural network is used to process the graph data with the node as the topological center to obtain graph structure features, including:

[0016] The first graph convolution layer receives the monitoring data matrix and the diffusion matrix, performs a graph convolution operation on the monitoring data matrix, and updates the monitoring data matrix with a result of the graph convolution operation as a first graph convolution feature;

[0017] The second graph convolution layer receives the first graph convolution feature and the diffusion matrix, performs a graph convolution operation on the first graph convolution feature, and updates the first graph convolution feature with a result of the graph convolution operation as a graph structure feature.

[0018] Optionally, calculating the similarity matrix of each of the monitored water area images includes:

[0019] For each of the monitored water area images, extracting image features representing pollution information of the monitored water area image;

[0020] A similarity matrix representing the similarity relationship between the image features is calculated.

[0021] Optionally, calculating a similarity matrix representing the similarity relationship between the image features includes:

[0022] The cosine similarity between any two of the image features is calculated to obtain a similarity matrix representing the similarity relationship between the image features.

[0023] Optionally, the second graph convolutional neural network includes a third graph convolutional layer, a fourth graph convolutional layer, and a fifth graph convolutional layer, and the similarity matrix and the graph structure features are processed by the second graph convolutional network to obtain pollution features, including:

[0024] Regularizing the similarity matrix to obtain a regularized matrix;

[0025] Inputting the regularization matrix and the graph structure feature into the third graph convolution layer for graph convolution processing to obtain a second graph convolution feature;

[0026] Inputting the regularization matrix and the second graph convolution feature into the fourth graph convolution layer for graph convolution processing to obtain a third graph convolution feature;

[0027] The regularization matrix and the third graph convolution feature are input into the fifth graph convolution layer for graph convolution processing to obtain a pollution feature.

[0028] Optionally, predicting the pollution detection result of each of the pollution monitoring sensors based on the pollution characteristics includes:

[0029] Inputting the pollution feature into a fully connected layer for full connection mapping, mapping the pollution feature to a category space of pollution detection, and obtaining a logical value;

[0030] Inputting the logic value into a classification function for processing to obtain a probability distribution of the category of pollution detection;

[0031] A pollution detection result of each of the pollution monitoring sensors is determined based on the probability distribution.

[0032] In a second aspect, the present invention further provides an urban water pollution detection device based on multimodal deep learning, comprising:

[0033] A data acquisition module, used to acquire the monitoring water area image of the water area responsible for each pollution monitoring sensor and the monitoring data collected by the multiple pollution monitoring sensors;

[0034] A topology data construction module, used to construct topology data with the pollution monitoring sensors as nodes, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection lines between the nodes;

[0035] A first graph convolution processing module is used to process graph data using a first graph convolution neural network for each of the nodes, taking the node as the topological center, to obtain graph structure features;

[0036] A similarity matrix calculation module, used to calculate the similarity matrix of each of the monitored water area images;

[0037] A second graph convolution processing module, used for processing the similarity matrix and the graph structure features by using a second graph convolution network to obtain pollution features;

[0038] The pollution detection result prediction module is used to predict the pollution detection result of each pollution monitoring sensor based on the pollution characteristics.

[0039] In a third aspect, the present invention further provides an electronic device, comprising:

[0040] one or more processors;

[0041] A storage device for storing one or more programs;

[0042] When the one or more programs are executed by the one or more processors, the one or more processors implement the urban water pollution detection method based on multimodal deep learning as provided in the first aspect of the present invention.

[0043] In a fourth aspect, the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the urban water pollution detection method based on multimodal deep learning as provided in the first aspect of the present invention.

[0044] The urban water pollution detection method based on multimodal deep learning provided by the present invention obtains the monitoring water area image of the water area responsible for each pollution monitoring sensor and the monitoring data collected by multiple pollution monitoring sensors, constructs a topological graph data with the pollution monitoring sensor as the node, the monitoring data as the node data, and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection line between the nodes, for each node, with the node as the topological center, using a first graph convolutional neural network to process the graph data to obtain graph structure features, calculates the similarity matrix of each monitoring water area image, uses a second graph convolutional network to process the similarity matrix and the graph structure features to obtain pollution features, and predicts the pollution detection result of each pollution monitoring sensor based on the pollution features. The dual-modal prediction of the monitoring water area image and the monitoring data is beneficial to improving the accuracy of water pollution detection. In addition, the graph convolutional neural network is used to fully explore the local features of the neighborhood with the node as the topological center, and the similarity relationship between the nodes to be detected, which can reduce the influence of other irrelevant features in the monitoring water area image on the detection result and improve the detection accuracy.

[0045] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0047] Figure 1 A flow chart of a method for detecting urban water pollution based on multimodal deep learning provided by the present invention;

[0048] Figure 2 A schematic diagram of the urban water pollution detection process based on multimodal deep learning provided by the present invention;

[0049] Figure 3 A schematic diagram of the structure of an urban water pollution detection device based on multimodal deep learning provided by the present invention;

[0050] Figure 4 The present invention provides a schematic structural diagram of an electronic device.

[0051] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0052] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0053] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0054] Figure 1 This is a flow chart of a method for detecting urban water pollution based on multimodal deep learning provided by the present invention. This embodiment can be applied to urban water pollution detection. The method can be performed by an urban water pollution detection device based on multimodal deep learning provided by an embodiment of the present invention. The device can be implemented by software and / or hardware, and is usually configured in an electronic device, such as Figure 1 As shown in FIG. 1 , the urban water pollution detection method based on multimodal deep learning specifically includes the following steps:

[0055] S101, obtaining a monitoring water area image of a water area that each pollution monitoring sensor is responsible for and monitoring data collected by multiple pollution monitoring sensors.

[0056] In the embodiment of the present invention, the monitored whole water area is distributed with multiple pollution monitoring sensors, and the local water area that each pollution monitoring sensor is responsible for is provided with a monitoring camera, and the pollution monitoring sensor is used to collect pollutant information, such as the type and concentration of pollutants, and the monitoring camera is used to take the monitoring water area image of the local water area. In the embodiment of the present invention, the monitoring water area image of the water area that each pollution monitoring sensor is responsible for and the monitoring data collected by multiple pollution monitoring sensors at the same time are obtained.

[0057] S102, constructing a topological graph data with pollution monitoring sensors as nodes, monitoring data as node data, and pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as connection line data between nodes.

[0058] In an embodiment of the present invention, a topological graph is constructed with pollution monitoring sensors as nodes and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection between the nodes. The monitoring data collected by the pollution monitoring sensors are assigned to each node in the topological graph to form graph data, which includes a monitoring data matrix composed of the monitoring data of each node and a diffusion matrix composed of the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors, wherein the diffusion speed is related to the flow rate of water and the diffusion rate of the pollutant itself. The graph data of the present invention integrates the pollutant diffusion speed and diffusion radius between pollution monitoring sensors, that is, takes into account the influence of the monitoring data of the adjacent monitoring points of the monitoring point, which is conducive to improving the accuracy of pollution prediction.

[0059] S103. For each node, take the node as the topological center and use the first graph convolutional neural network to process the graph data to obtain graph structure features.

[0060] Figure 2 The schematic diagram of the urban water pollution detection process based on multimodal deep learning provided by the present invention is as follows: Figure 2As shown, in the embodiment of the present invention, for each node, the node is taken as the topological center, and the first graph convolutional neural network is used to process the graph data to obtain the graph structure characteristics. The present invention uses the graph convolutional neural network to fully explore the local characteristics of the neighborhood with the node as the topological center, which is conducive to improving the accuracy of pollution prediction.

[0061] In some embodiments of the present invention, Figure 2 As shown, the first graph convolutional neural network GCN1 includes a first graph convolutional layer GCC1 and a second graph convolutional layer GCC2. The first graph convolutional neural network GCN1 processes graph data as follows:

[0062] 1. The first graph convolution layer GCC1 receives the monitoring data matrix E and the diffusion matrix T, performs a graph convolution operation on the monitoring data matrix E, and uses the result of the graph convolution operation to update the monitoring data matrix E as the first graph convolution feature E1.

[0063] Exemplarily, the diffusion matrix T is regularized to obtain the regularized matrix Then the graph convolution is performed using the following formula:

[0064]

[0065] Wherein, σ is an activation function, W1 is a graph convolution parameter of the first graph convolution layer GCC1, and illustratively, in an embodiment of the present invention, σ selects the ReLU function as the activation function.

[0066] 2. The second graph convolution layer GCC2 receives the first graph convolution feature E1 and the diffusion matrix T, performs a graph convolution operation on the first graph convolution feature E1, and uses the result of the graph convolution operation to update the first graph convolution feature E1 as the graph structure feature E2.

[0067] Exemplarily, the diffusion matrix T is regularized to obtain the regularized matrix Then the graph convolution is performed using the following formula:

[0068]

[0069] Wherein, σ is an activation function, W2 is a graph convolution parameter of the second graph convolution layer GCC2, and illustratively, in an embodiment of the present invention, σ selects the ReLU function as the activation function.

[0070] S104, calculating the similarity matrix of each monitored water area image.

[0071] In the embodiment of the present invention, the similarity matrix A of each monitored water area image is calculated. The similarity matrix A is used to characterize the similarity relationship between potential pollution in each monitored water area image.

[0072] In the embodiment of the present invention, the similarity between the monitored water area images can be characterized by calculating the Euclidean distance, Mahalanobis distance, cosine similarity or Cosin distance between the monitored water area images, which is not limited in the embodiment of the present invention.

[0073] Exemplarily, in some embodiments of the present invention, for each monitored water image, image features representing the pollution information of the monitored water image are extracted. Exemplarily, for each monitored water image, it is input into a pre-trained feature extraction network (Feature Extraction Network, FEN) for processing to obtain image features representing the pollution information of the monitored water image. Exemplarily, the feature extraction network may include multiple convolutional layers and pooling layers, which are not limited in the embodiments of the present invention. Then, a similarity matrix A representing the similarity relationship between each image feature is calculated. Exemplarily, the cosine similarity between any two image features is calculated to obtain a similarity matrix A representing the similarity relationship between each image feature.

[0074] S105. Use a second graph convolutional network to process the similarity matrix and graph structure features to obtain pollution features.

[0075] In an embodiment of the present invention, for each node, a second graph convolutional network is used to process the similarity matrix A and the graph structure feature E2 corresponding to the node to obtain the pollution feature. The graph convolutional neural network is used to fully explore the similarity relationship between the nodes to be detected, which can reduce the influence of other irrelevant features in the monitored water area image on the detection results and improve the detection accuracy.

[0076] Exemplarily, in some embodiments of the present invention, the second graph convolutional neural network GCN2 includes a third graph convolutional layer GCC3, a fourth graph convolutional layer GCC4, and a fifth graph convolutional layer GCC5, and the processing process of the second graph convolutional neural network GCN2 is as follows:

[0077] 1. Regularize the similarity matrix A to obtain the regularized matrix

[0078] In the embodiment of the present invention, the similarity matrix A is regularized to obtain a regularized matrix

[0079] 2. Regularize the matrix The graph structure feature E2 is input into the third graph convolution layer GCC3 for graph convolution processing to obtain the second graph convolution feature H1.

[0080] Exemplarily, the data expression of the convolution operation of the third graph convolution layer GCC3 is:

[0081]

[0082] Wherein, σ is an activation function, W3 is a graph convolution parameter of the third graph convolution layer GCC3, and illustratively, in an embodiment of the present invention, σ selects the ReLU function as the activation function.

[0083] 3. Regularize the matrix The second graph convolution feature H1 is input into the fourth graph convolution layer GCC4 for graph convolution processing to obtain the third graph convolution feature H2.

[0084] Exemplarily, the data expression of the convolution operation of the fourth graph convolution layer GCC4 is:

[0085]

[0086] Wherein, σ is an activation function, W4 is a graph convolution parameter of the fourth graph convolution layer GCC4, and illustratively, in an embodiment of the present invention, σ selects the ReLU function as the activation function.

[0087] 4. Regularize the matrix The third graph convolution feature H2 is input into the fifth graph convolution layer GCC5 for graph convolution processing to obtain the pollution feature H3.

[0088] Exemplarily, the data expression of the convolution operation of the fifth graph convolution layer GCC5 is:

[0089]

[0090] Wherein, σ is an activation function, W5 is a graph convolution parameter of the fifth graph convolution layer GCC5, and illustratively, in an embodiment of the present invention, σ selects the ReLU function as the activation function.

[0091] S106. Predict the pollution detection result of each pollution monitoring sensor based on the pollution characteristics.

[0092] In an embodiment of the present invention, the pollution detection result of each pollution monitoring sensor is predicted based on the pollution feature corresponding to each node. Exemplarily, the pollution feature can be input into a pre-trained classifier, and the pollution feature can be mapped to the category space of pollution detection through the classifier, and then converted into the probability distribution of the category of pollution detection, and the pollution detection result of each pollution monitoring sensor is determined based on the probability distribution.

[0093] Exemplarily, in an embodiment of the present invention, the pollution feature H3 is input into the fully connected layer FCC for full connection mapping, the pollution feature is mapped to the category space of pollution detection, and the logical value logits is obtained. The logical value is input into the classification function for processing to obtain the probability distribution P of the category of pollution detection, and the pollution detection result of each pollution monitoring sensor is obtained based on the probability distribution P. Exemplarily, the classification function can be sigmoid. In machine learning, especially deep learning, the sigmoid function is a very common and relatively important function, especially widely used in binary classification scenarios. In an embodiment of the present invention, the sigmoid function maps the input logical value logits to a probability value between 0 and 1 and outputs it. By setting an appropriate threshold t, when the probability value output by the sigmoid function is greater than the threshold t, it is considered that the pollution detection result of the pollution monitoring sensor is yes, that is, there is water pollution in the local waters that the pollution monitoring sensor is responsible for.

[0094] The urban water pollution detection method based on multimodal deep learning provided by the present invention obtains the monitoring water area image of the water area responsible for each pollution monitoring sensor and the monitoring data collected by multiple pollution monitoring sensors, constructs a topological graph data with the pollution monitoring sensor as the node, the monitoring data as the node data, and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection line between the nodes, for each node, with the node as the topological center, using a first graph convolutional neural network to process the graph data to obtain graph structure features, calculates the similarity matrix of each monitoring water area image, uses a second graph convolutional network to process the similarity matrix and the graph structure features to obtain pollution features, and predicts the pollution detection result of each pollution monitoring sensor based on the pollution features. The dual-modal prediction of the monitoring water area image and the monitoring data is beneficial to improving the accuracy of water pollution detection. In addition, the graph convolutional neural network is used to fully explore the local features of the neighborhood with the node as the topological center, and the similarity relationship between the nodes to be detected, which can reduce the influence of other irrelevant features in the monitoring water area image on the detection result and improve the detection accuracy.

[0095] Figure 3 A schematic diagram of the structure of a multimodal deep learning-based urban water pollution detection device provided by the present invention is shown in FIG. Figure 3 As shown, the urban water pollution detection device based on multimodal deep learning includes:

[0096] The data acquisition module 201 is used to acquire the monitoring water area image of the water area that each pollution monitoring sensor is responsible for and the monitoring data collected by the multiple pollution monitoring sensors;

[0097] A topology data construction module 202 is used to construct topology data with the pollution monitoring sensors as nodes, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection lines between the nodes;

[0098] A first graph convolution processing module 203 is used to process graph data using a first graph convolution neural network for each of the nodes, taking the node as the topological center, to obtain graph structural features;

[0099] A similarity matrix calculation module 204 is used to calculate the similarity matrix of each of the monitored water area images;

[0100] A second graph convolution processing module 205, used to process the similarity matrix and the graph structure features using a second graph convolution network to obtain pollution features;

[0101] The pollution detection result prediction module 206 is used to predict the pollution detection result of each of the pollution monitoring sensors based on the pollution characteristics.

[0102] In some embodiments of the present invention, the topology map data construction module 202 includes:

[0103] A topology map construction submodule, used to construct a topology map with the pollution monitoring sensors as nodes and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection between the nodes;

[0104] The graph data forming submodule is used to assign the monitoring data collected by the pollution monitoring sensor to each node in the topological graph to form graph data, wherein the graph data includes a monitoring data matrix composed of the monitoring data of each node and a diffusion matrix composed of the pollutant diffusion speed and diffusion radius between any two of the pollution monitoring sensors.

[0105] In some embodiments of the present invention, the first graph convolutional neural network includes a first graph convolutional layer and a second graph convolutional layer, and the first graph convolution processing module 203 includes:

[0106] A first graph convolution submodule, configured to receive the monitoring data matrix and the diffusion matrix at the first graph convolution layer, perform a graph convolution operation on the monitoring data matrix, and update the monitoring data matrix with a result of the graph convolution operation as a first graph convolution feature;

[0107] The second graph convolution submodule is used for the second graph convolution layer to receive the first graph convolution feature and the diffusion matrix, perform a graph convolution operation on the first graph convolution feature, and update the first graph convolution feature with the result of the graph convolution operation as a graph structure feature.

[0108] In some embodiments of the present invention, the similarity matrix calculation module 204 includes:

[0109] An image feature extraction submodule, for extracting image features representing pollution information of each monitored water area image;

[0110] The similarity matrix calculation submodule is used to calculate a similarity matrix representing the similarity relationship between the image features.

[0111] In some embodiments of the present invention, the similarity matrix calculation submodule includes:

[0112] The similarity calculation unit is used to calculate the cosine similarity between any two of the image features to obtain a similarity matrix representing the similarity relationship between the image features.

[0113] In some embodiments of the present invention, the second graph convolutional neural network includes a third graph convolutional layer, a fourth graph convolutional layer and a fifth graph convolutional layer, and the second graph convolution processing module 205 includes:

[0114] A regularization processing submodule, used for performing regularization processing on the similarity matrix to obtain a regularized matrix;

[0115] A third graph convolution submodule, used for inputting the regularization matrix and the graph structure feature into the third graph convolution layer for graph convolution processing to obtain a second graph convolution feature;

[0116] A fourth graph convolution submodule, used for inputting the regularization matrix and the second graph convolution feature into the fourth graph convolution layer for graph convolution processing to obtain a third graph convolution feature;

[0117] The fifth graph convolution submodule is used to input the regularization matrix and the third graph convolution feature into the fifth graph convolution layer for graph convolution processing to obtain pollution features.

[0118] In some embodiments of the present invention, the pollution detection result prediction module 206 includes:

[0119] A fully connected mapping submodule, used for inputting the pollution feature into the fully connected layer for full connection mapping, mapping the pollution feature to the category space of pollution detection, and obtaining a logical value;

[0120] A classification submodule, used for inputting the logic value into a classification function for processing to obtain a probability distribution of pollution detection categories;

[0121] The pollution detection result determination submodule is used to determine the pollution detection result of each pollution monitoring sensor based on the probability distribution.

[0122] The above-mentioned urban water pollution detection device based on multimodal deep learning can execute the urban water pollution detection method based on multimodal deep learning provided by the aforementioned embodiments of the present invention, and has the corresponding functional modules and beneficial effects for executing the urban water pollution detection method based on multimodal deep learning.

[0123] Figure 4 A schematic diagram of the structure of an electronic device provided by the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0124] like Figure 4 As shown, the electronic device includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0125] A number of components in the electronic device are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 11 executes the various methods and processes described above, such as the urban water pollution detection method based on multimodal deep learning.

[0127] In some embodiments, the urban water pollution detection method based on multimodal deep learning can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the urban water pollution detection method based on multimodal deep learning described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the urban water pollution detection method based on multimodal deep learning in any other appropriate manner (e.g., by means of firmware).

[0128] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0130] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0131] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0133] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0134] An embodiment of the present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the urban water pollution detection method based on multimodal deep learning as provided in any embodiment of the present application.

[0135] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code of the present invention, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (for example, using an Internet service provider to connect through the Internet).

[0136] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0137] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for detecting urban water pollution based on multimodal deep learning, characterized in that: include: Acquire monitoring water area images of the water area that each pollution monitoring sensor is responsible for and monitoring data collected by the plurality of pollution monitoring sensors; Constructing a topological graph data with the pollution monitoring sensors as nodes, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two of the pollution monitoring sensors as the connection lines between the nodes; For each of the nodes, taking the node as the topological center, using a first graph convolutional neural network to process the graph data to obtain graph structural features; Calculating a similarity matrix of each of the monitored water area images; Using a second graph convolutional network to process the similarity matrix and the graph structure features to obtain pollution features; A pollution detection result of each of the pollution monitoring sensors is predicted based on the pollution characteristics.

2. The urban water pollution detection method based on multimodal deep learning according to claim 1 is characterized in that: Constructing a topological graph data with the pollution monitoring sensor as a node, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two of the pollution monitoring sensors as the connection line between the nodes, including: Taking the pollution monitoring sensors as nodes and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection lines between the nodes, a topological graph is constructed; The monitoring data collected by the pollution monitoring sensor is assigned to each node in the topological graph to form graph data, which includes a monitoring data matrix composed of the monitoring data of each node and a diffusion matrix composed of the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors.

3. The urban water pollution detection method based on multimodal deep learning according to claim 2 is characterized in that: The first graph convolutional neural network includes a first graph convolutional layer and a second graph convolutional layer. For each of the nodes, the first graph convolutional neural network is used to process the graph data with the node as the topological center to obtain graph structure features, including: The first graph convolution layer receives the monitoring data matrix and the diffusion matrix, performs a graph convolution operation on the monitoring data matrix, and updates the monitoring data matrix with a result of the graph convolution operation as a first graph convolution feature; The second graph convolution layer receives the first graph convolution feature and the diffusion matrix, performs a graph convolution operation on the first graph convolution feature, and updates the first graph convolution feature with a result of the graph convolution operation as a graph structure feature.

4. The urban water pollution detection method based on multimodal deep learning according to any one of claims 1 to 3, characterized in that: Calculating the similarity matrix of each of the monitored water area images, including: For each of the monitored water area images, extracting image features representing pollution information of the monitored water area image; A similarity matrix representing the similarity relationship between the image features is calculated.

5. The urban water pollution detection method based on multimodal deep learning according to claim 4 is characterized in that: Calculating a similarity matrix representing the similarity relationship between the image features, including: The cosine similarity between any two of the image features is calculated to obtain a similarity matrix representing the similarity relationship between the image features.

6. The urban water pollution detection method based on multimodal deep learning according to any one of claims 1 to 3, characterized in that: The second graph convolutional neural network includes a third graph convolutional layer, a fourth graph convolutional layer and a fifth graph convolutional layer. The similarity matrix and the graph structure features are processed by the second graph convolutional network to obtain pollution features, including: Regularizing the similarity matrix to obtain a regularized matrix; Inputting the regularization matrix and the graph structure feature into the third graph convolution layer for graph convolution processing to obtain a second graph convolution feature; Inputting the regularization matrix and the second graph convolution feature into the fourth graph convolution layer for graph convolution processing to obtain a third graph convolution feature; The regularization matrix and the third graph convolution feature are input into the fifth graph convolution layer for graph convolution processing to obtain a pollution feature.

7. The urban water pollution detection method based on multimodal deep learning according to any one of claims 1 to 3, characterized in that: Predicting the pollution detection result of each of the pollution monitoring sensors based on the pollution characteristics includes: Inputting the pollution feature into a fully connected layer for full connection mapping, mapping the pollution feature to a category space of pollution detection, and obtaining a logical value; Inputting the logic value into a classification function for processing to obtain a probability distribution of the category of pollution detection; A pollution detection result of each of the pollution monitoring sensors is determined based on the probability distribution.

8. An urban water pollution detection device based on multimodal deep learning, characterized in that: include: A data acquisition module, used to acquire the monitoring water area image of the water area responsible for each pollution monitoring sensor and the monitoring data collected by the multiple pollution monitoring sensors; A topology data construction module, used to construct topology data with the pollution monitoring sensors as nodes, the monitoring data as node data, and the pollutant diffusion speed and diffusion radius between any two pollution monitoring sensors as the connection lines between the nodes; A first graph convolution processing module is used to process graph data using a first graph convolution neural network for each of the nodes, taking the node as the topological center, to obtain graph structure features; A similarity matrix calculation module, used to calculate the similarity matrix of each of the monitored water area images; A second graph convolution processing module, used for processing the similarity matrix and the graph structure features by using a second graph convolution network to obtain pollution features; The pollution detection result prediction module is used to predict the pollution detection result of each pollution monitoring sensor based on the pollution characteristics.

9. An electronic device, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the urban water pollution detection method based on multimodal deep learning as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the urban water pollution detection method based on multimodal deep learning as described in any one of claims 1 to 7 is implemented.

Images