Method for Identifying Ground Precipitation Phase by Fusing Surveillance Video and Dual-Polarization Radar

Abstract

The present invention provides a method for identifying the precipitation phase of ground precipitation by fusing surveillance videos and dual-polarization radars, including: for a given observation area, identifying the precipitation phase type from the videos captured by ground surveillance cameras; matching the coordinates of the surveillance cameras with the dual-polarization radar image map of the area, and obtaining the pixels at the positions of the cameras one by one; taking the obtained pixels as the vertices of the graph model, establishing the connection rules of the vertices of the graph model with the spatial distance between the cameras as the constraint, and generating the graph model with the dual-polarization radar series echo parameters of the pixels as the vertex feature values; taking the graph model as the input and the precipitation phase type synchronously identified from the videos as the true value label, constructing a graph convolutional neural network for identifying the surface precipitation phase by fusing the surveillance cameras and the dual-polarization radars, and used for identifying the ground precipitation phase. The present invention organically combines the surveillance cameras and the dual-polarization radars to form a new strategy for discriminating the surface precipitation phase by "ground-air" fusion, and improves the accuracy of identifying the ground precipitation phase.

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological monitoring, and particularly to a method for identifying the phase state of surface precipitation by fusing monitoring video and dual-polarization radar. Background Art

[0002] The phase state of surface precipitation has an important impact on industries such as urban transportation, communication, and electric power. Its accurate discrimination is of great significance to people's lives and social development.

[0003] Currently, the main methods for judging precipitation phase states include: ground disdrometers, airborne detection, dual-polarization radar, etc. However, 1) With the rapid development of urbanization, ground precipitation observation equipment in urban areas faces prominent problems such as high construction costs, difficult management and maintenance, and low station network density, resulting in limited spatial representativeness of ground observation results; 2) The airborne detection equipment and data are relatively limited and it is difficult to apply them in real time; 3) The dual-polarization radar can alternately or simultaneously transmit and receive polarized waves in the horizontal and vertical directions, so as to obtain echo information in different directions of the target scatterer to realize the identification of the phase state of cloud particles. However, precipitation particles will experience the combined action of multiple elements in the vertical direction of the atmosphere when falling from high altitude to the ground. Especially in urban areas, the temperature spatial difference is significant, resulting in large differences in the precipitation phase states in different regions. Therefore, there is still a certain difference between the detection of precipitation phase states in clouds by the dual-polarization radar and the ground. In addition, the lack of effective ground particle phase state verification data limits the application of the dual-polarization radar in identifying the phase state of surface precipitation. Summary of the Invention

[0004] In order to solve the above problems, the present invention provides a method for identifying the phase state of surface precipitation by fusing monitoring video and dual-polarization radar, which realizes the discrimination of the phase state of surface precipitation using a monitoring camera and fuses it with the dual-polarization radar to improve the accuracy of identifying the phase state of surface precipitation.

[0005] The present invention provides a method for identifying the phase state of surface precipitation by fusing monitoring video and dual-polarization radar, including:

[0006] For a given observation area, identify the type of precipitation phase state from the video captured by the ground monitoring camera;

[0007] Match the coordinates of the position where the monitoring camera is located with the dual-polarization radar image of the area, and sequentially obtain the pixels at the position where the monitoring camera is located from the dual-polarization radar image;

[0008] Take the pixels at the position where the monitoring camera is located as the vertices of the graph model, establish the connection rules of the vertices of the graph model with the spatial distance between the monitoring cameras as the constraint, and use the series of dual-polarization radar echo parameters of the pixels as the eigenvalue of the vertices to generate a graph model;

[0009] Taking the graph model as the input and the precipitation phase type synchronously recognized from the video as the true value label, a graph convolutional neural network for identifying the surface precipitation phase by fusing a monitoring camera and a dual-polarization radar is constructed for ground precipitation phase identification.

[0010] According to a method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar provided by the present invention, the generating steps of the graph model include:

[0011] Taking the spatial distance between the monitoring cameras as the weight and the topological structure between the monitoring cameras as the constraint, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

[0012] According to a method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar provided by the present invention, the optimizing the link relationship between the vertices corresponding to the monitoring cameras by taking the spatial distance between the monitoring cameras as the weight and the topological structure between the monitoring cameras as the constraint includes:

[0013] Determining the uniformity of the vertex distribution according to the distance between the monitoring cameras;

[0014] Determining the continuity of the vertex distribution according to the maximum and minimum values of the distances between the monitoring cameras;

[0015] Determining the energy function of the graph model according to the uniformity and continuity of the vertex distribution;

[0016] Optimizing the link relationship between the vertices corresponding to the monitoring cameras according to the energy function.

[0017] According to a method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar provided by the present invention, the energy function of the graph model is determined according to the following formula according to the uniformity and continuity of the vertex distribution:

[0018] ;

[0019] where E is the energy function, and are the weight coefficients of vertex uniformity and continuity respectively, is the total number of vertices in the graph model, is the mean value of the distances between vertices, D i is the distance between the i-th vertices, and are the maximum and minimum values of the distances between vertices respectively.

[0020] A method for identifying the precipitation phase state by fusing surveillance video and dual-polarization radar according to the present invention. Identifying the precipitation phase state type from the video captured by the ground surveillance camera includes:

[0021] Extracting spatial features from the video based on the ResNet101 network of the convolutional neural network;

[0022] Extracting temporal features from the video based on the temporal neural network;

[0023] Determining the precipitation phase state type in the video according to the spatial features and the temporal features.

[0024] A method for identifying the precipitation phase state by fusing surveillance video and dual-polarization radar according to the present invention. Identifying the precipitation phase state type from the video captured by the camera in the ground surveillance area includes:

[0025] Extracting spatial features from the video based on the ResNet101 network of the convolutional neural network;

[0026] Taking the spatial features as the input of the temporal neural network to obtain the temporal features output by the temporal neural network;

[0027] Determining the precipitation phase state type in the video according to the temporal features..

[0028] A method for identifying the precipitation phase state by fusing surveillance video and dual-polarization radar according to the present invention further includes:

[0029] Taking the height above the ground of the pixel as the eigenvalue of the vertex.

[0030] The present invention also provides a device for identifying the precipitation phase state by fusing surveillance video and dual-polarization radar, including:

[0031] An identification module for identifying the precipitation phase state type from the video captured by the ground surveillance camera for a given observation area;

[0032] A matching module for performing coordinate matching between the position where the surveillance camera is located and the dual-polarization radar image of the area, and obtaining the pixels at the position where the surveillance camera is located one by one from the dual-polarization radar image;

[0033] A generation module for taking the pixels at the position where the surveillance camera is located as the vertices of the graph model, establishing the connection rules of the vertices of the graph model with the spatial distance between the surveillance cameras as the constraint, and taking the dual-polarization radar series echo parameters of the pixels as the eigenvalues of the vertices to generate a graph model;

[0034] A building block for taking the graph model as an input and the precipitation phase type identified by video synchronization as the ground truth label, and constructing a graph convolutional neural network for identifying the surface precipitation phase by fusing a monitoring camera and a dual-polarization radar for ground precipitation phase identification.

[0035] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, it implements the method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar as described in any one of the above.

[0036] The present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it implements the method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar as described in any one of the above.

[0037] The present invention also provides a computer program product, including a computer program. When the computer program is executed by a processor, it implements the method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar as described in any one of the above.

[0038] The ground precipitation phase identification by fusing a monitoring video and a dual-polarization radar provided by the present invention differentiates the ground precipitation phase in real time and with high resolution by using a monitoring camera, and fuses it with a dual-polarization radar. While providing ground verification for the radar, it assists in correcting the radar inversion results, promotes the complementary advantages of the two observation means, constructs a new method for identifying the ground precipitation phase by combining ground and air, improves the accuracy of ground precipitation phase identification, and provides basic data support for research such as exploring the evolution process of particle falling phases and understanding the laws of urban micro-meteorological changes; by using existing monitoring cameras, there is no need for additional hardware costs, and the cost is relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0040] Figure 1 is a schematic flowchart of the method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar provided by the present invention;

[0041] Figure 2 is a schematic structural diagram of a video-radar fusion precipitation phase identification model based on a graph convolutional network in the method for identifying the ground precipitation phase by fusing a monitoring video and a dual-polarization radar provided by the present invention;

[0042] Figure 3 It is a schematic structural diagram of a graph convolutional neural network in the ground precipitation phase identification method for fusing surveillance video and dual-polarization radar provided by the present invention;

[0043] Figure 4 It is a schematic comparison diagram of surveillance pictures of precipitation particles in different phases in the ground precipitation phase identification method for fusing surveillance video and dual-polarization radar provided by the present invention;

[0044] Figure 5 It is a schematic structural diagram of a deep learning model for identifying the precipitation phase of a surveillance video in the ground precipitation phase identification method for fusing surveillance video and dual-polarization radar provided by the present invention;

[0045] Figure 6 It is a schematic structural diagram of the device for identifying the ground precipitation phase by fusing surveillance video and dual-polarization radar provided by the present invention;

[0046] Figure 7 It is a schematic structural diagram of the electronic device provided by the present invention. Specific Embodiments

[0047] To make the objectives, technical solutions and advantages of the present invention clearer, the technical solutions in the present invention will be clearly and completely described below with reference to the accompanying drawings in the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art without making creative efforts based on the embodiments in the present invention belong to the scope of protection of the present invention.

[0048] The following combines Figure 1 Describe a method for identifying the ground precipitation phase by fusing surveillance video and dual-polarization radar of the present invention, including:

[0049] Step 101, for a given observation area, identify the precipitation phase type from the video captured by the ground surveillance camera;

[0050] Dual-polarization radar can effectively detect the phase of cloud particles, but due to spatio-temporal non-uniformity, it is difficult to reflect the true ground precipitation phase. There is an urgent need to further carry out research on the method for retrieving the ground precipitation phase based on dual-polarization radar.

[0051] Statistical data shows that the number of surveillance cameras is large, and most of them are deployed in urban areas. A large number of surveillance cameras can continuously record the occurrence of ground precipitation and dynamically describe the change of particle phase. If each camera is regarded as a precipitation observation point, its popularization in the urban area can provide a hardware premise for high-spatial-resolution phase observation.

[0052] Meanwhile, the improvement of 4G and 5G technologies in the transmission rate of monitoring signals and the promotion of artificial intelligence algorithms in the rapid interpretation of monitoring data provide a software foundation for the perception of precipitation phase with high temporal resolution. In addition, precipitation observation tasks can be carried out on existing urban monitoring resources, with relatively low deployment and maintenance costs.

[0053] In summary, the precipitation observation network composed of monitoring cameras has the advantages of high density, fast transmission, and sustainability, providing a new opportunity for the high-resolution perception of urban surface precipitation and rich verification data for the identification and interpretation of precipitation phase by dual-polarization radar.

[0054] Step 102: Match the coordinates of the location where the monitoring camera is located with the dual-polarization radar image of the area, and sequentially obtain the pixels at the location where the monitoring camera is located from the dual-polarization radar image.

[0055] Since the radar image is a continuous regular grid, while urban monitoring cameras are randomly distributed, it is impossible to ensure that each radar pixel has corresponding ground precipitation phase true value data. If the traditional method of corresponding radar pixels to monitoring cameras one by one is adopted to construct the radar ground precipitation phase data set, a large number of radar pixels will be wasted due to the lack of true values. Moreover, to ensure that there are sufficient data samples for the training and testing of the subsequent deep learning model, higher requirements are put forward for the quantity and observation period of monitoring cameras.

[0056] Therefore, in this embodiment, a graph convolutional neural network is innovatively used to correspond a group of monitoring cameras to multiple rain gauges. Different from the conventional convolutional neural network, the graph convolutional neural network takes into account the relationship between vertices (or nodes) and neighboring vertices, thus taking into account the part in the middle of the vertices (i.e., those radar pixels without monitoring videos providing true ground phases). Through multi-layer graph convolutional operations, higher-dimensional features can be discovered. Therefore, in this embodiment, while considering the correlation of cross-camera precipitation phase recognition, the randomness of precipitation spatio-temporal changes is taken into account, thereby providing a reliable ground precipitation phase reference for the entire radar image.

[0057] Through the coordinate matching between the camera and the radar image, the radar image pixels with ground precipitation phase true values can be determined, and these pixels are used as the nodes of the graph model.

[0058] Step 103: Take the pixels at the location where the monitoring camera is located as the vertices of the graph model, establish the connection rules of the vertices of the graph model with the spatial distance between the monitoring cameras as the constraint, and use the dual-polarization radar series echo parameters of the pixels as the eigenvalue of the vertices to generate the graph model.

[0059] Take the radar echo parameters that can reflect the precipitation phase as the eigenvalue of the vertices of the graph model. First, six polarization parameters commonly used for precipitation phase identification can be used as eigenvalues, including reflectivity factor, differential reflectivity factor, differential phase constant, co-polar correlation coefficient, structural reflectivity standard deviation, and differential phase standard deviation, etc. In addition, the synchronous precipitation phase type of the surface monitoring video can be added to the eigenvalues to achieve the fusion of the monitoring video and the radar.

[0060] Since the coordinates of the positions where the monitoring cameras are located have been matched with the dual-polarization radar image, the spatial distance between the monitoring cameras is also the distance between the spatial coordinates of the pixels at the positions where the monitoring cameras are located.

[0061] Step 104: Take the graph model as the input and the precipitation phase type identified by the video synchronization as the true value label, and construct a graph convolutional neural network for surface precipitation phase identification by fusing the monitoring camera and the dual-polarization radar, which is used for ground precipitation phase identification.

[0062] Through training, the corresponding relationship between the radar echo parameters and the labels observed by the monitoring video can be determined, and according to this corresponding relationship, the precipitation phase type of the radar pixels can be predicted by the graph convolutional neural network model.

[0063] The graph convolutional neural network extracts the information of each node through multiple layers of convolution, so as to perform the subsequent precipitation phase discrimination task. The graph convolution operation first transforms each vertex feature separately, and then obtains the feature information of adjacent vertices through aggregation. The multiple convolutional layers can receive the vertex features from farther away.

[0064] A graph convolutional neural network with three convolutional layers can be used to convert the radar echo parameter feature vector and the precipitation phase monitoring result of the monitoring video into a graph structure G. After sparse processing, it is input into the graph convolutional neural network to obtain the final precipitation phase type of each pixel.

[0065] In this embodiment, by using the monitoring camera to distinguish the ground precipitation phase in real time and with high resolution and fusing it with the dual-polarization radar, while providing ground verification for the radar, it helps to correct the radar inversion result, promotes the complementary advantages of the two observation methods, constructs a new method for ground-air joint ground precipitation phase identification, improves the accuracy of ground precipitation phase identification, and provides basic data support for research such as exploring the evolution process of particle falling phase and understanding the laws of urban micro-meteorological changes; by using the existing monitoring cameras, there is no need for additional hardware costs, and the cost is relatively low.

[0066] On the basis of the above embodiment, the generation steps of the graph model in this embodiment further include:

[0067] Optimize the link relationship between the vertices corresponding to the monitoring cameras with the spatial distance between the monitoring cameras as the weight and the topological structure between the monitoring cameras as the constraint.

[0068] Graph structure data is generally represented as G=(V, E), where V represents the set of vertices (nodes) of the graph and E represents the set of edges of the graph. The rationality of the graph model is crucial for the effectiveness of the graph convolutional network. Whether the distribution of the vertices of the graph model (radar image pixels corresponding to the monitoring cameras) is too dense or sparse is not conducive to the reasonable description of the spatio-temporal distribution of rainfall. To suppress the problem of the decrease in the accuracy of the graph convolutional neural network caused by the uneven distribution of the vertices of the graph model, use the distance between the cameras as the weight and the topological structure between the monitoring cameras as the constraint, taking into account the uniformity and spatial continuity of the vertex distribution.

[0069] Based on the above embodiments, in this embodiment, the optimization of the link relationship between the vertices corresponding to the monitoring cameras with the spatial distance between the monitoring cameras as the weight and the topological structure between the monitoring cameras as the constraint includes:

[0070] Determine the uniformity of the vertex distribution according to the distance between the monitoring cameras;

[0071] Determine the continuity of the vertex distribution according to the maximum and minimum values of the distances between the monitoring cameras;

[0072] Determine the energy function of the graph model according to the uniformity and continuity of the vertex distribution;

[0073] Optimize the link relationship between the vertices corresponding to the monitoring cameras according to the energy function.

[0074] If the value of the energy function of the graph model is too high, adjust the vertex link relationship between the graph models so that the value of the energy function of the graph model is within a suitable range.

[0075] Introduce the energy function E into the vertex link rule, optimize the structure of the graph model, improve the rationality of the dynamic graph model for describing rainfall events, and the proposed camera-radar mapping model structure is as Figure 2 shown. The graph model is the basis of the graph convolutional neural network, and the setting of the vertex eigenvalue of the graph model and the graph model construction rule are as Figure 2 shown.

[0076] The radar phase state recognition model based on the graph convolutional neural network proposed in this embodiment takes the radar echo parameters as the input, the discrimination result of the monitoring camera as the label, and the spatio-temporal relationship between the camera nodes as the constraint to form the node link rule, realizing the organic integration of the ground-air precipitation observation results, improving the discrimination accuracy of the radar for the precipitation phase state, and further exerting the benefits of the radar.

[0077] Based on the above embodiments, in this embodiment, the energy function of the graph model is determined according to the uniformity and continuity of the vertex distribution through the following formula:

[0078] ;

[0079] where E is the energy function, and are the weight coefficients of vertex uniformity and continuity respectively, is the total number of vertices in the graph model, is the mean value of the distances between vertices, D i is the distance between the i-th vertices, and are the maximum and minimum values among the distances between vertices respectively.

[0080] In this embodiment, a spectral-based graph convolutional network can be adopted to implement feature extraction by using the eigenvalues and eigenvectors of the Laplacian matrix of the graph. The Laplacian matrix L after eigenvalue decomposition is expressed as:

[0081]

[0082] Then the Fourier transform of the graph is:

[0083] ( )

[0084] Thus, according to the convolution theorem, the convolution of the graph data structure is the inverse transform of the product of the Fourier transforms of the two, expressed as:

[0085]

[0086] The graph convolution operation of the graph convolutional network is essentially the result of the inverse Fourier transform of the multiplication of the eigenvector of the vertex and the graph convolution kernel:

[0087]

[0088] where σ represents the activation function, A is the identity matrix, H l represents the eigenvector of the vertex at the l-th layer, H l+1 represents the eigenvector of the vertex after convolution at the l + 1-th layer, and W l represents the parameter of the l-th layer convolution. The eigenvector after the Laplacian matrix decomposition is the basis of the Fourier transform of the graph, and its eigenvalue is the Fourier frequency.

[0089] The structure of the graph convolutional neural network is as Figure 3As shown, where V is the node information structure input to the graph convolutional neural network (GCN), which is determined by the dimensionality of the feature vector of the radar echo, and E is the edge structure that has undergone edge dilation neighborhood and sparsification processing.

[0090] Based on the above embodiments, in this embodiment, identifying the precipitation phase type from the video captured by the ground monitoring camera includes:

[0091] Extracting spatial features from the video based on the ResNet101 network of the convolutional neural network;

[0092] Extracting temporal features from the video based on the temporal neural network;

[0093] Determining the precipitation phase type in the video according to the spatial features and the temporal features.

[0094] A large number of comparative analyses have found that due to differences in the size, shape, and terminal velocity of precipitation particles, etc., the images of precipitation particles in different phases are also different in the monitoring video, which is mainly reflected in the following two aspects:

[0095] In the image domain: Affected by the joint influence of the particle falling velocity and the exposure time of the monitoring camera, the images of particles such as rain, snow, graupel, and hail have large differences in terms of brightness, color, geometric shape, etc. For the convenience of display, Figure 4 The night monitoring pictures of rain, snow, and graupel are compared to reflect the differences of particles in different phases;

[0096] In the time domain: Due to the differences in particle falling velocities, in different modal precipitation events, there are large differences in the distribution density, frequency, falling velocity, etc. of particles in the monitoring scene. A large number of analyses have pointed out that the time-dimensional features of the monitoring video are also an important basis for judging mixed phases such as sleet.

[0097] In this embodiment, a spatial feature extractor based on the ResNet101 network of the convolutional neural network is used to extract precipitation information in the image domain, and a temporal neural network (Recurrent Neural Networks, RNN) is used to establish a temporal feature extractor to mine the differences between temporal particles, so as to accurately discriminate the particle phase.

[0098] Based on the above embodiments, in this embodiment, identifying the precipitation phase type from the video captured by the camera in the ground monitoring area includes:

[0099] Extracting spatial features from the video based on the ResNet101 network of the convolutional neural network;

[0100] Taking the spatial features as the input of the temporal neural network to obtain the temporal features output by the temporal neural network;

[0101] Determine the precipitation phase type in the video according to the timing characteristics.

[0102] The structure of the constructed monitoring video precipitation phase recognition model is as Figure 5 shown. This model is applicable to both daytime and nighttime monitoring videos, with the advantages of all-weather and high precision. For the i-th to (i + n)-th frame images in the monitoring video, each image is used as the input of the ResNet101 network to extract spatial features, and S i to S i+n are obtained respectively. S i to S i+n are respectively input into the temporal neural network, and the temporal features of each S i and S i+1 are jointly used as the input of the temporal neural network to extract temporal features. Finally, the precipitation phase type is recognized according to the temporal features of S i+n .

[0103] In addition, for the training and testing of the deep learning model, a monitoring video precipitation phase dataset is created. The dataset divides the monitoring video into lengths of 5 s and may include one or more of the five phases of rain, snow, graupel, hail, and sleet.

[0104] The monitoring video rainfall phase recognition method proposed in this embodiment realizes the discrimination of phases with high discrimination difficulty such as rain, snow, graupel, and hail through the mining of video spatio-temporal information, and can accurately identify mixed phases such as sleet, effectively improving the reliability of surface precipitation phase discrimination, with the advantages of high precision, all-weather, and strong recognition ability.

[0105] Based on the above embodiment, this embodiment further includes:

[0106] Use the height from the ground of the pixel as the eigenvalue of the vertex.

[0107] Considering that the cloud heights detected by the radar are different, the probability of phase change of cloud particles falling to the ground also varies. Therefore, the height from the ground of each radar pixel is also added to the eigenvalue sequence. Finally, the feature of each vertex of the graph model is an 8×1-dimensional vector.

[0108] The following describes the ground precipitation phase recognition device that fuses the monitoring video and the dual-polarization radar provided by the present invention. The ground precipitation phase recognition device that fuses the monitoring video and the dual-polarization radar described below can be mutually referred to the ground precipitation phase recognition method described above.

[0109] As Figure 6 shown, the device includes an identification module 601, a matching module 602, a generation module 603, and a construction module 604, where:

[0110] The recognition module 601 is used to recognize the precipitation phase type from the video captured by the ground monitoring camera for a given observation area;

[0111] The matching module 602 is used to perform coordinate matching between the position where the monitoring camera is located and the dual-polarization radar image of the area, and sequentially obtain the pixels at the position where the monitoring camera is located from the dual-polarization radar image;

[0112] The generation module 603 is used to use the pixels at the position where the monitoring camera is located as the vertices of the graph model, establish the connection rules of the vertices of the graph model with the spatial distance between the monitoring cameras as the constraint, and use the dual-polarization radar series echo parameters of the pixels as the eigenvalue of the vertices to generate the graph model;

[0113] The construction module 604 is used to use the graph model as the input and the precipitation phase type recognized synchronously from the video as the true value label to construct a convolutional neural network for identifying the surface precipitation phase by fusing the monitoring camera and the dual-polarization radar for ground precipitation phase recognition.

[0114] In this embodiment, by using the ground monitoring camera to distinguish the ground precipitation phase in real time and with high resolution and fusing it with the dual-polarization radar, while providing ground verification for the radar, it helps to correct the radar inversion result, promotes the complementary advantages of the two observation means, constructs a new method for identifying the ground precipitation phase by combining ground and air, improves the accuracy of ground precipitation phase recognition, and provides basic data support for research such as exploring the evolution process of particle falling phase and understanding the laws of urban micro-meteorological changes; by using the existing monitoring camera, there is no need for additional hardware cost and the cost is low.

[0115] Figure 7 An example of the physical structure diagram of an electronic device is shown in Figure 7As shown in the figure, the electronic device may include: a processor 710, a communications interface 720, a memory 730, and a communication bus 740. Among them, the processor 710, the communications interface 720, and the memory 730 complete their mutual communication through the communication bus 740. The processor 710 may call the logical instructions in the memory 730 to execute the method for identifying the ground precipitation phase state by fusing the monitoring video and the dual-polarization radar. The method includes: for a given observation area, identifying the precipitation phase state type from the video captured by the ground monitoring camera; matching the coordinates of the monitoring camera with the dual-polarization radar image map of the area, and obtaining the pixels at the position of the camera one by one; using the obtained pixels as the vertices of the graph model, establishing the connection rules of the vertices of the graph model with the spatial distance between the cameras as the constraint, and generating the graph model with the dual-polarization radar series echo parameters of the pixels as the vertex feature values; using the graph model as the input and the precipitation phase state type synchronously identified in the video as the true value label, constructing a graph convolutional neural network for identifying the surface precipitation phase state by fusing the monitoring camera and the dual-polarization radar, for ground precipitation phase state identification.

[0116] In addition, when the logical instructions in the above-mentioned memory 730 can be implemented in the form of software functional units and sold or used as an independent product, they can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of the present invention. The foregoing storage medium includes: various media such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, or an optical disc that can store program codes.

[0117] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the method for identifying the precipitation phase state by fusing the monitoring video and the dual-polarization radar provided by the above-mentioned various methods. The method includes: for a given observation area, identifying the precipitation phase state type from the video captured by the ground monitoring camera; matching the coordinates of the monitoring camera with the dual-polarization radar image map of the area, and obtaining the pixels at the positions where the camera is located one by one; using the obtained pixels as the vertices of the graph model, establishing the connection rules of the vertices of the graph model with the spatial distance between the cameras as the constraint, and generating the graph model with the series of dual-polarization radar echo parameters of the pixels as the vertex feature values; using the graph model as the input and the precipitation phase state type synchronously identified from the video as the true value label, constructing a graph convolutional neural network for identifying the surface precipitation phase state by fusing the monitoring camera and the dual-polarization radar for ground precipitation phase state identification.

[0118] In another aspect, the present invention also provides a non-transitory computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, it realizes the method for identifying the precipitation phase state by fusing the monitoring video and the dual-polarization radar provided by the above-mentioned various methods. The method includes: for a given observation area, identifying the precipitation phase state type from the video captured by the ground monitoring camera; matching the coordinates of the monitoring camera with the dual-polarization radar image map of the area, and obtaining the pixels at the positions where the camera is located one by one; using the obtained pixels as the vertices of the graph model, establishing the connection rules of the vertices of the graph model with the spatial distance between the cameras as the constraint, and generating the graph model with the series of dual-polarization radar echo parameters of the pixels as the vertex feature values; using the graph model as the input and the precipitation phase state type synchronously identified from the video as the true value label, constructing a graph convolutional neural network for identifying the surface precipitation phase state by fusing the monitoring camera and the dual-polarization radar for ground precipitation phase state identification.

[0119] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed to multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art can understand and implement it without creative labor.

[0120] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on such an understanding, the essence of the above technical solution, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar, characterized in that: include: For a given observation area, identify the precipitation phase type from the video captured by the ground monitoring camera; Matching the coordinates of the location of the monitoring camera with the dual-polarization radar image of the area, and acquiring pixels of the location of the monitoring camera from the dual-polarization radar image one by one; The pixels at the positions of the monitoring cameras are used as vertices of the graph model, the connection rules of the vertices of the graph model are established with the spatial distance between the monitoring cameras as a constraint, and the dual-polarization radar series echo parameters of the pixels are used as the characteristic values ​​of the vertices to generate the graph model; The graph model is used as input, and the precipitation phase type synchronously identified by the video is used as the true value label, to construct a surface precipitation phase recognition graph convolutional neural network that integrates monitoring camera and dual-polarization radar for ground precipitation phase recognition.

2. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1 is characterized in that: The steps of generating the graph model include: The spatial distance between the monitoring cameras is used as a weight, and the topological structure between the monitoring cameras is used as a constraint to optimize the link relationship between the vertices corresponding to the monitoring cameras.

3. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 2 is characterized in that: The optimizing the link relationship between the vertices corresponding to the monitoring cameras by taking the spatial distance between the monitoring cameras as a weight and the topological structure between the monitoring cameras as a constraint includes: Determining the uniformity of the vertex distribution according to the distance between the monitoring cameras; Determining the continuity of the vertex distribution according to the maximum and minimum values ​​of the distances between the monitoring cameras; Determining an energy function of the graph model according to uniformity and continuity of the vertex distribution; According to the energy function, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

4. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 3 is characterized in that: The energy function of the graph model is determined according to the uniformity and continuity of the vertex distribution by the following formula: ； in, E is the energy function, and are the weight coefficients of vertex uniformity and continuity, is the total number of vertices in the graph model, is the mean distance between vertices, D i is the distance between the ith vertices, and are the maximum and minimum distances between vertices, respectively.

5. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1 is characterized in that: The method of identifying the precipitation phase type from the video captured by the ground monitoring camera includes: A ResNet101 network based on a convolutional neural network extracts spatial features from the video; Extracting temporal features from the video based on a temporal neural network; The precipitation phase type in the video is determined according to the spatial feature and the temporal feature.

6. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1, characterized in that: The method of identifying the precipitation phase type from the video captured by the ground monitoring camera includes: A ResNet101 network based on a convolutional neural network extracts spatial features from the video; Using the spatial features as input of a temporal neural network to obtain temporal features output by the temporal neural network; The precipitation phase type in the video is determined according to the time series characteristics.

7. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1, characterized in that: Also includes: The height of the pixel from the ground is used as the feature value of the vertex.

8. A ground precipitation phase recognition device integrating surveillance video and dual polarization radar, characterized in that: include: An identification module is used to identify the precipitation phase type from the video captured by the ground monitoring camera for a given observation area; A matching module, used for matching the coordinates of the position of the monitoring camera with the dual-polarization radar image of the area, and obtaining pixels of the position of the monitoring camera from the dual-polarization radar image one by one; A generation module, used to use the pixels at the positions of the monitoring cameras as vertices of the graph model, establish connection rules for the vertices of the graph model with the spatial distance between the monitoring cameras as constraints, and use the dual-polarization radar series echo parameters of the pixels as the characteristic values ​​of the vertices to generate the graph model; A construction module is used to take the graph model as input, take the precipitation phase type synchronously identified by the video as a true value label, and construct a surface precipitation phase recognition graph convolutional neural network that integrates a surveillance camera and a dual-polarization radar for ground precipitation phase recognition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the ground precipitation phase recognition method of integrating the surveillance video and the dual-polarization radar as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying ground precipitation phase by fusing surveillance video with dual-polarization radar as described in any one of claims 1 to 7 is implemented.

Images