Surveillance video and dual-polarization radar fused ground precipitation phase state identification method

Through the fusion method of monitoring video and dual polarization radar, the graph convolutional neural network is used to optimize the vertex link relationship, and the problem of insufficient accuracy of ground precipitation phase state recognition is solved, and high-resolution and low-cost precipitation phase state recognition is achieved, which promotes the promotion of Joint observation of ground and space.

CN119068427AActive Publication Date: 2024-12-03NANJING UNIV +1
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411570869.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-06
Publication Date
2024-12-03
Estimated Expiration
2044-11-06

AI Technical Summary

Technical Problem

In the prior art, the method of judging the phase state of ground precipitation has high construction costs, difficulty in management and maintenance, low station network density, limited application of airborne detection equipment, and differences in the recognition of phase state of ground precipitation are found, and effective ground particles are lacking. Phase state verification data leads to insufficient accuracy in recognition of precipitation phase states.

Method used

The fusion method of surveillance video and dual polarization radar is adopted to identify the precipitation phase state through the ground monitoring camera. Combined with the dual polarization radar image map, a graph model is established, and a graph convolutional neural network is used for identification, optimizing the vertex link relationship, and improving the accuracy of identification sex.

Benefits of technology

The high-resolution distinction of ground precipitation phase states is achieved complementary to the advantages of dual polarization radar, which improves the accuracy of precipitation phase state recognition, provides ground verification data, reduces hardware costs, and promotes joint ground-space observations.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119068427A_ABST
    Figure CN119068427A_ABST
Patent Text Reader

Abstract

The invention provides a monitoring video and dual-polarization radar fused ground precipitation phase state identification method, which comprises the following steps of: for a given observation area, identifying a precipitation phase state type from a video shot by a ground monitoring camera; the monitoring camera is matched with the dual-polarization radar image coordinates of the area, and pixels of the position where the camera is located are obtained one by one; taking the obtained pixels as vertexes of a graph model, establishing a connection rule of the vertexes of the graph model by taking a spatial distance between cameras as a constraint, and generating the graph model by taking dual-polarization radar series echo parameters of the pixels as vertex feature values; and taking the graph model as input, taking the rainfall phase state type synchronously identified by the video as a truth value label, and constructing a surface rainfall phase state identification graph convolutional neural network fused by the monitoring camera and the dual-polarization radar for surface rainfall phase state identification. According to the invention, the monitoring camera and the dual-polarization radar are organically combined to form a new ground-air fusion ground surface precipitation phase state discrimination strategy, and the accuracy of ground surface precipitation phase state identification is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of meteorological monitoring, and in particular to a ground precipitation phase recognition method by integrating monitoring video with dual-polarization radar. Background Art

[0002] The surface precipitation phase has an important impact on urban transportation, communications, electricity and other industries, and its accurate identification is of great significance to people's lives and social development.

[0003] At present, the main methods for judging precipitation phase state include: ground raindrop spectrometer, airborne detection, dual polarization radar, etc. However, 1) with the rapid development of urbanization, urban ground precipitation observation equipment faces prominent problems such as high construction cost, difficult management and maintenance, and low station network density, resulting in limited spatial representativeness of ground observation results; 2) airborne detection equipment and data are relatively limited, making it difficult to apply in real time; 3) dual polarization radar can transmit alternately or simultaneously, and receive polarized waves in both horizontal and vertical directions, so as to obtain echo information in different directions of the target scatterer to realize the identification of the phase state of particles in the cloud. However, precipitation particles falling from high altitude will experience the combined action of multiple factors in the vertical direction of the complex atmosphere, especially in urban areas, where the temperature space difference is significant, resulting in large differences in precipitation phase states in different regions. Therefore, there is still a certain difference between the precipitation phase state detected by dual polarization radar in the cloud and on the ground. In addition, the lack of effective ground particle phase state verification data limits the application of dual polarization radar to the identification of ground precipitation phase state. Summary of the invention

[0004] In order to solve the above problems, the present invention provides a ground precipitation phase recognition method by integrating monitoring video and dual-polarization radar, which can distinguish the ground precipitation phase by using a monitoring camera and integrate it with the dual-polarization radar to improve the accuracy of ground precipitation phase recognition.

[0005] The present invention provides a ground precipitation phase recognition method by integrating monitoring video and dual-polarization radar, comprising:

[0006] For a given observation area, identify the precipitation phase type from the video captured by the ground monitoring camera;

[0007] Matching the coordinates of the location of the monitoring camera with the dual-polarization radar image of the area, and acquiring pixels of the location of the monitoring camera from the dual-polarization radar image one by one;

[0008] The pixels at the positions of the monitoring cameras are used as vertices of the graph model, the connection rules of the vertices of the graph model are established with the spatial distance between the monitoring cameras as constraints, and the dual-polarization radar series echo parameters of the pixels are used as the characteristic values ​​of the vertices to generate the graph model;

[0009] The graph model is used as input, and the precipitation phase type synchronously identified by the video is used as the true value label, to construct a surface precipitation phase recognition graph convolutional neural network that integrates monitoring camera and dual-polarization radar for ground precipitation phase recognition.

[0010] According to a method for identifying ground precipitation phases by integrating surveillance video and dual-polarization radar provided by the present invention, the step of generating the graphical model includes:

[0011] The spatial distance between the monitoring cameras is used as a weight, and the topological structure between the monitoring cameras is used as a constraint to optimize the link relationship between the vertices corresponding to the monitoring cameras.

[0012] According to a method for identifying ground precipitation phases by integrating surveillance video and dual-polarization radar provided by the present invention, the spatial distance between the surveillance cameras is used as a weight, the topological structure between the surveillance cameras is used as a constraint, and the link relationship between the vertices corresponding to the surveillance cameras is optimized, including:

[0013] Determining the uniformity of the vertex distribution according to the distance between the monitoring cameras;

[0014] Determining the continuity of the vertex distribution according to the maximum and minimum values ​​of the distances between the monitoring cameras;

[0015] Determining an energy function of the graph model according to uniformity and continuity of the vertex distribution;

[0016] According to the energy function, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

[0017] According to a ground precipitation phase recognition method fused with surveillance video and dual-polarization radar provided by the present invention, the energy function of the graph model is determined according to the uniformity and continuity of the vertex distribution by the following formula:

[0018]

[0019] Where E is the energy function, and are the weight coefficients of vertex uniformity and continuity, is the total number of vertices in the graph model, is the mean distance between vertices, D i is the distance between the ith vertices, and are the maximum and minimum distances between vertices, respectively.

[0020] According to a method for identifying ground precipitation phases by integrating surveillance video and dual-polarization radar provided by the present invention, the method for identifying precipitation phase types from videos captured by ground surveillance cameras includes:

[0021] A ResNet101 network based on a convolutional neural network extracts spatial features from the video;

[0022] Extracting temporal features from the video based on a temporal neural network;

[0023] The precipitation phase type in the video is determined according to the spatial feature and the temporal feature.

[0024] According to a method for identifying ground precipitation phases by integrating surveillance video and dual-polarization radar provided by the present invention, the method for identifying precipitation phase types from videos captured by ground surveillance cameras includes:

[0025] A ResNet101 network based on a convolutional neural network extracts spatial features from the video;

[0026] Using the spatial features as input of a temporal neural network to obtain temporal features output by the temporal neural network;

[0027] According to the time series feature, the precipitation phase type in the video is determined.

[0028] A method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar provided by the present invention also includes:

[0029] The height of the pixel from the ground is used as the feature value of the vertex.

[0030] The present invention also provides a ground precipitation phase recognition device integrating monitoring video and dual-polarization radar, comprising:

[0031] An identification module is used to identify the precipitation phase type from the video captured by the ground monitoring camera for a given observation area;

[0032] A matching module, used for matching the coordinates of the position of the monitoring camera with the dual-polarization radar image of the area, and obtaining pixels of the position of the monitoring camera from the dual-polarization radar image one by one;

[0033] A generation module, used to use the pixels at the positions of the monitoring cameras as vertices of the graph model, establish connection rules for the vertices of the graph model with the spatial distance between the monitoring cameras as constraints, and use the dual-polarization radar series echo parameters of the pixels as the characteristic values ​​of the vertices to generate the graph model;

[0034] A construction module is used to take the graph model as input, take the precipitation phase type synchronously identified by the video as a true value label, and construct a surface precipitation phase recognition graph convolutional neural network that integrates a surveillance camera and a dual-polarization radar for ground precipitation phase recognition.

[0035] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a method for identifying ground precipitation phase by fusing surveillance video with a dual-polarization radar as described in any one of the above-mentioned methods is implemented.

[0036] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for identifying the phase of ground precipitation by fusing the surveillance video with the dual-polarization radar as described in any one of the above is implemented.

[0037] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements a ground precipitation phase recognition method by fusing monitoring video with a dual-polarization radar as described in any one of the above.

[0038] The ground precipitation phase state recognition by integrating monitoring video and dual-polarization radar provided by the present invention uses the monitoring camera to distinguish the ground precipitation phase state in real time and with high resolution, and integrates it with the dual-polarization radar to provide ground verification for the radar while assisting in correcting the radar inversion result, promoting the complementary advantages of the two observation means, constructing a new ground-air joint ground precipitation phase state recognition method, improving the accuracy of ground precipitation phase state recognition, and providing basic data support for research on exploring the evolution process of particle landing phase state, understanding the law of urban micro-meteorological changes, etc.; using existing monitoring cameras, no additional hardware cost is required, and the cost is relatively low. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0040] Figure 1 It is a flow chart of the ground precipitation phase recognition method by integrating monitoring video and dual polarization radar provided by the present invention;

[0041] Figure 2 It is a schematic diagram of the structure of a video-radar fusion precipitation phase recognition model based on a graph convolutional network in the ground precipitation phase recognition method of the monitoring video and dual-polarization radar fusion provided by the present invention;

[0042] Figure 3 It is a structural schematic diagram of a graph convolutional neural network in the ground precipitation phase recognition method by integrating monitoring video and dual-polarization radar provided by the present invention;

[0043] Figure 4 It is a schematic diagram of monitoring picture comparison of precipitation particles in different phases in the ground precipitation phase recognition method by integrating monitoring video and dual-polarization radar provided by the present invention;

[0044] Figure 5 It is a schematic diagram of the structure of a deep learning model for monitoring video precipitation phase recognition in the ground precipitation phase recognition method by integrating monitoring video with dual-polarization radar provided by the present invention;

[0045] Figure 6 It is a structural schematic diagram of a ground precipitation phase recognition device fused with monitoring video and dual-polarization radar provided by the present invention;

[0046] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] Combine the following Figure 1 A method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar is described, comprising:

[0049] Step 101, for a given observation area, identifying the precipitation phase type from the video captured by the ground monitoring camera;

[0050] Dual-polarization radar can effectively detect the phase state of particles in clouds, but due to its temporal and spatial heterogeneity, it is difficult to reflect the true precipitation phase state on the ground. There is an urgent need to further study the ground precipitation phase inversion method based on dual-polarization radar.

[0051] Statistics show that there are a large number of surveillance cameras, and most of them are deployed in urban areas. A large number of surveillance cameras can continuously record the occurrence of ground precipitation and dynamically describe the changes in particle phases. If each camera is regarded as a precipitation observation point, its popularity in urban areas can provide the hardware prerequisite for high spatial resolution phase observation.

[0052] At the same time, the improvement of the monitoring signal transmission rate by 4G and 5G technologies and the promotion of rapid interpretation of monitoring data by artificial intelligence algorithms provide a software foundation for high-temporal resolution precipitation phase perception. In addition, precipitation observation tasks can be carried out on existing urban monitoring resources, with low deployment and maintenance costs.

[0053] In summary, the precipitation observation network composed of surveillance cameras has the advantages of high density, fast transmission, and sustainability. It provides a new opportunity for high-resolution perception of urban ground precipitation and also provides rich verification data for dual-polarization radar precipitation phase identification and interpretation.

[0054] Step 102, coordinate matching of the location of the monitoring camera with the dual-polarization radar image of the area, and obtaining pixels of the location of the monitoring camera from the dual-polarization radar image one by one;

[0055] Since radar images are continuous regular grids, and urban surveillance cameras are randomly distributed, it is impossible to guarantee that each radar pixel has corresponding ground precipitation phase truth data. If the traditional method of matching radar pixels with surveillance cameras one by one is adopted to construct the radar ground precipitation phase data set, a large number of radar pixels will be wasted due to the lack of truth values. In addition, in order to ensure that there are enough data samples to provide for the subsequent training and testing of deep learning models, higher requirements are placed on the volume and observation period of surveillance cameras.

[0056] To this end, this embodiment innovatively uses a graph convolutional neural network to correspond the surveillance camera group to multiple rain gauges. Unlike conventional convolutional neural networks, graph convolutional neural networks take into account the relationship between vertices (or nodes) and neighboring vertices, thereby taking into account the part in the middle of the vertex (that is, those radar pixels for which there is no surveillance video to provide the real ground phase state). Through multi-layer graph convolution operations, higher-dimensional features can be discovered. Therefore, while considering the correlation of precipitation phase state recognition across cameras, this embodiment takes into account the randomness of precipitation temporal and spatial changes, thereby providing a reliable ground precipitation phase state reference for the entire radar image.

[0057] By matching the coordinates between the camera and radar images, the radar image pixels with the true value of the ground precipitation phase can be determined and used as nodes of the graph model.

[0058] Step 103, taking the pixels at the positions of the monitoring cameras as the vertices of the graph model, establishing connection rules for the vertices of the graph model with the spatial distance between the monitoring cameras as constraints, and taking the dual-polarization radar series echo parameters of the pixels as the eigenvalues ​​of the vertices to generate the graph model;

[0059] The radar echo parameters that can reflect the precipitation phase are used as the eigenvalues ​​of the graph model vertices. First, the six polarization parameters commonly used in precipitation phase identification can be used as eigenvalues, including reflectivity factor, differential reflectivity factor, differential phase constant, common partial correlation coefficient, structural reflectivity standard deviation, and differential phase standard deviation. In addition, the synchronous precipitation phase type of the surface monitoring video can be added to the eigenvalue to achieve the fusion of monitoring video and radar.

[0060] Since the positions of the monitoring cameras and the dual-polarization radar image have been matched in coordinates, the spatial distance between the monitoring cameras is also the distance between the spatial coordinates of the pixels at the positions of the monitoring cameras.

[0061] Step 104, taking the graph model as input and the precipitation phase type synchronously identified by the video as a true value label, constructing a surface precipitation phase recognition graph convolutional neural network integrating a surveillance camera and a dual-polarization radar for ground precipitation phase recognition.

[0062] Through training, the correspondence between radar echo parameters and labels of surveillance video observations can be determined. Based on this correspondence, the graph convolutional neural network model can predict the precipitation phase type of radar pixels.

[0063] The graph convolutional neural network extracts the information of each node through multiple layers of convolution, so as to perform the subsequent precipitation phase discrimination task. The graph convolution operation first transforms the features of each vertex separately, and then obtains the feature information of adjacent vertices through aggregation. Multiple layers of convolution can receive features from vertices farther away.

[0064] A graph convolutional neural network with three convolutional layers can be used to convert the radar echo parameter feature vectors and the precipitation phase monitoring results of the surveillance video into a graph structure G. After sparse processing, it is input into the graph convolutional neural network to obtain the final precipitation phase type of each pixel.

[0065] This embodiment uses a monitoring camera to distinguish the ground precipitation phase in real time and with high resolution, and integrates it with a dual-polarization radar to provide ground verification for the radar while assisting in correcting the radar inversion result, promoting the complementary advantages of the two observation methods, and constructing a new ground-air joint ground precipitation phase identification method to improve the accuracy of ground precipitation phase identification, and provide basic data support for the study of the evolution process of particle landing phase and the understanding of the law of urban micro-meteorological changes; using existing monitoring cameras, there is no need for additional hardware costs, and the cost is relatively low.

[0066] Based on the above embodiment, the step of generating the graph model in this embodiment further includes:

[0067] Taking the spatial distance between the monitoring cameras as a weight and the topological structure between the monitoring cameras as a constraint, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

[0068] Graph structure data is generally expressed as G=(V, E), where V represents the vertex (node) set of the graph and E represents the edge set of the graph. The rationality of the graph model is crucial to the effectiveness of the graph convolutional network. The distribution of graph model vertices (radar image pixels corresponding to the monitoring camera) is too dense or sparse, which is not conducive to the reasonable description of the spatial and temporal distribution of rainfall. In order to suppress the problem of decreased accuracy of the graph convolutional neural network caused by uneven distribution of graph model vertices, the distance between cameras is used as the weight, and the topological structure between monitoring cameras is used as the constraint, taking into account the uniformity and spatial continuity of vertex distribution.

[0069] On the basis of the above embodiment, the present embodiment uses the spatial distance between the surveillance cameras as a weight and the topological structure between the surveillance cameras as a constraint to optimize the link relationship between the vertices corresponding to the surveillance cameras, including:

[0070] Determining the uniformity of the vertex distribution according to the distance between the monitoring cameras;

[0071] Determining the continuity of the vertex distribution according to the maximum and minimum values ​​of the distances between the monitoring cameras;

[0072] Determining an energy function of the graph model according to uniformity and continuity of the vertex distribution;

[0073] According to the energy function, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

[0074] If the energy function value of the graph model is too high, the vertex link relationship between the graph models is adjusted so that the energy function value of the graph model is within an appropriate range.

[0075] The energy function E is introduced into the vertex linking rule to optimize the structure of the graph model and improve the rationality of the dynamic graph model in describing rainfall events. The structure of the proposed camera-radar mapping model is as follows: Figure 2 The graph model is the basis of the graph convolutional neural network. The setting of the vertex feature values ​​of the graph model and the construction rules of the graph model are as follows: Figure 2 shown.

[0076] The radar phase recognition model based on graph convolutional neural network proposed in this embodiment takes radar echo parameters as input, monitoring camera discrimination results as labels, and the spatiotemporal relationship between camera nodes as constraints to form node linking rules, thereby realizing the organic integration of ground-air precipitation observation results, improving the radar's discrimination accuracy for precipitation phases, and further exerting the benefits of radar.

[0077] Based on the above embodiment, this embodiment determines the energy function of the graph model according to the uniformity and continuity of the vertex distribution through the following formula:

[0078]

[0079] Where E is the energy function, and are the weight coefficients of vertex uniformity and continuity, is the total number of vertices in the graph model, is the mean distance between vertices, D i is the distance between the ith vertices, and are the maximum and minimum distances between vertices, respectively.

[0080] This embodiment can use a spectrum-based graph convolutional network to extract features using the eigenvalues ​​and eigenvectors of the Laplacian matrix of the graph. The Laplacian matrix L after eigendecomposition is expressed as:

[0081]

[0082] in, is a matrix consisting of the eigenvectors of the Laplacian matrix, for The transposed matrix of is a diagonal matrix whose diagonal elements are the corresponding eigenvalues.

[0083] Then the Fourier transform of the graph converts the graph signal Convert to frequency domain Expressed as:

[0084] (The inverse transform is: )

[0085] Therefore, according to the convolution theorem, the convolution of the graph data structure is the inverse transform of the product of the two Fourier transforms, expressed as:

[0086]

[0087] in, and is the graph signal, Convolution for graph data structures.

[0088] The graph convolution operation of the graph convolutional network is essentially the result of the Fourier inverse transform of the multiplication of the vertex feature vector and the graph convolution kernel:

[0089]

[0090] Among them, σ represents the activation function, is the identity matrix, is the degree matrix, H l represents the feature vector of the vertex at the lth layer, H l+1 Represents the feature vector of the vertex in the l+1 layer after convolution, W l Represents the parameters of the lth convolution layer. The eigenvector after Laplace matrix decomposition is the basis of the Fourier transform of the graph, and its eigenvalue is the Fourier frequency.

[0091] The structure of the graph convolutional neural network is as follows Figure 3 As shown in the figure, V is the node information structure of the input graph convolutional neural network GCN, which is determined by the feature vector dimension of the radar echo, and E is the edge structure after edge expansion neighborhood and sparse processing.

[0092] Based on the above embodiment, the method of identifying the precipitation phase type from the video captured by the ground monitoring camera in this embodiment includes:

[0093] A ResNet101 network based on a convolutional neural network extracts spatial features from the video;

[0094] Extracting temporal features from the video based on a temporal neural network;

[0095] The precipitation phase type in the video is determined according to the spatial feature and the temporal feature.

[0096] A large number of comparative analyses have found that due to differences in the size, shape and final velocity of precipitation particles, the images of precipitation particles in different phases in surveillance videos are also different, which is mainly reflected in the following two aspects:

[0097] In the image domain: Due to the combined influence of the particle falling speed and the exposure time of the surveillance camera, the images of rain, snow, sleet and hail particles have great differences in brightness, color, geometry, etc. For ease of display, Figure 4 Comparison of nighttime surveillance images of rain, snow, and sleet shows the differences in particles of different phases;

[0098] In the time domain: Due to the differences in particle falling speeds, there are large differences in the distribution density, frequency, and falling speed of particles in the monitoring scenes in different modes of precipitation events. A large number of analyses have pointed out that the time dimension characteristics of monitoring videos are also an important basis for judging mixed phases such as sleet.

[0099] In this embodiment, a ResNet101 network based on a convolutional neural network is used to build a spatial feature extractor to extract precipitation information in the image domain, and a temporal feature extractor is established using a temporal neural network (RNN) to mine the differences between temporal particles, thereby achieving accurate discrimination of the particle phase state.

[0100] Based on the above embodiment, the method of identifying the precipitation phase type from the video captured by the ground monitoring camera in this embodiment includes:

[0101] A ResNet101 network based on a convolutional neural network extracts spatial features from the video;

[0102] Using the spatial features as input of a temporal neural network to obtain temporal features output by the temporal neural network;

[0103] The precipitation phase type in the video is determined according to the time series characteristics.

[0104] The structure of the monitoring video precipitation phase recognition model is as follows: Figure 5 As shown in Figure 1, the model is applicable to both daytime and nighttime surveillance videos and has the advantages of all-weather and high precision. For the i to i+n frames in the surveillance video, each image is used as the input of the ResNet101 network to extract spatial features and obtain S i To S i+n . i To S i+n Input them into the time series neural network respectively, and each S i The timing characteristics and S i+1 Together they serve as the input of the time series neural network to extract time series features. i+n The precipitation phase type can be identified based on the temporal characteristics of the precipitation.

[0105] In addition, a surveillance video precipitation phase dataset is created for the training and testing of deep learning models. The dataset divides surveillance videos into 5-second segments, which can include one or more of the five phases: rain, snow, sleet, hail, and sleet.

[0106] The method for identifying rainfall phases in surveillance videos proposed in this embodiment, through the mining of video spatiotemporal information, can distinguish phases that are difficult to distinguish, such as rain, snow, sleet, and hail, and can accurately identify mixed phases such as sleet, effectively improving the reliability of surface precipitation phase identification, and has the advantages of high precision, all-weather, and strong recognition capabilities.

[0107] Based on the above embodiment, this embodiment also includes:

[0108] The height of the pixel from the ground is used as the feature value of the vertex.

[0109] Considering that the cloud layer detected by the radar has different heights, the probability of the phase change of the cloud particles falling to the ground is also different. Therefore, the height of each radar pixel from the ground is also added to the eigenvalue sequence. Finally, the feature of each vertex of the graph model is an 8×1 dimensional vector.

[0110] The ground precipitation phase recognition device integrating surveillance video and dual-polarization radar provided by the present invention is described below. The ground precipitation phase recognition device integrating surveillance video and dual-polarization radar described below and the ground precipitation phase recognition method integrating surveillance video and dual-polarization radar described above can be referenced to each other.

[0111] like Figure 6 As shown, the device includes an identification module 601, a matching module 602, a generation module 603 and a construction module 604, wherein:

[0112] The identification module 601 is used to identify the precipitation phase type from the video captured by the ground monitoring camera for a given observation area;

[0113] The matching module 602 is used to match the coordinates of the location of the monitoring camera with the dual-polarization radar image of the area, and obtain pixels of the location of the monitoring camera from the dual-polarization radar image one by one;

[0114] The generation module 603 is used to use the pixels at the positions of the monitoring cameras as the vertices of the graph model, establish the connection rules of the vertices of the graph model with the spatial distance between the monitoring cameras as the constraint, and use the dual-polarization radar series echo parameters of the pixels as the eigenvalues ​​of the vertices to generate the graph model;

[0115] The construction module 604 is used to take the graph model as input, take the precipitation phase type synchronously identified by the video as the true value label, and construct a surface precipitation phase recognition graph convolutional neural network that integrates the monitoring camera and the dual-polarization radar for ground precipitation phase recognition.

[0116] This embodiment uses a monitoring camera to distinguish the ground precipitation phase in real time and with high resolution, and integrates it with a dual-polarization radar to provide ground verification for the radar while assisting in correcting the radar inversion result, promoting the complementary advantages of the two observation methods, and constructing a new ground-air joint ground precipitation phase identification method to improve the accuracy of ground precipitation phase identification, and provide basic data support for the study of the evolution process of particle landing phase and the understanding of the law of urban micro-meteorological changes; using existing monitoring cameras, there is no need for additional hardware costs, and the cost is relatively low.

[0117] Figure 7 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 7As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730 and a communication bus 740, wherein the processor 710, the communication interface 720 and the memory 730 communicate with each other through the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute the ground precipitation phase recognition method fused with the monitoring video and the dual polarization radar, the method comprising: for a given observation area, identifying the precipitation phase type from the video taken by the ground monitoring camera; matching the monitoring camera with the dual polarization radar image map coordinates of the area, and obtaining the pixels at the position of the camera one by one; using the obtained pixels as the vertices of the graph model, establishing the connection rules of the graph model vertices with the spatial distance between the cameras as the constraint, and generating the graph model with the dual polarization radar series echo parameters of the pixels as the vertex feature values; using the graph model as input, using the precipitation phase type synchronously recognized by the video as the true value label, and constructing a surface precipitation phase recognition graph convolutional neural network fused with the monitoring camera and the dual polarization radar for ground precipitation phase recognition.

[0118] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0119] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the ground precipitation phase recognition method provided by the above-mentioned methods that integrates monitoring video and dual-polarization radar. The method includes: for a given observation area, identifying the precipitation phase type from the video taken by a ground monitoring camera; matching the monitoring camera with the coordinates of the dual-polarization radar image map of the area, and obtaining the pixels at the camera position one by one; using the obtained pixels as the vertices of the graph model, establishing the connection rules of the graph model vertices with the spatial distance between the cameras as constraints, and generating the graph model using the dual-polarization radar series echo parameters of the pixels as the vertex eigenvalues; using the graph model as input, and using the precipitation phase type synchronously identified by the video as a true value label, to construct a surface precipitation phase recognition graph convolutional neural network that integrates the monitoring camera and the dual-polarization radar for ground precipitation phase recognition.

[0120] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the ground precipitation phase recognition method provided by the above-mentioned methods by integrating surveillance video and dual-polarization radar, the method comprising: for a given observation area, identifying the precipitation phase type from the video taken by a ground surveillance camera; matching the coordinates of the surveillance camera and the dual-polarization radar image map of the area, and obtaining the pixels of the camera position one by one; using the obtained pixels as the vertices of the graph model, establishing connection rules for the vertices of the graph model with the spatial distance between cameras as constraints, and generating the graph model using the dual-polarization radar series echo parameters of the pixels as vertex eigenvalues; using the graph model as input and the precipitation phase type synchronously identified by the video as a true value label, constructing a surface precipitation phase recognition graph convolutional neural network integrating the surveillance camera and the dual-polarization radar for ground precipitation phase recognition.

[0121] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0122] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0123] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar, characterized in that: include: For a given observation area, identify the precipitation phase type from the video captured by the ground monitoring camera; Matching the coordinates of the location of the monitoring camera with the dual-polarization radar image of the area, and acquiring pixels of the location of the monitoring camera from the dual-polarization radar image one by one; The pixels at the positions of the monitoring cameras are used as vertices of the graph model, the connection rules of the vertices of the graph model are established with the spatial distance between the monitoring cameras as constraints, and the dual-polarization radar series echo parameters of the pixels are used as the characteristic values ​​of the vertices to generate the graph model; The graph model is used as input, and the precipitation phase type synchronously identified by the video is used as the true value label, to construct a surface precipitation phase recognition graph convolutional neural network that integrates monitoring camera and dual-polarization radar for ground precipitation phase recognition.

2. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1 is characterized in that: The steps of generating the graph model include: The spatial distance between the monitoring cameras is used as a weight, and the topological structure between the monitoring cameras is used as a constraint to optimize the link relationship between the vertices corresponding to the monitoring cameras.

3. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 2 is characterized in that: The optimizing the link relationship between the vertices corresponding to the monitoring cameras by taking the spatial distance between the monitoring cameras as a weight and the topological structure between the monitoring cameras as a constraint includes: Determining the uniformity of the vertex distribution according to the distance between the monitoring cameras; Determining the continuity of the vertex distribution according to the maximum and minimum values ​​of the distances between the monitoring cameras; Determining an energy function of the graph model according to uniformity and continuity of the vertex distribution; According to the energy function, the link relationship between the vertices corresponding to the monitoring cameras is optimized.

4. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 3 is characterized in that: The energy function of the graph model is determined according to the uniformity and continuity of the vertex distribution by the following formula: ; Where E is the energy function, and are the weight coefficients of vertex uniformity and continuity, is the total number of vertices in the graph model, is the mean distance between vertices, D i is the distance between the ith vertices, and are the maximum and minimum distances between vertices, respectively.

5. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1 is characterized in that: The method of identifying the precipitation phase type from the video captured by the ground monitoring camera includes: A ResNet101 network based on a convolutional neural network extracts spatial features from the video; Extracting temporal features from the video based on a temporal neural network; The precipitation phase type in the video is determined according to the spatial feature and the temporal feature.

6. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1, characterized in that: The method of identifying the precipitation phase type from the video captured by the ground monitoring camera includes: A ResNet101 network based on a convolutional neural network extracts spatial features from the video; Using the spatial features as input of a temporal neural network to obtain temporal features output by the temporal neural network; The precipitation phase type in the video is determined according to the time series characteristics.

7. The method for identifying ground precipitation phase by integrating surveillance video and dual-polarization radar according to claim 1, characterized in that: Also includes: The height of the pixel from the ground is used as the feature value of the vertex.

8. A ground precipitation phase recognition device integrating surveillance video and dual polarization radar, characterized in that: include: An identification module is used to identify the precipitation phase type from the video captured by the ground monitoring camera for a given observation area; A matching module, used for matching the coordinates of the position of the monitoring camera with the dual-polarization radar image of the area, and obtaining pixels of the position of the monitoring camera from the dual-polarization radar image one by one; A generation module, used to use the pixels at the positions of the monitoring cameras as vertices of the graph model, establish connection rules for the vertices of the graph model with the spatial distance between the monitoring cameras as constraints, and use the dual-polarization radar series echo parameters of the pixels as the characteristic values ​​of the vertices to generate the graph model; A construction module is used to take the graph model as input, take the precipitation phase type synchronously identified by the video as a true value label, and construct a surface precipitation phase recognition graph convolutional neural network that integrates a surveillance camera and a dual-polarization radar for ground precipitation phase recognition.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the ground precipitation phase recognition method of integrating the surveillance video and the dual-polarization radar as described in any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for identifying ground precipitation phase by fusing surveillance video with dual-polarization radar as described in any one of claims 1 to 7 is implemented.

Citation Information

Patent Citations

  • Multi-scale target detection method and system based on roadside camera and millimeter wave radar

    CN118411508A

  • Video radar monitoring method and related equipment

    CN118549923A