Intelligent generation method of urban waterlogging point and surface monitoring data based on multimodal data

Through the intelligent generation method of multimodal data, deep learning and adversarial generation networks are used to generate spatial and temporal urban flooding data, solving the problems of sparse data and discontinuity in the existing technology, and improving the data support capabilities of urban flood control.

CN118536057BActive Publication Date: 2024-12-03ZHEJIANG UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410547853.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-06
Publication Date
2024-12-03
Estimated Expiration
2044-05-06

AI Technical Summary

Technical Problem

The actual data on existing urban flood monitoring data are sparse and the source is too fixed, resulting in the failure of the model and cannot meet the needs of urban flood control, especially the lack of continuity in time and space.

Method used

Using intelligent generation methods based on multimodal data, through deep learning algorithms and adversarial generation networks, combined with conventional meteorological and hydrological stations, satellite remote sensing, social media and camera video data, continuous rainstorm and flood information in time and space are extracted and generated to build urban flooding. Point-face combined with continuous monitoring database.

Benefits of technology

It provides continuous, accurate and timely urban flooding monitoring data, solves the problems of sparse and fixed sources of traditional monitoring data, and enhances the effectiveness of flood control measures.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118536057B_ABST
    Figure CN118536057B_ABST
Patent Text Reader

Abstract

The present invention discloses a method for intelligently generating urban waterlogging point-surface monitoring data based on multimodal data. The method comprises the following steps: 1) collecting multimodal data from multiple sources to form a multimodal data set; 2) intelligently extracting rainstorm and flood information based on a deep learning algorithm; 3) matching rainstorm and flood information in time and space scales; 4) generating time-space continuous rainstorm and flood data through an adversarial generative network algorithm; 5) performing deduplication processing using a comprehensive evaluation method to generate a point-surface combined continuous monitoring database for urban waterlogging based on multimodal data. Based on traditional rainstorm and flood monitoring data, the present invention supplements multimodal data collected from multiple sources, and intelligently extracts rainstorm and flood information contained in the multimodal data based on a deep learning algorithm, constructing a point-surface combined continuous monitoring database for urban waterlogging based on multimodal data, and solving the defects of the traditional monitoring data being scarce and having fixed sources.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of urban flood monitoring, and in particular relates to an intelligent generation method for urban waterlogging point and surface monitoring data based on multimodal data. Background Art

[0002] Active flood prevention actions usually include temporary measures such as evacuation of residents, road closures and construction of temporary embankments, as well as operational decisions on controlling key infrastructure (such as dams, weirs and barrages). However, these measures are based on the simulation and prediction results of urban flood disasters by physical models or data-driven models. These urban flood models require a large amount of measured data for calibration and verification to ensure the accuracy of simulation predictions.

[0003] However, there is a huge problem at present, that is, there is little measured data on urban rainstorm floods and the data sources are too fixed. Traditional urban flood risk monitoring data mainly comes from meteorological stations and hydrological stations, which generally only contain water depth change curves of a small number of points in extreme rainfall. The observation data has the disadvantage of lag and is greatly limited in space. Calibration and verification of urban flood models based on a small amount of spatially discontinuous measured data may lead to model failure, resulting in a reduction in the effectiveness of flood control measures for the entire city.

[0004] With the development of deep learning, scholars at home and abroad have begun to try to extract flood-related information from multimodal data. However, the extracted information still lacks continuity in time and space and cannot meet the needs of urban flood control. Summary of the invention

[0005] The purpose of the present invention is to propose a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data, thereby solving the aforementioned problems existing in the prior art. The present invention intends to extract rainstorm and flood information from multimodal data through intelligent algorithms, combine the extracted point and surface information, and generate spatiotemporally continuous rainstorm and flood data based on an adversarial generative network algorithm. After deduplication processing, a point-surface combined continuous monitoring database for urban waterlogging based on multimodal data is obtained, solving the problem that traditional monitoring data is scarce and has a fixed source. The method aims to provide continuous rainstorm and flood monitoring data for cities susceptible to waterlogging, so as to help city managers understand and respond to challenges such as urban waterlogging more accurately and timely.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] A method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data comprises the following steps:

[0008] Step S1: Collect multimodal data from multiple sources to form a multimodal data set. The basic principles of multimodal data collection and acquisition are openness, legality, and non-confidentiality;

[0009] Step S2: Based on the deep learning algorithm, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data;

[0010] Step S3: By dividing the study area and study time period into grids and time periods, the matching of flood information in space and time scales is achieved;

[0011] Step S4: Generate temporally and spatially continuous rainstorm and flood data by combining point and surface data through a generative adversarial network algorithm;

[0012] Step S5: Use a comprehensive evaluation method to perform deduplication processing and generate a point-area combined continuous monitoring database for urban waterlogging based on multimodal data.

[0013] Furthermore, the step S1 is to obtain multimodal data, and the step S1 is specifically:

[0014] S1.1: Data collection based on conventional meteorological and hydrological stations. Obtain hydrological and meteorological data and river water level data from each station in the study area from the hydrological and meteorological station. The acquired data contains the spatial information of the station and time series data such as rainfall and water level. The characteristics of this type of data are that the data is continuous in time, but the disadvantage is that it is not continuous in space, and the spatial location is limited by the location of the station;

[0015] S1.2: Satellite remote sensing data collection. Due to the limitation of the revisit period of a single satellite, when collecting satellite remote sensing data, a multi-satellite joint encryption method will be used to obtain high-resolution remote sensing images from multiple satellites. Commonly used satellite information is shown in Table 1. The typical characteristics of this type of data are that it is continuous in space, but the time interval between images is long and not continuous in time;

[0016] Table 1 Common satellite information

[0017]

[0018]

[0019] S1.3: Social media data collection. Through web crawler technology, relevant text, image, voice and video information about rainstorms and floods are obtained from major social media platforms using keywords such as "flood", "flood disaster", "extreme rainfall" and "heavy rain". This type of data has the advantages of real-time, multi-type and visualization, but has defects such as limited quality and accuracy, excessive false information, limited user groups and cultural differences;

[0020] S1.4: Camera video image data collection. Collect and store all available camera image data from public places in the study area. The characteristics of this type of data are that the rainstorm and flood information obtained is image data with time continuity, and the required rainstorm and flood information can only be obtained after the image data is processed;

[0021] Furthermore, the characteristics of different modal data are different, so it is necessary to intelligently extract flood information from single-modal data according to their data characteristics. The step S2 is to obtain the rainstorm flood information contained in each modal data from the multi-modal data. The step S2 is specifically as follows:

[0022] S2.1: Extraction of rainstorm and flood information from conventional meteorological and hydrological station data. Delete or fill in some missing or abnormal value data in rainfall information and river water level data, and perform interpolation processing to obtain hydrological and meteorological or river water level time series data with a time interval of 30min-1h. The rainstorm and flood information extracted based on conventional meteorological and hydrological station data has the advantage of time continuity;

[0023] S2.2: Extraction of rainstorm and flood information from satellite remote sensing image data. Different satellite remote sensing images have different spatial resolutions. Taking 10m resolution as the target, the nearest neighbor sampling method is used to use the ratio of the width of the target image to the width of the source image, and the pixel points at the relative position of the source image are taken as the value of the target pixel points to unify the resolution between different images. The remote sensing images are cropped to a uniform size, such as 256*256 pixels. The U-Net++ deep learning algorithm can effectively capture the local and global features of the image, increase the training samples through data enhancement technology, and train the U-Net++ model to identify the flood inundation range and flooding depth in the remote sensing image. Finally, the identified images are spliced ​​to form a global flood inundation image. The flood information extracted based on satellite remote sensing image data has the advantage of spatial continuity;

[0024] S2.3: Extraction of rainstorm flood information from social media data. Use Selenium automation tools to simulate login to obtain text and picture data when flood disasters occur. Based on the acquired flood picture data, use YOLOv5 convolutional neural network to identify key objects and key parts in the flood, convert the recognition results of flood text information and picture information into water level depth, and extract spatial information and time information of the occurrence location based on text information. Flood information extracted based on social media data has the advantages of real-time, multi-type, and large data volume;

[0025] S2.4: Extraction of rainstorm and flood information from camera images. Using frame extraction processing, we obtain image sequence data with fixed time intervals from video images. According to the camera orientation, shooting position and information contained in the image, we build UniFormer intelligent models to extract water level, flood depth, rainfall, rainstorm and flood information from the image sequence data. The rainstorm and flood information extracted from camera images has the advantages of diversity and temporal continuity.

[0026] Furthermore, the step S3 is to match the extracted rainstorm and flood information data in spatial and temporal scales, so as to achieve a unified representation of rainstorm and flood information extracted in different modes. The step S3 is specifically as follows:

[0027] S3.1: Divide the study area into grids. The grid size can be selected according to the size and characteristics of the study area, such as a 10m*10m rectangular grid. The time is divided into certain time intervals, such as 1h.

[0028] S3.2: Use four dimensions x, y, t, and z to represent the information extracted from different modal data, where x and y represent spatial position, t represents time information, and z represents rainfall, flooding depth, or water level information;

[0029] S3.3: For all information extracted from multimodal data, calculate the x and y coordinates according to the spatial position of the information and divide it into grids; obtain the time period t to which the information belongs based on the time information of the information; complete the spatiotemporal matching of information extracted from different modal data.

[0030] Furthermore, step S4 adopts a generative adversarial network algorithm, combines the rainstorm flood point information extracted from conventional meteorological, hydrological stations, social media and camera video images, and the rainstorm flood surface information extracted from remote sensing images, to generate spatiotemporally continuous urban flood data; the specific method is:

[0031] S4.1: Establish a generative adversarial network model. Collect the information of rainstorm and flood points in the same time period t to form an image x (fill the grids without rainstorm and flood information with zeros), such as a 256*256*1 image. Image x is the input of the generator. After the generator G, the complete rainstorm and flood information G(x) of this time period is obtained. The rainstorm and flood surface data extracted from the remote sensing image in the same time period is used as the real sample y, which enters the discriminator D for judgment and training to obtain the generative adversarial network model.

[0032] S4.2: Use the trained generative adversarial network model to input images formed by rainstorm and flood point information in each time period t, and generate rainstorm and flood information that is continuous in both time and space, solving the defects of point information lacking spatial continuity and surface information being discontinuous in time.

[0033] Furthermore, the rainstorm flood information extracted from different modal data in step S2 and generated based on the adversarial generative network in step S4 are different in authenticity and effectiveness. Step S5 removes duplicates of similar data belonging to the same spatial grid and the same time period according to a comprehensive evaluation method, and generates a point-surface combined continuous monitoring database of urban waterlogging based on multimodal data. Step S5 is specifically as follows:

[0034] S5.1: When a grid has rainstorm and flood information extracted from multiple modes (four modes in step S2 and one mode in step S4) at a certain moment, a comprehensive evaluation method is used to remove the duplicate rainstorm and flood information. The specific method is shown in the formula:

[0035]

[0036] In the formula, x and y represent the spatial coordinates of the grid, t represents the time, and w i represents the weight of the i-th mode, z i represents the rainfall, flooding depth or water level information extracted by the i-th mode, and n represents the grid including the rainstorm and flood information extracted from the n-mode data at time t;

[0037] S5.2: Construct a point-surface continuous monitoring database for urban waterlogging based on multimodal data. The database has four dimensions, x, y, t, and z, the same as in step S3.2, where x and y represent the spatial position of the grid, t represents the time period, and z represents the rainfall, flooding depth or water level of the grid. The deduplicated rainstorm and flood information is imported into the database.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] Based on traditional monitoring data, the present invention supplements multimodal data collected from multiple sources, and intelligently extracts the rainstorm and flood information contained in the multimodal data based on a deep learning algorithm, constructs a point-surface combined continuous monitoring database for urban waterlogging based on multimodal data, and solves the defects of the traditional monitoring data being scarce and the sources being too fixed.

[0040] In the process of intelligent extraction of rainstorm and flood information contained in multimodal data, the rainstorm and flood point information extracted from conventional meteorological, hydrological stations, social media and camera video images has the characteristics of temporal continuity but spatial discontinuity, and the rainstorm and flood surface information extracted from remote sensing images has the characteristics of spatial continuity but temporal discontinuity. Through the point-surface combination, the adversarial generative network algorithm is used to generate spatiotemporal and temporal continuous urban flood data to provide data support for urban flood control. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 It is a schematic diagram of the process of the present invention.

[0042] Figure 2 This is an example of multimodal data obtained in an embodiment of the present invention.

[0043] Figure 3 This is an example of a point-area combined continuous monitoring database for urban waterlogging based on multimodal data in an embodiment of the present invention. DETAILED DESCRIPTION

[0044] The specific implementation scheme and technical effects of the embodiments of the present application will be described in detail below in conjunction with specific application cases.

[0045] Figure 1 The present invention is a flow chart of a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data, the method comprising the following steps:

[0046] Step S1: Collect multimodal data from multiple sources to obtain a multimodal dataset.

[0047] Obtain conventional meteorological and hydrological station data from the municipal hydrological station, obtain Gaofen-1 satellite remote sensing images from https: / / sasclouds.com / chinese / home / , obtain rainstorm and flood related data from Weibo through web crawler technology, and obtain camera video images from online platforms to form a Figure 2 The multimodal dataset shown.

[0048] Step S2: Based on the deep learning algorithm, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data.

[0049] Conventional meteorological and hydrological station data are used to extract rainstorm and flood information through zero-filling and interpolation processing; satellite remote sensing image data are used to extract flood inundation range and depth based on the U-net++ intelligent model; social media data are identified based on the YOLOv5 intelligent algorithm and then converted into rainstorm and flood information; camera image data are processed through frame extraction, and the UniFormer intelligent model is used to extract rainstorm and flood information.

[0050] Step S3: Match the extracted rainstorm and flood information data in spatial and temporal scales.

[0051] Divide the study area into grids and rectangles of 10m*10m; divide the time periods into 30min time intervals, and divide a day into 48 time periods in total. The information extracted in step S2 is represented according to the spatial position (x, y) of the information source and the time period t corresponding to the information acquisition time, so as to achieve spatiotemporal matching of the information.

[0052] Step S4: Generate spatiotemporally continuous urban flood data based on the adversarial generative network model.

[0053] The rainstorm and flood point information belonging to the same time period t is collected to form an image with a dimension of 256*256*1 (zero is filled for grids without rainstorm and flood information), and this image is used as the generator input. After the generator G, the complete rainstorm and flood information G(x) of this time period is obtained. The rainstorm and flood surface information extracted from the remote sensing image is used as the real sample y to construct an adversarial generative network intelligent model, which generates rainstorm and flood information that is continuous in time and space.

[0054] Step S5: De-duplication processing is performed according to the comprehensive evaluation function to generate a point-area combined continuous monitoring database for urban waterlogging based on multimodal data.

[0055] According to the weights in Table 2, the rainstorm and flood information is deduplicated using the comprehensive evaluation method. The data weight w generated by the adversarial generative network is the smallest, which is only 0.4. This is because the data learned by the adversarial generative network itself comes from the rainstorm and flood surface information extracted from remote sensing images. Therefore, the authenticity and effectiveness of the information extracted by other modalities are stronger than the data generated by the adversarial generative network. After deduplication, the following can be obtained: Figure 3 The shown is a point-area combined continuous monitoring database for urban waterlogging based on multimodal data. Figure 3 In (a), the horizontal axis is time T, and the vertical axis is water level, which means that the database has continuity in time scale, where T1, T2, T3, T4, and T5 refer to the rainstorm and flood information at five moments. Figure 3 (b) shows the rainstorm flood data at these five moments, indicating that the database is also continuous in spatial scale.

[0056] Table 2 Weights of each mode in the comprehensive evaluation method

[0057] Data Source Conventional meteorological and hydrological stations Remote sensing images Social Media Camera image Generative Adversarial Networks <![CDATA[Weight w i > 0.90 0.80 0.60 0.75 0.40

Claims

1. A method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data, characterized in that: The steps include: Step S1: Collect multimodal data from multiple sources to form a multimodal dataset; Step S2: Based on the deep learning algorithm, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data; Step S3: by performing grid division and time period division on the study area and the study time period respectively, the matching of the rainstorm and flood information in space and time scales is achieved; Step S4: using a generative adversarial network model and combining the point-surface information of rainstorm floods to generate spatiotemporally continuous rainstorm flood data; Step S5: using a comprehensive evaluation method to perform deduplication processing on the spatiotemporal continuous rainstorm and flood data, and generating a point-surface combined continuous monitoring database for urban waterlogging based on multimodal data; The training method of the adversarial generative network model described in step S4 specifically comprises the following steps: 1) collecting rainstorm flood point information extracted from conventional meteorological and hydrological stations, social media, and camera video images; 2) The point information belonging to the same time period is synthesized into an image, and the grids lacking rainstorm and flood information in the image are filled with zeros, and the processed image is used as the input of the generator G; 3) The rainstorm and flood surface information extracted from satellite remote sensing images is used as the real sample; 4) The output of the generator G and the real sample are sent to the discriminator for judgment, and the adversarial generative network model is obtained through training.

2. According to claim 1, a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data is characterized in that: In step S1, the characteristics of the multimodal data obtained are different, but all contain information about rainstorms and floods; the multimodal data include: 1) conventional meteorological and hydrological station data; 2) satellite remote sensing image data; 3) social media data, including text, voice, photos, and videos; 4) camera video image data.

3. According to claim 2, a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data is characterized in that: In step S2, the flood information contained in the satellite remote sensing image data is intelligently extracted. The specific method is as follows: first, the resolutions of different images are unified through the nearest neighbor sampling method, then the remote sensing images are cropped to a uniform size, and finally, the U-Net++ model is used to identify the flood inundation range and inundation depth in the remote sensing images to complete the intelligent extraction of flood information.

4. According to claim 1, a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data is characterized in that: The rainstorm and flood information in the camera images is intelligently extracted. The specific method is as follows: first, the image sequence data is obtained through frame extraction processing, and then the change sequence values ​​of rainfall, flood depth, and water level in the camera images are extracted through the UniFormer intelligent model to complete the intelligent extraction of rainstorm and flood information.

5. According to claim 1, a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data is characterized in that: In step S3, by dividing the study area into grids and time periods, the information belonging to the same grid and time period is classified into the same spatial grid (x, y) and time period t, so as to achieve spatiotemporal matching of information extracted from different modal data.

6. According to claim 1, a method for intelligently generating urban waterlogging point and surface monitoring data based on multimodal data is characterized in that: In step S5, a comprehensive evaluation method is used to perform deduplication processing on the spatiotemporally continuous rainstorm and flood data, which is specifically implemented by the following formula: In the formula, x and y represent the spatial coordinates of the grid, t represents the time, and w i represents the weight of the i-th mode, z i represents the rainfall, flooding depth or water level information extracted by the i-th mode, and n represents the grid including the rainstorm flood information extracted from the n-mode data at time t.