Intelligent generation method for urban waterlogging point-surface combined continuous monitoring data based on multi-modal data

Abstract

Disclosed in the present invention is an intelligent generation method for urban waterlogging point-surface combined continuous monitoring data based on multi-modal data. The method comprises the following steps: 1) collecting multi-modal data from a plurality of sources to form a multi-modal data set; 2) intelligently extracting rainstorm flood information on the basis of a deep learning algorithm; 3) matching the rainstorm flood information on a spatio-temporal scale; 4) generating spatio-temporally continuous rainstorm flood data by means of a generative adversarial network algorithm; and 5) using a comprehensive evaluation method for deduplication processing to generate an urban waterlogging point-surface combined continuous monitoring database based on multi-modal data. In the present invention, on the basis of conventional rainstorm flood monitoring data, multi-modal data collected from the plurality of sources is supplemented, rainstorm flood information contained in the multi-modal data is intelligently extracted on the basis of the deep learning algorithm, the urban waterlogging point-surface combined continuous monitoring database based on the multi-modal data is constructed, and the defects that conventional monitoring data is scarce and has fixed sources are overcome.

Description

A method for intelligently generating continuous monitoring data of urban flooding points and areas based on multimodal data. Technical Field

[0001] This invention belongs to the field of urban flood monitoring, specifically relating to an intelligent method for generating continuous monitoring data of urban flooding points and areas based on multimodal data. Background Technology

[0002] Under the combined influence of global climate change and rapid urbanization, the water cycle has undergone drastic changes, extreme weather events have become more frequent and intense, the urban heat island effect and rain island effect have become more prominent, runoff generation and confluence mechanisms have changed, and the mismatch between urban infrastructure construction and urban development has exacerbated the increasingly severe urban flooding situation. Statistics show that in 2022 alone, more than 57 million people worldwide were affected by floods, 8,000 people lost their lives, and losses amounted to US$45 billion.

[0003] Proactive flood control measures typically include temporary measures such as evacuating residents, closing roads, and constructing temporary dikes, as well as making operational decisions regarding the control of critical infrastructure such as dams, weirs, and weirs. However, these measures are implemented based on simulations and predictions of urban flooding using physical or data-driven models. These urban flood models require extensive field data for calibration and validation to ensure the accuracy of the simulations and predictions.

[0004] However, a significant problem exists: limited and overly fixed sources of measured data on urban stormwater and flooding. Traditional urban flood risk monitoring data primarily comes from meteorological and hydrological stations, typically containing only water depth change curves for a small number of locations during extreme rainfall. This data suffers from data lag and is severely spatially limited. Calibrating and validating urban flood models based on limited and spatially discontinuous measured data may lead to model failure and reduce the effectiveness of overall urban flood control measures.

[0005] With the development of deep learning, scholars at home and abroad have begun to try to extract flood-related information from multimodal data. However, the extracted information still lacks continuity in time and space, and cannot meet the needs of urban flood control.

[0006] Summary of the Invention

[0007] The purpose of this invention is to propose an intelligent method for generating continuous urban flooding monitoring data based on multimodal data, combining point and area data, thereby solving the aforementioned problems in existing technologies. This invention aims to extract rainstorm and flood information from multimodal data using intelligent algorithms, combine the extracted point and area information, and generate spatiotemporally continuous rainstorm and flood data based on a generative adversarial network algorithm. After deduplication, a continuous urban flooding monitoring database based on multimodal data is obtained, solving the problems of scarce and fixed-source traditional monitoring data. This method aims to provide flood-prone cities with continuous rainstorm and flood monitoring data to help city managers more accurately and promptly understand and respond to challenges such as urban flooding.

[0008] To achieve the above objectives, the present invention employs the following technical solution:

[0009] A method for intelligently generating continuous monitoring data of urban flooding points and areas based on multimodal data includes the following steps:

[0010] Step S1: Collect multimodal data from multiple sources to form a multimodal dataset. The basic principles for collecting and acquiring multimodal data are openness, legality, and non-confidentiality.

[0011] Step S2: Based on deep learning algorithms, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data;

[0012] Step S3: By dividing the study area and study time period into grids and time periods respectively, the flood information is matched in terms of spatial and temporal scales;

[0013] Step S4: Generate spatiotemporally continuous rainstorm and flood data by combining point and surface data using an adversarial generative network algorithm;

[0014] Step S5: Use a comprehensive evaluation method to remove duplicates and generate a continuous monitoring database of urban waterlogging points and areas based on multimodal data.

[0015] Furthermore, step S1 is for acquiring multimodal data, and step S1 specifically includes:

[0016] S1.1: Data collection based on conventional meteorological and hydrological stations. Hydrological and meteorological data and river water level data of various stations in the study area are obtained from hydrological and meteorological stations. The obtained data includes spatial information of the stations and time series data such as rainfall and water level. The characteristic of this type of data is that the data is continuous in time, but the disadvantage is that it is not continuous in space, and the spatial location is limited by the location of the stations.

[0017] S1.2: Satellite remote sensing data collection. Due to the limitation of the revisit period of a single satellite, a multi-satellite joint encryption method will be adopted when collecting satellite remote sensing data to obtain high-resolution remote sensing images from multiple satellites. Commonly used satellite information is shown in Table 1. The typical characteristic of this type of data is that it has spatial continuity, but the time interval between images is long, and it does not have temporal continuity.

[0018] Table 1 Commonly Used Satellite Information

[0019] S1.3: Social Media Data Collection. This involves using web crawling technology to collect text, images, audio, and video information related to rainstorms and floods from various social media platforms using keywords such as "flood," "flood disaster," "extreme rainfall," and "torrential rain." While this type of data offers advantages such as real-time availability, diverse data types, and visualization, it also suffers from drawbacks including limited quality and accuracy, an abundance of misinformation, a limited user base, and cultural differences.

[0020] S1.4: Camera video image data collection. Collect and store all available camera image data from public locations within the study area. This type of data is characterized by acquiring temporally continuous rainstorm and flood information, and the acquired image data can only yield the required rainstorm and flood information after processing.

[0021] Furthermore, different modal data have different characteristics, therefore it is necessary to intelligently extract flood information from single-modal data based on its data characteristics. Step S2 is to obtain the rainstorm and flood information contained in each modal data from the multimodal data. Step S2 specifically includes:

[0022] S2.1: Extraction of rainstorm and flood information from conventional meteorological and hydrological station data. Missing or outlier data in rainfall and river level data are deleted or padded with zeros, and interpolation is performed to obtain hydrological and meteorological or river level time series data with time intervals of 30 minutes to 1 hour. Rainstorm and flood information extracted from conventional meteorological and hydrological station data has the advantage of temporal continuity.

[0023] S2.2: Extraction of Rainstorm and Flood Information from Satellite Remote Sensing Image Data. Different satellite remote sensing images have different spatial resolutions. Using a 10m resolution as the target, the nearest neighbor sampling method is used, taking the pixel value of the target image relative to the source image's width as the target pixel value, thus unifying the resolution between different images. The remote sensing images are cropped to a uniform size, such as 256*256 pixels. The U-Net++ deep learning algorithm effectively captures local and global features of the images. Data augmentation techniques are used to increase training samples, training the U-Net++ model to identify the flood inundation range and depth in the remote sensing images. Finally, the identified images are stitched together to form a global flood inundation image. Flood information extracted from satellite remote sensing image data has the advantage of spatial continuity.

[0024] S2.3: Extraction of Rainstorm and Flood Information from Social Media Data. Using the Selenium automation tool to simulate logins, text and image data of the flood disaster were obtained. Based on the acquired flood image data, a YOLOv5 convolutional neural network was used to identify key objects and key parts in the flood. The recognition results of the flood text and image information were converted into water level depth, and spatial and temporal information of the occurrence location was extracted based on the text information. Flood information extracted from social media data has advantages such as real-time performance, multi-type data, and large data volume.

[0025] S2.4: Extraction of Rainstorm and Flood Information from Camera Imagery. Frame extraction is employed to obtain image sequence data with fixed time intervals from video footage. UniFormer intelligent models are constructed based on the camera's orientation, shooting position, and information contained in the images to extract water level, inundation depth, and rainfall / flood information from the image sequence data. Rainstorm and flood information extracted from camera imagery has the advantages of diversity and temporal continuity.

[0026] Furthermore, step S3 is to match the extracted rainstorm and flood information data on spatial and temporal scales, thereby achieving a unified representation of rainstorm and flood information extracted from different modalities. Specifically, step S3 involves:

[0027] S3.1: Divide the study area into grids. The grid size can be selected according to the size and characteristics of the study area, such as a 10m*10m rectangular grid. Divide the time into a certain time interval, such as 1 hour.

[0028] S3.2: Use four dimensions x, y, t, z to represent the information extracted from different modal data, where x and y represent spatial location, t represents temporal information, and z represents rainfall, flood depth, or water level information;

[0029] S3.3: For all information extracted from multimodal data, calculate the x and y coordinates according to the spatial location of the information and the grid division; according to the temporal information of the information, divide it according to the time period to obtain the time period t to which it belongs; complete the spatiotemporal matching of information extracted from different modal data.

[0030] Furthermore, step S4 employs a generative adversarial network algorithm, combining rainstorm and flood point information extracted from conventional meteorological and hydrological stations, social media, and camera video images with rainstorm and flood surface information extracted from remote sensing images to generate spatiotemporally continuous urban flood data; the specific method is as follows:

[0031] S4.1: Establish an adversarial generative network model. Collect information on rainstorm and flood points belonging to the same time period t to form an image x (with zero-padding for grids without rainstorm and flood information), such as a 256*256*1 image. Image x is the input of the generator. After passing through the generator G, the complete rainstorm and flood information G(x) for this time period is obtained. The rainstorm and flood surface data extracted from remote sensing images in the same time period are used as real samples y and fed into the discriminator D for judgment. The adversarial generative network model is then trained.

[0032] S4.2: Using a trained adversarial generative network model, images formed by rainstorm and flood point information are input at each time period t to generate rainstorm and flood information that is continuous in both time and space, thus solving the defects of point information lacking spatial continuity and surface information lacking temporal continuity.

[0033] Furthermore, the rainstorm and flood information extracted from different modal data in step S2 differs from that generated based on the adversarial generative network in step S4 in terms of accuracy and effectiveness. Step S5 uses a comprehensive evaluation method to deduplicate similar data belonging to the same spatial grid and the same time period, and generates a continuous monitoring database of urban waterlogging points and areas based on multimodal data. Specifically, step S5 involves:

[0034] S5.1: When a grid contains rainstorm and flood information extracted from multiple modes (step S2 includes four modes, and step S4 includes one mode) at a certain moment, a comprehensive evaluation method is used to deduplicate the rainstorm and flood information. The specific method is shown in the formula:

[0035] In the formula, x and y represent the spatial coordinates of the grid, t represents time, and w i Let z represent the weight of the i-th mode. i This represents the rainfall, inundation depth, or water level information extracted from the i-th mode, and n represents the rainstorm and flood information extracted from the n-mode data at time t.

[0036] S5.2: Construct a continuous monitoring database for urban flooding points and areas based on multimodal data. Similar to step S3.2, this database has four dimensions: x, y, t, and z. Here, x and y represent the spatial location of the grid, t represents the time period, and z represents the rainfall, inundation depth, or water level of the grid. Import the deduplicated rainstorm and flood information into the database.

[0037] Compared with the prior art, the present invention has the following advantages:

[0038] This invention supplements traditional monitoring data with multimodal data collected from multiple sources, and uses deep learning algorithms to intelligently extract rainstorm and flood information contained in the multimodal data, constructing a continuous monitoring database of urban waterlogging points and areas based on multimodal data, thus solving the shortcomings of traditional monitoring data being scarce and from overly fixed sources.

[0039] In the process of intelligently extracting rainstorm and flood information from multimodal data, the rainstorm and flood point information extracted from conventional meteorological and hydrological stations, social media, and camera video images has the characteristics of being continuous in time but discontinuous in space, while the rainstorm and flood surface information extracted from remote sensing images has the characteristics of being continuous in space but discontinuous in time. By combining point and surface information, spatiotemporally continuous urban flood data is generated based on the adversarial generative network algorithm, providing data support for urban flood control. Attached Figure Description

[0040] Figure 1 is a schematic diagram of the process of the present invention.

[0041] Figure 2 shows an example of multimodal data obtained in an embodiment of the present invention.

[0042] Figure 3 shows an example of a continuous monitoring database for urban flooding points and areas based on multimodal data in an embodiment of the present invention. Detailed Implementation

[0043] The specific implementation schemes and their technical effects of the embodiments of this application will be described in detail below with reference to specific application cases.

[0044] Figure 1 is a schematic diagram of the intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data according to the present invention. The method includes the following steps:

[0045] Step S1: Collect multimodal data from multiple sources to obtain a multimodal dataset.

[0046] The dataset was obtained by acquiring routine meteorological and hydrological data from the municipal hydrological station, remote sensing images from the Gaofen-1 satellite from https: / / sasclouds.com / chinese / home / , rainstorm and flood-related data from Weibo using web crawling technology, and video images from cameras from online platforms, forming a multimodal dataset as shown in Figure 2.

[0047] Step S2: Based on deep learning algorithms, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data.

[0048] Conventional meteorological and hydrological station data are processed through zero-padding and interpolation to extract rainstorm and flood information; satellite remote sensing image data is used to extract the flood inundation range and depth based on the U-net++ intelligent model; social media data is identified based on the YOLOv5 intelligent algorithm and then converted into rainstorm and flood information; camera image data is processed by frame extraction and then the UniFormer intelligent model is used to extract rainstorm and flood information.

[0049] Step S3: Match the extracted rainstorm and flood information data at spatial and temporal scales.

[0050] The study area was divided into grids, forming 10m*10m rectangles. Time periods were divided at 30-minute intervals, resulting in 48 time periods per day. The information extracted in step S2 was represented according to the spatial location (x, y) of the information source and the time period t corresponding to the information acquisition time, thus achieving spatiotemporal matching of information.

[0051] Step S4: Generate spatiotemporally continuous urban flood data based on the adversarial generative network model.

[0052] Information on rainstorms and floods within the same time period t is collected to form an image with dimensions of 256*256*1 (zero padding is used for grids without rainstorm and flood information). This image is then used as input to a generator, which generates complete rainstorm and flood information G(x) for this time period. Rainstorm and flood surface information extracted from remote sensing images is used as real samples y. An adversarial generative network intelligent model is constructed, which generates rainstorm and flood information that is continuous in both time and space.

[0053] Step S5: Perform deduplication based on the comprehensive evaluation function to generate a continuous monitoring database of urban waterlogging points and areas based on multimodal data.

[0054] According to the weights in Table 2, the rainstorm and flood information was deduplicated using the comprehensive evaluation method. The data generated by the Generative Adversarial Network (GAN) had the smallest weight w, only 0.4. This is because the data learned by the GAN itself originates from rainstorm and flood surface information extracted from remote sensing images; therefore, the information extracted by other modalities has higher authenticity and effectiveness than the data generated by the GAN. After deduplication, the urban waterlogging point-area combined continuous monitoring database based on multimodal data, as shown in Figure 3, can be obtained. In Figure 3(a), the horizontal axis represents time T, and the vertical axis represents water level, indicating that the database has continuity in terms of time scale. T1, T2, T3, T4, and T5 refer to the rainstorm and flood information at five specific moments. Figure 3(b) shows the rainstorm and flood data at these five moments, indicating that the database also has continuity in terms of spatial scale.

[0055] Table 2 Weights of each mode in the comprehensive evaluation method

Claims

1. A method for intelligently generating continuous monitoring data of urban flooding points and areas based on multimodal data, characterized in that, Includes the following steps: Step S1: Collect multimodal data from multiple sources to form a multimodal dataset; Step S2: Based on deep learning algorithms, intelligent extraction of rainstorm and flood information is performed according to the characteristics of different modal data; Step S3: By dividing the study area and study time period into grids and time periods respectively, the rainstorm and flood information is matched in terms of spatial and temporal scales; Step S4: Using an adversarial generative network model, combined with point-area information of rainstorms and floods, generate spatiotemporally continuous rainstorm and flood data; Step S5: Use a comprehensive evaluation method to deduplicatize the spatiotemporally continuous rainstorm and flood data to generate a continuous monitoring database of urban waterlogging points and areas based on multimodal data.

2. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 1, characterized in that, In step S1, the acquired multimodal data features are different, but they all contain information on rainstorms and floods; the multimodal data includes: 1) conventional meteorological and hydrological station data; 2) satellite remote sensing image data; 3) social media data, including four types of data: text, voice, photos, and videos; 4) camera video image data.

3. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 2, characterized in that, In step S2, the flood information contained in the satellite remote sensing image data is intelligently extracted. The specific method is as follows: first, the resolution between different images is unified by the nearest neighbor sampling method, then the remote sensing images are cropped to a uniform size, and finally the U-Net++ model is used to identify the flood inundation range and inundation depth in the remote sensing images to complete the intelligent extraction of flood information.

4. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 1, characterized in that, The method for intelligently extracting rainstorm and flood information from camera images is as follows: First, image sequence data is obtained through frame extraction. Then, the UniFormer intelligent model is used to extract the sequence values ​​of rainfall, flood depth, and water level changes in the camera images, thus completing the intelligent extraction of rainstorm and flood information.

5. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 1, characterized in that, In step S3, by dividing the study area into grids and time periods, information belonging to the same grid and time period is grouped into the same spatial grid (x,y) and time period t, thereby achieving spatiotemporal matching of information extracted from different modal data.

6. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 1, characterized in that, The training method for the adversarial generative network model described in step S4 includes the following steps: 1) Collecting information on rainstorm and flood points extracted from conventional meteorological and hydrological stations, social media, and camera video images; 2) Synthesize point information belonging to the same time period into an image, and fill in zeros for grids in the image that lack rainstorm and flood information, and use the processed image as the input of generator G; 3) Use rainstorm and flood surface information extracted from satellite remote sensing images as real samples; 4) The output of generator G and the real samples are fed into the discriminator for judgment, and the adversarial generative network model is trained.

7. The intelligent generation method for continuous monitoring data of urban flooding points and areas based on multimodal data as described in claim 1, characterized in that, In step S5, a comprehensive evaluation method is used to deduplicate the spatiotemporally continuous rainstorm and flood data. Specifically, this is achieved using the following formula. In the formula, x and y represent the spatial coordinates of the grid, t represents time, and w i Let z represent the weight of the i-th mode. i This represents the rainfall, inundation depth, or water level information extracted from the i-th modality, and n represents the rainstorm and flood information extracted from the n modality data at time t.

Images