Geological disaster monitoring method and system based on deep learning and spatial data query

Abstract

The present invention relates to a geological disaster monitoring method and system based on deep learning and spatial data query, wherein the detection method comprises building a geological disaster positioning and rating model based on a twin neural network; obtaining pre-disaster remote sensing image data of a disaster-stricken area; obtaining post-disaster remote sensing image data of the disaster-stricken area from a spatial database based on a materialized view query method, a TBDR-Tree indexing method, and a graphic contour segmentation method; inputting the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model to identify the location and disaster level of the disaster-stricken area, and correspondingly obtaining disaster location information and disaster level information; and outputting the disaster location information and the disaster level information. The present invention greatly optimizes the query time, reduces the memory overhead, shortens the running time, and realizes the functions of efficient and accurate disaster detection, disaster area identification, and disaster level classification for remote sensing images.

Description

Technical Field

[0001] The present invention relates to the field of deep learning and spatial database optimization query, and in particular to a geological disaster monitoring method and system based on deep learning and spatial data query. Background Art

[0002] Remote sensing data is digital data obtained by sampling and quantifying electromagnetic waves from the surface of the earth using remote sensors. Analyzing remote sensing data can help us understand the use of land resources and land cover types, and is an important part of the national geographic survey and monitoring. By efficiently obtaining accurate and objective land use conditions and monitoring land changes, we can timely understand the deformation dynamics of disaster bodies, analyze their stability, make predictions in advance, and prevent disasters. In particular, it has played a great role in the field of geological disaster monitoring. When faced with landslides and mudslides, the system analyzes and compares monitoring data and historical data to obtain early warning information, provide decision-making basis, protect people's lives and property, and reduce casualties and asset losses.

[0003] There are two main traditional methods for solving geological monitoring problems in this field: (1) Based on manual field research and survey, although this method can easily obtain actual accurate data information of the survey area, it consumes a lot of manpower and material resources, has low real-time performance, and has a very high risk factor. (2) Based on drone remote sensing image survey, it has many disadvantages such as time-consuming, high investment, and danger. In this dilemma, the geological disaster monitoring system based on remote sensing data and deep learning methods has the advantages of high real-time performance, high accuracy, and large amount of information, and has become a new survey method. Artificial intelligence and deep learning technology have the advantages of high-precision recognition. Compared with previous conventional detection methods, they have the advantages of improving detection and recognition rates and overcoming previous research difficulties. However, most of the current neural networks have the disadvantages of excessive model memory overhead, long running time, excessive energy loss, and long query time. Summary of the invention

[0004] In order to solve the technical problems in the existing geological monitoring technology, such as excessive storage overhead, long running time, excessive energy loss, and long query time in the process of geological detection using a neural network model, the present invention provides a geological disaster monitoring method and system based on deep learning and spatial data query.

[0005] The technical solution of the present invention to solve the above technical problems is as follows:

[0006] The geological disaster monitoring method based on deep learning and spatial data query includes the following steps:

[0007] Constructing a geological disaster location rating model based on twin neural networks;

[0008] Obtain pre-disaster remote sensing image data of the disaster-affected area;

[0009] Based on the materialized view query method, TBDR-Tree index method and graph contour segmentation method, post-disaster remote sensing image data of the disaster-stricken area are obtained from the spatial database;

[0010] By inputting the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model, the location and disaster level of the disaster-affected area are identified, and the disaster location information and disaster level information are correspondingly obtained;

[0011] The disaster-affected location information and the disaster-affected level information are output.

[0012] The beneficial effects of the present invention are: by adopting the TBDR-Tree index structure algorithm, materialized view and spatiotemporal query algorithm of trajectory segmentation, the query time is greatly optimized, the memory overhead is reduced, and the running time is shortened. At the same time, the research on the compression and acceleration of convolutional neural networks is carried out from the software algorithm level. It is planned to adopt the knowledge distillation method, through the spatiotemporal data optimization technology, to explore the deep meaning of remote sensing data, design and develop a lightweight remote sensing image disaster detection deep learning model for geological disaster monitoring, and realize the functions of efficient and accurate disaster detection, disaster area identification, and disaster level classification for remote sensing images.

[0013] Based on the above technical solution, the present invention can also be improved as follows.

[0014] Furthermore, a geological disaster location rating model is constructed based on the twin neural network, including the following steps:

[0015] Acquire a plurality of historical pre-disaster remote sensing image data before the disaster, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a backbone training data set;

[0016] Establish a backbone network model based on the image segmentation method in deep learning;

[0017] Using the backbone training data set to train the backbone network model to obtain a disaster location recognition model;

[0018] Constructing a twin network that shares model weights with the backbone network model, and migrating the encoder of the disaster location recognition model to the encoder of the twin network;

[0019] Acquire a plurality of historical post-disaster remote sensing image data corresponding to the historical pre-disaster remote sensing image data, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a twin training data set;

[0020] Using the backbone training data set and the twin training data set to train the twin network to obtain a disaster degree determination model;

[0021] Integrate the disaster location recognition model with the disaster extent determination model to obtain an original training model;

[0022] Construct a student model based on the transfer learning method of knowledge distillation;

[0023] The original training model is transferred to the student model as a teacher model to obtain the geological disaster location rating model.

[0024] Furthermore, the backbone network model includes an encoder and a decoder; the encoder is a mobile flipping bottleneck convolution module, and the inverse residual activation function of the mobile flipping bottleneck convolution module is a Swish activation function; the mobile flipping bottleneck convolution module includes a first point-by-point convolution layer, a depth convolution layer, an attention mechanism layer and a second point-by-point convolution layer.

[0025] Furthermore, the disaster level information includes four different levels of disaster degree information; the geological disaster location rating model has five output channels, one of which outputs the disaster location information, and the remaining four output channels output the disaster degree information representing four different levels respectively.

[0026] Furthermore, based on the materialized view query method, the TBDR-Tree index method and the graph contour segmentation method, the post-disaster remote sensing image data of the disaster-stricken area is obtained from the spatial database, including the following steps:

[0027] Constructing the spatial database based on the materialized view query method;

[0028] Based on the graphic contour segmentation method, the satellite remote sensing image data is segmented into a plurality of image data fragments and then stored in the spatial database;

[0029] Dynamically update the satellite remote sensing image data after the disaster in the spatial database by a preset data update calculation method;

[0030] Based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are queried from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data.

[0031] Further, dynamically updating the satellite remote sensing image data after the disaster in the spatial database by a preset data update calculation method comprises the following steps:

[0032] Collect real-time satellite remote sensing image data;

[0033] Determine whether the real-time satellite remote sensing image data intersects with the satellite remote sensing image data after the disaster in the spatial database;

[0034] If so, the spatial database is not updated;

[0035] If not, the spatial database is updated.

[0036] Further, based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are queried from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data, including the following steps:

[0037] Establishing an R-tree in the master boot sector; wherein the R-tree includes a plurality of nodes, a leaf node of the R-tree is a minimum boundary oblique rectangle, and each of the minimum boundary oblique rectangles corresponds to one of the image data fragments;

[0038] Based on the key value indexing method, by searching the root node and the parent node of each of the minimum boundary tilted rectangles in the R tree, the plurality of image data fragments after the disaster are queried;

[0039] The post-disaster remote sensing image data is obtained by splicing the plurality of image data fragments into a remote sensing image.

[0040] Further, outputting the disaster location information and the disaster level information comprises the following steps:

[0041] Build WEB page map based on Openlayer front-end map development library;

[0042] The disaster-affected location information and the disaster-affected level information are displayed on the WEB page map.

[0043] In order to solve the above technical problems, the present invention also provides a geological disaster monitoring system based on deep learning and spatial data query, and its specific technical solution is as follows:

[0044] A geological disaster monitoring system based on deep learning and spatial data query, including a model building module, a data acquisition module, a data query module, a disaster assessment module, and an output display module;

[0045] The model building module is used to build a geological disaster positioning and rating model based on a twin neural network;

[0046] The data acquisition module is used to acquire pre-disaster remote sensing image data of the disaster-stricken area;

[0047] The data query module is used to obtain post-disaster remote sensing image data of the disaster-stricken area from the spatial database based on the materialized view query method, the TBDR-Tree index method and the graphic contour segmentation method;

[0048] The disaster assessment module is used to input the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model to identify the location and disaster level of the disaster-affected area, and obtain disaster location information and disaster level information accordingly;

[0049] The output display module is used to output the disaster location information and the disaster level information.

[0050] In order to solve the above technical problems, the present invention further provides a computer, and its specific technical solution is as follows:

[0051] A computer includes a memory and one or more processors, wherein the memory stores executable code, and when one or more of the processors execute the executable code, the steps of the above-mentioned geological disaster monitoring method based on deep learning and spatial data query are implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 It is a flowchart of a geological disaster monitoring method based on deep learning and spatial data query in an embodiment of the present invention;

[0053] Figure 2 This is a schematic diagram of the architecture of the geological disaster positioning and rating model in an embodiment of the present invention;

[0054] Figure 3 It is a schematic diagram of the structure of the backbone network in the geological disaster positioning and rating model in an embodiment of the present invention;

[0055] Figure 4 Schematic diagram of the structure of the inverted residual module in an embodiment of the present invention;

[0056] Figure 5 This is a schematic diagram of the structure of the compression and excitation module in an embodiment of the present invention;

[0057] Figure 6 It is a structural schematic diagram of a mobile flip bottleneck convolution module in an embodiment of the present invention;

[0058] Figure 7 This is a schematic diagram of the structure of the knowledge distillation framework integration model in an embodiment of the present invention;

[0059] Figure 8 A schematic diagram of a materialized view structure in an embodiment of the present invention;

[0060] Fig. 9This is a schematic diagram of the TBDR-Tree structure in an embodiment of the present invention;

[0061] Fig.10 A schematic diagram of a hierarchical search index structure in an embodiment of the present invention;

[0062] Fig.11 This is a schematic diagram of an R-tree query structure in an embodiment of the present invention;

[0063] Fig.12 Schematic diagram of the track segment index structure in an embodiment of the present invention;

[0064] Fig.13 This is a schematic diagram of the structure of the physical model of the disaster-stricken area in an embodiment of the present invention;

[0065] Fig.14 It is a schematic diagram of the structure of the physical model of the disaster-stricken building in an embodiment of the present invention;

[0066] Fig.15 This is a schematic diagram of the ER model structure in an embodiment of the present invention;

[0067] Fig.16 A diagram showing a table structure of a disaster situation database in an embodiment of the present invention;

[0068] Fig.17 A schematic diagram of the structure of a geological disaster monitoring system based on deep learning and spatial data query in an embodiment of the present invention;

[0069] Fig.18 The query optimization experiment results in the embodiment of the present invention are Figure 1 ;

[0070] Fig.19 The query optimization experiment results in the embodiment of the present invention are Figure 2 ;

[0071] Fig. 20 The query optimization experiment results in the embodiment of the present invention are Figure 3 . DETAILED DESCRIPTION

[0072] The principles and features of the present invention are described below in conjunction with the accompanying drawings. The examples given are only used to explain the present invention and are not used to limit the scope of the present invention.

[0073] like Figure 1 As shown, this embodiment provides a geological disaster monitoring method based on deep learning and spatial data query, comprising the following steps:

[0074] S1. Constructing a geological disaster location rating model based on twin neural network;

[0075] S2, obtaining pre-disaster remote sensing image data of the disaster-stricken area;

[0076] S3, based on the materialized view query method, TBDR-Tree index method and graph contour segmentation method, obtain post-disaster remote sensing image data of the disaster-stricken area from the spatial database;

[0077] S4, inputting the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model to identify the location and disaster level of the disaster-affected area, and correspondingly obtaining disaster location information and disaster level information;

[0078] S5. Output the disaster-affected location information and the disaster-affected level information.

[0079] Specifically, the establishment of the geological disaster location rating model is a lightweight remote sensing image model based on knowledge distillation. The specific scheme of the model is as follows:

[0080] The core function of this model is to identify the content of remote sensing images. A lightweight remote sensing image deep learning model based on knowledge distillation is proposed. A large number of labeled pre-disaster images and post-disaster images in different time domains are trained to identify the contours of the disaster-stricken areas and classify the disaster levels. By constructing a variety of network structures and knowledge distillation techniques, the model realizes the functions of contour recognition and classification.

[0081] Specifically, Figure 2 As shown in the figure, the lightweight remote sensing image model based on knowledge distillation takes two remote sensing images of the same geographical location in different time domains as input, and can directly predict the location of the building and the degree of damage. The model adopts a twin network structure with shared weights. The training of the model includes two consecutive stages: the first stage is the pre-training of building location of the backbone network (Backbone); the second stage is the training of disaster degree determination based on knowledge distillation. The above two stages of training are used to maximize the model performance.

[0082] like Figure 3 As shown in the figure, the backbone network is mainly based on the U-net model, which concatenates and fuses the high-level feature maps in the Decoder with the low-level feature maps of the same size and number of channels in the Encoder to achieve more accurate positioning performance.

[0083] As shown in Table 1, the encoder of this model uses the lightweight network EfficientNet-B0, which uses the Mobile inverted Bottleneck Convolution (MBConv) module as the basic block, which can achieve more accurate and faster model reasoning.

[0084] like Figure 4As shown in Figure 1, the structure of the inverted residual module in MBConv is similar to that in MobielNetV2. The difference is that MBConv replaces the ReLu6 activation function in the inverted residual module with a smooth, non-monotonic Swish activation function, i.e., f(x) = x·σ(x), where σ (x) is the sigmoid activation function (1 / (1+exp(x))).

[0085] like Figure 5 As shown in Figure 1, MBConv and MobielNetV2 have added a compression and excitation module (Squeeze-and-Excitation, SE). Figure 6 As shown in the figure, the first convolution of the mobile flip bottleneck convolution module, i.e., the MBConv module, is a 1×1 point-by-point convolution (Conv 1×1), the purpose of which is to expand the channels of the feature map to enrich the number of features; the second convolution is a depthwise convolution (Dwise Convolution), that is, each channel only uses a convolution kernel with a channel number of 1 to perform a convolution operation to reduce the number of parameters and calculations exponentially, and its output has the same shape as the input feature map; the SE module provides a channel attention mechanism for the feature map after two convolution operations; and the final point-by-point convolution reduces the number of channels of the feature map.

[0086]

[0087]

[0088] Table 1 EfficientNet-B0 architecture details

[0089] The backbone network pre-training method is as follows: In this stage of training, only the above backbone network is used for preliminary building location training. This stage is equivalent to a pre-training process, the purpose of which is to enable the encoder to have a certain building location perception ability in advance, so that the second stage of fine-tuning the building damage level classification can benefit from it.

[0090] The model only takes the remote sensing image x before the disaster as input, and finally outputs a single-channel feature map f(x) through the above backbone network model. The model is iteratively trained through formula (2). Before training, the training data is subjected to data enhancement operations such as flipping, translation, rotation, scaling, histogram, brightness, contrast and elastic transformation.

[0091] min L pre (x,B)=α·D(σ(f(x)),B)+β·F(σ(f(x)),B), (2)

[0092] Where σ(f(x)) is the sigmoid activation function, D(σ(f(x)), B) and F(σ(f(x)), B) are the Dice loss and the Focal loss shown in Formulas 3 and 4, respectively. α and β correspond to the weight hyperparameters of the above two losses, respectively. B represents the mask matrix label of the building location, that is, the value of the pixel where the building is located is 1, and the rest are 0.

[0093]

[0094]

[0095] Among them, yi and y'i represent the label value and predicted value of the i-th pixel respectively, and N is the total number of pixels.

[0096] The disaster judgment training method is as follows: The disaster judgment adopts a twin network structure with shared weights, such as Figure 2 As shown. The model uses the remote sensing images x and x' of the same geographical location before and after the disaster and in different time domains as the model input after the same data enhancement operation as the first stage, and splices and fuses them at the end of the model. Before training, the encoder of the first stage needs to be migrated to the encoder of the twin network so that the model can quickly converge to a better state. The model finally outputs f(x,x') with a feature map of C channels and the same size as the input for locating buildings and classifying the degree of disaster. Among them, C is the number of categories, including four levels of disaster degree of non-buildings and buildings, and the four levels of disaster degree are undamaged, slightly damaged, moderately damaged, and severely damaged. Among them, in most data, the number of pixels occupied by buildings is very small compared to non-buildings, and there is a serious problem of pixel category imbalance. If each channel represents a category and cross entropy is used for training directly, this will not benefit the training process from the pre-training process of location perception in the first stage. Therefore, we still use the first channel as the building location judgment, and the four channels 2-5 represent the degree of disaster in turn. The objective function is shown in Formula 5.

[0097]

[0098] Among them, M represents the third-order mask label tensor of multi-classification, and M[i] represents the mask label matrix of the i-th channel, that is, the value of the pixel where the building (i=0) or the building with disaster degree i is located is 1, and the rest are 0.

[0099] In the inference phase, each pixel value in f(x,x') is compared with the set threshold θ. If it is greater than the threshold, it is 1, otherwise it is 0. Among them, the 2nd to 5th four channels are intersected with the 0th channel in turn. If the i∈[1,5]th channel and the j∈[1,5]th channel have an intersection, the intersection is determined as the lowest level of disaster damage.

[0100] The knowledge transfer method based on integrated distillation is as follows: Different models can extract features from different perspectives. Therefore, model integration can integrate features from multiple perspectives, which can often achieve better performance than a single model. However, model integration will multiply the internal resources of the device consumed by the inference system, such as memory and computing power, and is not suitable for actual deployment. In order to ensure that the model can still obtain the superior performance of model integration without increasing internal resource consumption, model integration and knowledge distillation are combined. Knowledge distillation is to transfer the knowledge covered by a large-scale model to a small-scale model, so that the performance of the small-scale model can be close to that of the large-scale model. Based on this idea, the knowledge of multiple models is transferred to the model, that is, the output feature maps of all teacher models are jointly used to supervise the training of the model. The distillation loss is shown in Formula 6.

[0101]

[0102] in, is the mean square error, F s The last layer of the student model decoder is Figure 7 The input feature map of Decoder[-1] in F T i Represents the input feature map of the last layer of the decoder of the i-th teacher model. The teacher model uses several models such as ResNet34, ResNetx, Dpn94, and its training process is the same as the training process of the student model mentioned above. L KD Collaborative L M Optimize the student model to further improve its performance:

[0103] L T =(1-γ)·L M +γ·L KD . (7)

[0104] Among them, γ is the adjustment label loss L M and distillation loss L KD The impact factor of L KD Collaborative L M Optimizing the student model can transfer the integrated teacher model knowledge to the student model, so that the performance of the student model can be close to the performance of the model integration.

[0105] The optimization method for geological disaster monitoring data query is as follows: The present invention realizes efficient query of disaster information of remote sensing images through the organic combination of comprehensive utilization of materialized view query, TBDR-Tree index, minimum contour segmentation and other technologies, and ensures the efficiency of short-term and large-scale data query of the disaster monitoring system in actual application, which provides escort for disaster forecasting, post-disaster relief strategy formulation and implementation.

[0106] A materialized view is a database object that contains query results. It can be used to pre-calculate and save operation results, such as table joins and aggregation operations, to avoid these time-consuming processes when executing queries and get results quickly. Materialized views cannot be updated automatically and can only be updated manually, which makes the operation cumbersome and may sometimes result in inaccurate query data. Figure 8 As shown, the present invention addresses the shortcoming that materialized views cannot be updated in real time, and implements dynamic data update operations by writing functions so that they can change in real time. The spatiotemporal query algorithm code based on materialized views is as follows:

[0107]

[0108] In the spatiotemporal query algorithm code based on materialized views, the input data of the spatiotemporal query algorithm based on materialized views is the movement trajectory data (mgeometry) in a given spatiotemporal range, the movement trajectory data array (geometry) stored in the database, and a given time range (period). The output of the algorithm is to determine whether the movement data in the given spatiotemporal range intersects with the movement trajectory data in the database. If so, it is true, otherwise it is false (update the database). In lines 1-15 of the algorithm code, variables such as f_mgeometry, f_geometry, f_period, f_mgeometry_segtable_name, sql, cnt, geos, res, meta_key, meta_value, sql_text, table_key, session_key, session_value, and tmp_table are defined. In lines 16-17, values ​​are assigned to f_mgeometry_segtable_name and session_key respectively. In lines 18-23, the current session_key setting is queried and the query result is assigned to session_value. The next step is to determine whether session_key has not been assigned a value. If not, it is set first, and then the query result of the session_key setting is assigned to session_value. Lines 24-31 determine whether the value of session_value is 0. If it is 0, the value of session_key is set again, and the sql_text statement is executed in the database storing the mobile trajectory data within a given time range. The session part adopts a materialized view-based method and implements dynamic data update operations by writing functions. In lines 32-33, sql_text is reassigned, and then the execution result of sql_text is assigned to cnt for counting. In lines 34-37, it is determined whether the value of cnt is greater than 0. If it is greater than zero, a true value is returned, which means whether the mobile data in the given time and space range intersects with the mobile trajectory data in the database; if the value of cnt is not greater than zero, a false value is returned, which means whether the mobile data in the given time and space range does not intersect with the mobile trajectory data in the database.

[0109] The data query method based on indexing technology is as follows: an R-tree is established based on the MBR, i.e. the master boot record, and the MBR of the spatial object is used to approximate the spatial object, which has the advantage of directly indexing the spatial object. Its disadvantages are also obvious. If there are too many spatial overlapping areas, the retrieval efficiency will be reduced. GeoVideoIndex is an efficient indexing method for retrieving georeferenced videos. The minimum bounding tilted rectangle (MBTR) is used in the leaf node because MBTR can effectively represent the remote sensing image area. However, constructing MBTR based only on the linear changes of the segmentation of the disaster-affected contour of the remote sensing image is only suitable for indexing fast-moving and small-direction moving trajectories, such as high-speed cars and drones.

[0110] The present invention proposes a TBDR-Tree structure, wherein the TB tree is an index structure, D represents the direction, and the R-Tree is also an index structure. Fig. 9 As shown in the figure, TBDR-Tree uses MBTR in leaf nodes, and each MBTR can be indexed by three 6-bit key values, which represent the positions of nodes in the 1st to 3rd layers from left to right. Therefore, TBDR-Tree provides efficient storage and direction optimization, providing fast key-to-pointer lookup instead of fixed-size arrays.

[0111] like Fig.10 As shown in the figure, the fields in the Affected_area table include Area_ID (area number), Affected_Type (disaster type), Affected_Grade (disaster grade), Affected_Time (disaster time), etc. The fields in the Affected_area_01_Build table include Area_ID (area number), Affected_Type (disaster type), Affected_Grade (disaster grade), mbr (disaster outline), etc., among which the cars table and the mpoint table are associated through the foreign key Build_ID. The TBDR-Tree index structure is used in the two data tables. Here, we divide the steps of retrieving trajectory data that meet the query requirements into Filterstep and Refinementstep. In the Filter step, the index structure is searched level by level, and then candidates are generated according to the needs of filtering.

[0112] In this step, a large amount of invalid data is filtered out through fast calculation, which effectively reduces the query time. In the refinement step, the data is retrieved in the materialized appearance to further find the trajectory data that meets the query conditions. The code of the spatiotemporal query algorithm based on contour segmentation is as follows:

[0113]

[0114] Table 3 Spatiotemporal query algorithm based on contour segmentation

[0115] In the code of the spatiotemporal query algorithm based on contour segmentation, the input data of the algorithm for filtering and generating candidates is the movement trajectory data (mgeometry) in a given spatiotemporal range, the movement trajectory data array (geometry) stored in the database, and a given time range (period). The output of the algorithm is to determine whether the movement data in the given spatiotemporal range intersects with the movement trajectory data in the database. If so, it is true, otherwise it is false. In lines 1-8 of the algorithm code, the f_mgeometry, f_geometry, f_period, f_mgeometry_segtable_name, sql, cnt, geos, and res variables are defined. In lines 9-10, the f_mgeometry_segtable_name of the mobile trajectory data in a given time and space range is queried and assigned a value. In lines 11-12, the f_mgeometry_segtable_name table in the database is queried within a given time range and the query results are assigned to geos. Lines 13-17 determine whether geos is nonull. If geos is not empty, res is assigned a value. Finally, a true value is returned, which means whether the mobile data in the given time and space range intersects with the mobile trajectory data in the database. If it is empty, no value is assigned. Finally, a false value is returned, which means whether the mobile data in the given time and space range does not intersect with the mobile trajectory data in the database.

[0116] In addition, trajectory fragments are optimized based on the R-tree. For the R-tree, its data is stored in memory, i.e., pages. Each page stores multiple data items. The number of data items does not exceed a pre-defined maximum value, and is generally more than the minimum value. Each piece of data on a non-leaf node consists of an identifier pointing to a child node and the circumscribed rectangle of the child node. The leaf node stores the data required for each object, generally a circumscribed rectangle and an identifier pointing to the data. If it is point data, the leaf node only needs to store the points themselves. If it is polygonal data, the general practice is to store the minimum circumscribed rectangle of the polygon and an identifier of the data pointing to the polygon in the leaf node.

[0117] R-tree represents data composed of two-dimensional or higher-dimensional regions, which are expressed as data regions. The internal nodes of an R-tree correspond to an internal region, or "region", which is not an ordinary data region. In principle, the region can be of any shape, although in practice it is often a rectangle or other simple shape. For example, in the figure above, a is an R-tree, and one of its internal nodes R3, R4, and R5 represents a region in b, which is contained in R1. The nodes of the R-tree use sub-regions instead of keys. The sub-regions represent the contents of the sub-nodes of the node. For example, R3, R4, and R5 are the keys in nodes R3, R4, and R5, each of which represents a sub-region in b. Note that the sub-region does not cover the entire region. It meets the requirements as long as all data regions in the large region are completely contained in a small region. Furthermore, sub-regions are allowed to overlap partially. For example, R3 and R4 overlap each other, and we hope that the overlapping part is as small as possible.

[0118] Indexing technology mainly includes Insert() algorithm, SplitAndAdjust() algorithm and optimization algorithm. The insertion algorithm code of R-tree based on R-tree spatial indexing technology is as follows:

[0119]

[0120] In the R-tree insertion algorithm code, the R-tree insertion algorithm is used to find a suitable insertion node in the R-tree. First, an initial search is performed from the root, and a path to the leaf is selected through a while loop until a suitable insertion position is found (code lines 1-4). After that, the insertion operation is performed. If the node overflows, the tree is split and adjusted, otherwise the path is adjusted (code lines 5-9). The R-tree adjustment path algorithm code is as follows:

[0121]

[0122] In the R-tree path adjustment algorithm code, the R-tree path adjustment algorithm is used to select a path in the R-tree to find a suitable insertion point. First, find the parent node of the current node and adjust the entry path of the parent node. If the entry has been modified, adjust the path of the parent node. The R-tree splitting and adjustment algorithm code is as follows:

[0123]

[0124] In the split and adjust algorithm code of the R-tree, the split and adjust algorithm first creates a new node and distributes the path. If it is a root node, a new node is created. Otherwise, the parent node of the node is obtained, the path in its parent node is adjusted, and the new node is inserted (code lines 1-3). If the parent node overflows, the path is split and adjusted. Otherwise, only the path is adjusted (code lines 4-8). The R-tree trajectory fragment index algorithm code is as follows:

[0125]

[0126] In the R-tree trajectory fragment indexing algorithm code, the trajectory fragment indexing is mainly based on the R-tree, where the size of the trajectory fragment can be selected to be less than or equal to 8K, and the structure of the R-tree is optimized based on the direction, further improving the query efficiency.

[0127] like Figures 11 to 12 As shown in the figure, five different tracks are selected from the Video DB and their visual direction information is analyzed. Through direction optimization and continuous iteration using track segment index, where each segment is no larger than 8K, the final required data can be determined more quickly, and the angle data of the four directions can be obtained. This is also applicable to track index scenarios with large changes in direction.

[0128] The data processing method based on the minimum contour segmentation of the disaster area is as follows: The contour segmentation of the disaster area divides the trajectory into fragments according to the spatial shape and time interval factors, and then performs classification and clustering, which can obtain better results. Each trajectory can be surrounded by a minimum frame, which covers the spatial range and time range of the contour. As the moving time of the moving object increases, the invalid index space of the contour increases, and the index efficiency decreases.

[0129] Since the leaf nodes store contours that are no larger than 8K in size, we store the contour data in segments according to the data storage principle of PostpreSQL per page, and through contour segmentation, the irregular complex contours are segmented into several regular small contours, thereby significantly improving the query performance. Because in general, setting the memory capacity of the pass cache area to 25% of the physical memory can obtain the optimal data query processing performance. The spatiotemporal query algorithm code based on trajectory segmentation is as follows:

[0130]

[0131] Algorithm description: In the spatiotemporal query algorithm code based on trajectory segmentation, the spatiotemporal query algorithm based on trajectory segmentation is used to segment the mobile data in a given spatiotemporal range, and finally return the segmentation result to facilitate the query of trajectory data. First, the mobile trajectory data (mgeometry) in the given spatiotemporal range, the mobile trajectory data array (geometry) stored in the database, and the given timeline (timeline) data are input. After defining the relevant variables (code lines 1-12), the relevant algorithm statements designed in this patent assign these variables and submit the stored procedure errors and messages, and then assign the execution result of sql to cnt_mpid, and then reassign sql, and assign the result of the reassigned sql to tp_seg_size (code lines 13-21).

[0132] Then determine whether cnt_mpid is less than 1. If it is less than 1, execute the insert statement under the given constraints (code lines 22-24). By determining that the cnt_mpid value is greater than 0, assign the segid segmentation id with a new sql statement, submit the stored procedure error and message, and assign the execution result of the new sql statement to max_tpseg_count. Then determine whether segid is empty and whether max_tpseg_count is greater than tp_seg_size. If segid is not empty and max_tpseg_count is less than tp_seg_size, execute the update statement under the given constraints, otherwise assign new_segid (new_segid<-segid+1), and then execute the insert statement under the given constraints. Finally, return the segmented mgeometry (code lines 25-38).

[0133] In some embodiments, the database uses PostgreSQL database. The performance of a database directly determines the performance of the system. Reasonable database design can reduce the difficulty of subsequent development, as well as the convenience and stability of operation and maintenance. The present invention is a big data disaster monitoring platform that needs to store a large amount of pictures and disaster information, taking into account the development cost and storage requirements.

[0134] The ER model is a model that converts the real world into an abstract model. The main components of the model are entities, relationships and attributes. In the present invention, the following database entities are designed, including disaster-affected area entities and disaster-affected building entities.

[0135] like Fig.13 ,The disaster affected area entity includes attributes such as area number, disaster type, disaster level, included buildings, pictures before and after the disaster, disaster time, etc. Among them, the area number is the primary key, and the included buildings are the foreign key.

[0136] like Fig.14 ,The disaster affected building entity includes attributes such as building number, disaster level, region to which it belongs, building area, etc. Its primary key is building number.

[0137] like Fig.15 , the relationship of the ER model is used to describe the relationship between entities. One way is to examine whether there is a relationship between two entities in the local structure based on the results of the demand analysis. If there is, determine whether it is a 1:n, m:n or 1:1 relationship. In addition, it is also necessary to consider whether there is a relationship within the entity type. Whether there is a relationship between two entity types, etc.

[0138] like Fig.16 , create data tables corresponding to entities according to the database concept design. There are two types of data tables. The table structures of the two are introduced below:

[0139] As shown in Table 2, the disaster-affected area information table is used to store the disaster information of all disaster-affected areas, including the disaster-affected area ID, disaster type, disaster level, building ID, pre-disaster photos and post-disaster photos; wherein the disaster-affected area ID is numbered in an automatic numbering manner.

[0140]

[0141] Table 2 Structure of Affected_area_01 table

[0142] As shown in Table 3, the disaster-affected building information table contains various types of information specific to a building in the disaster-affected area, including the area ID, disaster level, building ID, and the outline of the disaster-affected building.

[0143]

[0144] Table 3 Structure of Affected_area_01_Build01 table

[0145] In some embodiments, Fig.17 As shown, the present invention designs a complete disaster remote sensing information system for real-time monitoring, identification and response. According to the technical content, it is mainly divided into three parts, namely, the disaster detection feedback front-end interface based on Openlayer, the spatial database and query optimization algorithm back-end based on PostgreSQL-PostGIS, and the lightweight remote sensing image model for remote sensing image processing based on knowledge distillation.

[0146] First, a lightweight remote sensing image model is trained locally to enable it to recognize the contours of the disaster-affected areas and classify the disaster levels in post-disaster remote sensing images, and then tested. Then, a PostgreSQL-PostGIS spatial database is built on the server to store the disaster remote sensing images and the spatial coordinate data of the disaster contours. JDBC is used on the back end to connect to the database and write a materialized view-based algorithm to achieve query optimization. Finally, after the front end and back end based on Openlayer are interconnected, users can realize its functions.

[0147] In some embodiments, the evaluation indicators used are mean Accuracy (mAcc), mean Intersection over Union (mIoU), and coefficient.

[0148] The calculation formula for average category accuracy (mAcc) is as follows:

[0149]

[0150] Among them, p ij It indicates the number of points that originally belong to the i-th class but are predicted to be of the j-th class.

[0151] The calculation formula for the average class intersection over union (mIoU) is as follows:

[0152]

[0153] Among them, mIoU can be interpreted as the average intersection over union ratio, that is, the IoU value calculated on each category is the number of true samples divided by the sum of the number of true samples, the number of false negative samples, and the number of false positive samples.

[0154] The experimental results and analysis are as follows:

[0155] First, the performance of the query optimization algorithm is experimentally verified. The verification indicator is the response speed of the algorithm. The X-axis represents the number of request queries, and the Y-axis represents the query execution time. The selected data set is the remote sensing image training set and test set xBD that have been predicted by deep learning. Among them, the size of xBD is 30GB, containing 22068 high-resolution satellite remote sensing images with a resolution of 1024×1024. The comparative analysis algorithm is to perform data query processing on the non-optimized query algorithm, the index-based query optimization algorithm, the materialized view-based query optimization algorithm, and the materialized view optimization and trajectory segmentation technology. Finally, the four experimental results are compared to draw conclusions.

[0156] For the three types of queries, namely time range query, spatial range query, and spatiotemporal range query, the three designed query algorithms, namely the non-optimized query algorithm, the index-based query optimization algorithm, and the query optimization algorithm based on optimized materialized views, were tested respectively.

[0157] like Figures 18 to 20 It can be seen that with the increase in the number of irregular remote sensing disaster area contour image objects, the query time of target data is also increasing. This is because as the amount of query data increases, the data processing time required for querying is also increasing. Comparing the query algorithm without optimization, the index-based query optimization algorithm and the materialized view method only, it can be seen that the optimization algorithm based on materialized view is superior to other query algorithms, and the query optimization algorithm based on optimized materialized view proposed in the present invention shows a shorter query time among the four and has a faster query speed. This is because displaying data in the form of materialized view optimizes the data query process, reduces more cumbersome operation processes such as table connection, thereby speeding up the query process, and adding trajectory segmentation technology to divide the query data into small interval segments, which is more conducive to accelerating the query speed.

[0158] The experiment and analysis of the geological disaster location rating model are as follows:

[0159] The dataset used in the experiment is the disaster image dataset xBD

[14] released by MIT in 2020. The xBD dataset is the first building damage level assessment dataset to date and one of the largest and highest quality public datasets of annotated high-resolution remote sensing images. It is also the designated dataset in the xView2 Challenge. The dataset is 30GB in size and contains 22,068 high-resolution satellite remote sensing images with a resolution of 1024×1024. The images record 19 different natural disasters and accidents such as earthquakes, floods, wildfires, volcanic eruptions, and car accidents. Each sample includes pre-disaster and post-disaster images, which can be used for building location and damage assessment, respectively. The sample labels are presented in JSON files, and each image sample corresponds to a JSON file. A single JSON file includes several polygon outline coordinates representing the location of the building. The post-disaster image label adds the corresponding damage level to each polygon, including no-damage, minor-damage, major-damage, and destroyed.

[0160] The dataset includes four subsets: Train, Test, Holdout, and Tier3. The Holdout and Test sets are used to test the generalization performance of the model and the final ranking of the competition, and their labels are not published. Therefore, in the experimental stage, only the Train and Tier3 datasets were used for experiments. After merging the two datasets, the training and test sets were divided into a ratio of 9:1.

[0161] Evaluation criteria: The experiment uses IoU (Intersection over Union) value, Dice Similarity Coefficient and F1 value to evaluate the prediction performance of the model. The IoU value is the ratio of the area of ​​the overlapping part C of the actual area A and the predicted area B of the object to the area occupied by the two areas (A and B) as a whole. The calculation is shown in formula (1); the calculation of the Dice value is shown in formula (2).

[0162]

[0163] Among them, TP, FN, and FP are the total number of pixels of true positives, false positives, and false negatives, respectively.

[0164] In addition, the F1 value is used as one of the evaluation indicators, as shown in formula (3).

[0165]

[0166] Ablation experiment:

[0167]

[0168] Table 4 Experimental results of ablation experiment

[0169] In order to prove that each technique of the proposed method is necessary, the following three ablation experiments were conducted:

[0170] No pre-training: When training the model, the first-stage building location perception pre-training process is omitted, and the twin network with randomly initialized parameters is directly used to perform building damage level judgment training through integrated distillation.

[0171] No integrated distillation: The training process uses LM loss to iteratively update the model.

[0172] Cross entropy loss: Cross entropy loss is used instead of sigmoid loss for training.

[0173] In addition, the three teacher models of DPN92, SEnet154 and SE-ResNet50 are integrated, and the results are shown in the “Model Integration” in Table 1. The ablation experiment results are shown in the table, where mIoU and Dice are the mean values ​​of IoU and Dice of the four damage levels, LocationF1 represents the F1 value of building location determination, and F1(overall) is the mean value of F1 of the four damage levels.

[0174] It can be observed from Table 4 that:

[0175] Although the final performance of the proposed method is not as good as that of model ensemble, it is extremely close to the final performance of model ensemble, with mIoU, mDice, and F1 values ​​differing by only 0.42%, 0.29%, and 0.41%, respectively. In addition, the resource consumption of the proposed method is much lower than that of model ensemble, and it is more applicable than model ensemble.

[0176] The proposed method achieved the best performance compared with the other three ablation experiments. In terms of mIoU, mDice, and F1 value, the proposed method is 2.23%, 1.59%, and 2.02% higher than "no pre-training"; 2.93%, 2.09%, and 2.85% higher than "no integrated distillation"; and 0.82%, 0.58%, and 0.49% higher than "cross entropy loss". The above fully proves the effectiveness of the proposed method and the necessity of each part.

[0177] Resource consumption comparison: In this section, we briefly analyze the resource consumption of the model. We run the proposed method, model integration method and some other mainstream networks on Windows 10 Intel Core i5-9500 CPU. The execution results are shown in Table 12, where the network structure is a twin network with shared weights, Params is the number of parameters, Size is the memory size occupied by the parameters, FLOPs is the floating point number calculation, and Time is the average inference time of a single sample.

[0178]

[0179] Table 5 Comparison of resource consumption

[0180] As can be seen from Table 5, the number of parameters and the amount of calculation of the model of the present invention are much smaller than those of other baselines, and the inference speed is also the fastest. The integrated distillation method proposed in the present invention can reduce the number of parameters and the amount of calculation of the model integration by 95.89% and 96.54%, and improve the inference speed by 65.27%. For actual deployment, the method provided by the present invention has an absolute advantage over other baselines.

[0181] In some embodiments, a computer is also provided, including a memory and one or more processors, wherein the memory stores executable code, and when one or more of the processors execute the executable code, the steps of the geological disaster monitoring method based on deep learning and spatial data query in Example 1 are implemented.

[0182] The memory may be an internal storage unit of any device with data processing capability described in any of the aforementioned embodiments, such as a hard disk or a memory. The memory may also be an external storage device of any device with data processing capability, such as a plug-in hard disk, a smart memory card, an SD card, a flash memory card, etc. equipped on the device. Furthermore, the memory may include both an internal storage unit and an external storage device of any device with data processing capability. The memory is used to store the computer program and other programs and data required by any device with data processing capability, and may also be used to temporarily store data that has been output or is to be output.

[0183] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the concept and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A geological disaster monitoring method based on deep learning and spatial data query, characterized in that: The steps include: Constructing a geological disaster location rating model based on twin neural networks; Obtain pre-disaster remote sensing image data of the disaster-affected area; Based on the materialized view query method, TBDR-Tree index method and graph contour segmentation method, post-disaster remote sensing image data of the disaster-stricken area are obtained from the spatial database; By inputting the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model, the location and disaster level of the disaster-affected area are identified, and the disaster location information and disaster level information are correspondingly obtained; Outputting the disaster-affected location information and the disaster-affected level information; The geological disaster location rating model is constructed based on the twin neural network, which includes the following steps: Acquire a plurality of historical pre-disaster remote sensing image data before the disaster, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a backbone training data set; Establish a backbone network model based on the image segmentation method in deep learning; Using the backbone training data set to train the backbone network model to obtain a disaster location recognition model; Constructing a twin network that shares model weights with the backbone network model, and migrating the encoder of the disaster location recognition model to the encoder of the twin network; Acquire a plurality of historical post-disaster remote sensing image data corresponding to the historical pre-disaster remote sensing image data, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a twin training data set; Using the backbone training data set and the twin training data set to train the twin network to obtain a disaster degree determination model; Integrate the disaster location recognition model with the disaster extent determination model to obtain an original training model; Construct a student model based on the transfer learning method of knowledge distillation; Migrating the original training model as a teacher model to the student model to obtain the geological disaster location rating model; Based on the materialized view query method, TBDR-Tree index method and graph contour segmentation method, the post-disaster remote sensing image data of the disaster-stricken area is obtained from the spatial database, including the following steps: Constructing the spatial database based on the materialized view query method; Based on the graphic contour segmentation method, the satellite remote sensing image data is segmented into a plurality of image data fragments and then stored in the spatial database; Dynamically update the satellite remote sensing image data after the disaster in the spatial database by a preset data update calculation method; Based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are searched from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data; Based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are queried from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data, including the following steps: Establishing an R-tree in the master boot sector; wherein the R-tree includes a plurality of nodes, a leaf node of the R-tree is a minimum boundary oblique rectangle, and each of the minimum boundary oblique rectangles corresponds to one of the image data fragments; Based on the key value indexing method, by searching the root node and the parent node of each of the minimum boundary tilted rectangles in the R tree, the plurality of image data fragments after the disaster are queried; The post-disaster remote sensing image data is obtained by splicing the plurality of image data fragments into a remote sensing image.

2. The geological disaster monitoring method based on deep learning and spatial data query according to claim 1 is characterized in that: The backbone network model includes an encoder and a decoder; the encoder is a mobile flip bottleneck convolution module, and the inverse residual activation function of the mobile flip bottleneck convolution module is a Swish activation function; the mobile flip bottleneck convolution module includes a first point-by-point convolution layer, a depth convolution layer, an attention mechanism layer and a second point-by-point convolution layer.

3. The geological disaster monitoring method based on deep learning and spatial data query according to claim 1 is characterized in that: The disaster level information includes four different levels of disaster degree information; the geological disaster location rating model has five output channels, one of which outputs the disaster location information, and the remaining four output channels output the disaster degree information representing four different levels respectively.

4. The geological disaster monitoring method based on deep learning and spatial data query according to claim 1 is characterized in that: Dynamically updating the satellite remote sensing image data after the disaster in the spatial database by a preset data update calculation method includes the following steps: Collect real-time satellite remote sensing image data; Determine whether the real-time satellite remote sensing image data intersects with the satellite remote sensing image data after the disaster in the spatial database; If so, the spatial database is not updated; If not, the spatial database is updated.

5. The geological disaster monitoring method based on deep learning and spatial data query according to claim 1 is characterized in that: Outputting the disaster location information and the disaster level information includes the following steps: Build WEB page map based on Openlayer front-end map development library; The disaster-affected location information and the disaster-affected level information are displayed on the WEB page map.

6. A geological disaster monitoring system based on deep learning and spatial data query, characterized in that: It includes model building module, data acquisition module, data query module, disaster assessment module and output display module; The model building module is used to build a geological disaster positioning and rating model based on a twin neural network; The data acquisition module is used to acquire pre-disaster remote sensing image data of the disaster-stricken area; The data query module is used to obtain post-disaster remote sensing image data of the disaster-stricken area from the spatial database based on the materialized view query method, the TBDR-Tree index method and the graphic contour segmentation method; The disaster assessment module is used to input the pre-disaster remote sensing image data and the post-disaster remote sensing image data into the geological disaster positioning and rating model to identify the location and disaster level of the disaster-affected area, and obtain disaster location information and disaster level information accordingly; The output display module is used to output the disaster location information and the disaster level information; The geological disaster location rating model is constructed based on the twin neural network, which includes the following steps: Acquire a plurality of historical pre-disaster remote sensing image data before the disaster, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a backbone training data set; Establish a backbone network model based on the image segmentation method in deep learning; Using the backbone training data set to train the backbone network model to obtain a disaster location recognition model; Constructing a twin network that shares model weights with the backbone network model, and migrating the encoder of the disaster location recognition model to the encoder of the twin network; Acquire a plurality of historical post-disaster remote sensing image data corresponding to the historical pre-disaster remote sensing image data, and integrate the plurality of historical pre-disaster remote sensing image data into a data set to obtain a twin training data set; Using the backbone training data set and the twin training data set to train the twin network to obtain a disaster degree determination model; Integrate the disaster location recognition model with the disaster extent determination model to obtain an original training model; Construct a student model based on the transfer learning method of knowledge distillation; Migrating the original training model as a teacher model to the student model to obtain the geological disaster location rating model; Based on the materialized view query method, TBDR-Tree index method and graph contour segmentation method, the post-disaster remote sensing image data of the disaster-stricken area is obtained from the spatial database, including the following steps: Constructing the spatial database based on the materialized view query method; Based on the graphic contour segmentation method, the satellite remote sensing image data is segmented into a plurality of image data fragments and then stored in the spatial database; Dynamically update the satellite remote sensing image data after the disaster in the spatial database by a preset data update calculation method; Based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are searched from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data; Based on the TBDR-Tree indexing method, multiple image data fragments after the disaster are queried from the spatial database, and the multiple image data fragments after the disaster are spliced ​​into a remote sensing image to obtain the post-disaster remote sensing image data, including the following steps: Establishing an R-tree in the master boot sector; wherein the R-tree includes a plurality of nodes, a leaf node of the R-tree is a minimum boundary oblique rectangle, and each of the minimum boundary oblique rectangles corresponds to one of the image data fragments; Based on the key value indexing method, by searching the root node and the parent node of each of the minimum boundary tilted rectangles in the R tree, the plurality of image data fragments after the disaster are queried; The post-disaster remote sensing image data is obtained by splicing the plurality of image data fragments into a remote sensing image.

7. A computer, characterized in that: It includes a memory and one or more processors, wherein the memory stores executable code, and when one or more of the processors execute the executable code, the steps of the geological disaster monitoring method based on deep learning and spatial data query as described in any one of claims 1 to 5 are implemented.

Images