A method for mining clues and warning of river and lake encroachment based on multimodal images

Abstract

The present invention discloses a method for mining clues and warning of river and lake encroachment based on multimodal images. Firstly, a training sample set is constructed with the encroachment area of ​​rivers and lakes for time-series high-resolution satellite remote sensing images, and a residual network is used to construct a river and lake encroachment clue mining model for edge detection of remote sensing images of four kinds of disorderly encroachment of rivers and lakes. The range boundaries and encroachment time range of clues of four kinds of disorderly behaviors of rivers and lakes are mined, and further shore-based monitoring video images of the mining clue area are retrieved. The LSP graph convolutional network model is used to identify the type, location and time of river and lake encroachment, and form alarm list information, so as to provide support for timely and accurate supervision of the four kinds of disorderly encroachment of rivers and lakes.

Description

Technical Field

[0001] The present invention belongs to the technical field of remote sensing image intelligent processing, and in particular relates to the design of a river and lake encroachment clue mining and alarm method based on multimodal images. Background Art

[0002] Rivers and lakes are important natural spaces for ensuring flood control safety. With the expansion of urbanization, the natural space of rivers and lakes has been encroached upon. Illegal occupation, mining, dumping and construction (the four chaotic behaviors) frequently occur within rivers and lakes, posing a serious threat to flood discharge and the ecological environment safety of rivers and lakes. Because the four chaotic behaviors are numerous and widespread and relatively hidden, it is difficult to discover clues and issue warning information in a timely manner, which restricts the efficiency of precise supervision of rivers and lakes.

[0003] Traditional clues of river and lake encroachment mostly rely on manual reporting. This method requires a lot of public opinion information, and due to the untimely information, it is often impossible to achieve accurate supervision. There are also related studies that use manual river patrols to dig clues and issue alarms, but due to the large range of rivers and lakes, a lot of manpower is required, clue mining and alarming time is long, and the automation level is low, which is difficult to meet the requirements of smart river and lake supervision. Some studies use time-series satellite high-resolution satellite remote sensing data to carry out river and lake supervision, but due to the frequency of remote sensing monitoring, it is impossible to dynamically discover clues and alarms in real time. There are also studies that use video systems built on the shores around rivers and lakes to detect four illegal behaviors in real time, but the clues mined cannot achieve intelligent alarms. Summary of the invention

[0004] The purpose of the present invention is to solve the problem that it is difficult to conduct real-time and accurate supervision of the four disorderly behaviors in rivers and lakes over a large area in the prior art, and a method for mining clues and warning of river and lake encroachment based on multimodal images is proposed.

[0005] The technical solution of the present invention is: a method for mining clues and warning of river and lake encroachment based on multimodal images, comprising the following steps:

[0006] S1. Obtain high-resolution satellite remote sensing data of the river and lake supervision area, and pre-process the high-resolution satellite remote sensing data.

[0007] S2. Based on the residual network, a river and lake encroachment clue mining model is constructed, and the central coordinate position and the start and end time of the river and lake encroachment range are obtained from the pre-processed high-resolution satellite remote sensing data through the river and lake encroachment clue mining model.

[0008] S3. Obtain the shore-based video surveillance image of the river and lake supervision range according to the central coordinate position of the river and lake encroachment range and the start and end time of the encroachment, and obtain the alarm list information of river and lake encroachment based on the shore-based video surveillance image and the LSP graph convolutional network.

[0009] Furthermore, the high-resolution satellite remote sensing data in step S1 is optical remote sensing data with a spatial resolution better than 2 m and without cloud or snow cover.

[0010] Furthermore, in step S1, the high-resolution satellite remote sensing data is preprocessed including geometric precision correction, atmospheric correction and cloud mask processing.

[0011] Further, step S2 includes the following sub-steps:

[0012] S21. Delineate the types and ranges of river and lake encroachment in the preprocessed high-resolution satellite remote sensing data as samples to construct a training sample set X.

[0013] S22. Taking the actual river and lake encroachment range as the target set, the training sample set X is input into the residual network, and the residual network is trained using the back propagation algorithm and the gradient descent method to obtain the river and lake encroachment clue mining model.

[0014] S23. Input the pre-processed high-resolution satellite remote sensing data into the river and lake encroachment clue mining model, output the river and lake encroachment range, and extract the central coordinate position and the start and end time of the river and lake encroachment range through the image information.

[0015] Further, step S23 includes the following sub-steps:

[0016] S231, using the preprocessed high-resolution satellite remote sensing data as an input image, performing a convolution operation on the input image to obtain a convolution output feature map:

[0017]

[0018] where Y i,j Represents the element of the convolution output feature map at position (i, j), X i+m,j+n Represents the pixel value of the input image at position (i+m,j+n), K m,n Represents the value of the convolution kernel at position (m,n).

[0019] S232. Activate the convolution output feature map through the ReLU function to obtain the activated feature map:

[0020] Y′ i,j =max(0,Y i,j )

[0021] where Y′ i,j Represents the element at position (i, j) of the activated feature map.

[0022] S233, perform a maximum pooling operation on the activated feature map to obtain a pooled output feature map:

[0023]

[0024] Where Y″ i,j Represents the element at the (i, j) position of the output feature map after pooling, N represents the pooling window, b and c represent the element index in the pooling window N, the value range of b and c is (0, a-1), and a represents the pooling window size.

[0025] S234, perform residual block calculation on the output feature map after pooling to obtain a residual block output feature map Z:

[0026] Z=F(Y″ i,j ,W i )+Y″ i,j

[0027] Where F(·) represents the residual mapping, W i Represents a set of weights.

[0028] S235. Classify or regress the residual block output feature map Z through the fully connected layer to obtain the river and lake encroachment range:

[0029] y=f(WZ+b)

[0030] Where y represents the output of the fully connected layer, f(·) represents the activation function, W represents the weight matrix, and b represents the bias vector. The remote sensing image is divided into a binary image through y, where 0 represents the non-occupied area and 1 represents the occupied area. According to the coordinate range of the occupied area, the central coordinate position (X0, Y0) of the river and lake occupied range is extracted, and the image time before and after the image occupation is obtained as the occupation start time (T1, T2).

[0031] Further, step S3 includes the following sub-steps:

[0032] S31. According to the central coordinate position of the river and lake encroachment range and the start and end time of the encroachment, the shore-based video surveillance image of the river and lake supervision range is retrieved so that it can completely cover the river and lake encroachment area.

[0033] S32. Mark the shore-based video surveillance images with encroachment and non-encroachment data labels according to the encroachment type, and obtain a video image feature set Y of river and lake encroachment data labels.

[0034] S33. Construct a multimodal video image training sample library Dataset = {X, Y} according to the training sample set X and the river and lake encroachment data label video image feature set Y.

[0035] S34. Train the LSP graph convolutional network through the multimodal video image training sample library Dataset to obtain the river and lake encroachment alarm model.

[0036] S35. Input the multimodal video image data within the river and lake supervision range into the river and lake encroachment alarm model, and output the type, location and time of the river and lake encroachment to form an alarm list information of the river and lake encroachment.

[0037] Further, step S34 includes the following sub-steps:

[0038] S341, each sample point x in the multimodal video image training sample library Dataset i It is connected with its neighborhood sample points to form a graph G(υ,ε), where v represents the data set composed of nodes on the graph, and ε represents the set of edges, that is, the neighborhood point x j With sample point x i distance.

[0039] S342, calculate sample point x i With neighboring point x j The local structure vector LS i,j :

[0040]

[0041] Where (j,i) represents the connected neighborhood point x j With sample point x i , SIM(·) represents the function that quantifies the similarity of node pairs, and ‖·‖2 represents the 2-norm.

[0042] S343, according to the local structure vector LS i,j Calculate the similarity S of the local structure of each sample point i :

[0043]

[0044] Where D KL (·) represents relative entropy, represents the local structure vector of the student model in the LSP graph convolutional network, Represents the local structure vector of the teacher model in the LSP graph convolutional network.

[0045] S344, based on the similarity S of the local structure i Construct the loss function L of the LSP graph convolutional network:

[0046]

[0047] L=H(y s ,y r )+λL LSP

[0048] Where L LSPrepresents the local structure loss function, M represents the number of samples, H(·) represents the cross entropy loss function, y r Represents the true sample data label, y s represents the value predicted by the student model, and λ represents the hyperparameter that balances the two losses.

[0049] S345. The LSP graph convolutional network is trained with the goal of minimizing the loss function L to obtain a river and lake encroachment alarm model.

[0050] The beneficial effects of the present invention are:

[0051] (1) The present invention constructs a method for mining clues and warning of river and lake encroachment based on satellite remote sensing and shore-based video, breaking through the technical bottleneck of the original large-scale river and lake disorderly behaviors that are difficult to monitor in real time and accurately.

[0052] (2) The present invention constructs a river and lake encroachment clue mining model based on high-resolution remote sensing images and residual network ResNet, which realizes the rapid mining of clues range boundaries and encroachment time of the four disorderly behaviors of rivers and lakes.

[0053] (3) The present invention adopts the shore-based video continuous imaging method to obtain continuous video image data within the period of encroachment clues, and adopts the LSP graph convolutional network algorithm to accurately obtain the warning information of river and lake encroachment behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 The figure shows a flow chart of a method for mining river and lake encroachment clues and warning based on multimodal images provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Now, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings. It should be understood that the embodiments shown and described in the accompanying drawings are only exemplary and are intended to explain the principles and spirit of the present invention, rather than to limit the scope of the present invention.

[0056] The embodiment of the present invention provides a method for mining clues and warning of river and lake encroachment based on multimodal images, such as Figure 1 As shown, the following steps S1 to S3 are included:

[0057] S1. Obtain high-resolution satellite remote sensing data of the river and lake supervision area, and pre-process the high-resolution satellite remote sensing data.

[0058] In the embodiment of the present invention, according to the application requirements of river and lake disorder supervision, high-resolution satellite remote sensing data is optical remote sensing data with a spatial resolution better than 2m and no cloud or snow cover, including remote sensing data of Gaofen-1, Gaofen-2, Gaofen-6, etc. Preprocessing of high-resolution satellite remote sensing data includes geometric precision correction, atmospheric correction and cloud mask processing.

[0059] S2. Based on the residual network (ResNet), a river and lake encroachment clue mining model is constructed, and the central coordinate position and the start and end time of the river and lake encroachment range are obtained from the preprocessed high-resolution satellite remote sensing data through the river and lake encroachment clue mining model.

[0060] Step S2 includes the following sub-steps S21 to S23:

[0061] S21. Delineate the types and ranges of river and lake encroachment in the preprocessed high-resolution satellite remote sensing data as samples to construct a training sample set X.

[0062] S22. Taking the actual river and lake encroachment range as the target set, the training sample set X is input into the residual network, and the residual network is trained using the back propagation algorithm and the gradient descent method to obtain the river and lake encroachment clue mining model.

[0063] In an embodiment of the present invention, the river and lake encroachment clue mining model is composed of multiple residual blocks, each residual block includes two or more convolutional layers and a skip connection.

[0064] S23. Input the pre-processed high-resolution satellite remote sensing data into the river and lake encroachment clue mining model, output the river and lake encroachment range, and extract the central coordinate position and the start and end time of the river and lake encroachment range through the image information.

[0065] Step S23 includes the following sub-steps S231 to S235:

[0066] S231, using the preprocessed high-resolution satellite remote sensing data as an input image, performing a convolution operation on the input image to obtain a convolution output feature map:

[0067]

[0068] where Y i,j Represents the element of the convolution output feature map at position (i, j), X i+m,j+n Represents the pixel value of the input image at position (i+m,j+n), K m,n Represents the value of the convolution kernel at position (m,n). The purpose of the convolution operation is to extract local features of remote sensing images.

[0069] S232. Activate the convolution output feature map through the ReLU function to obtain the activated feature map:

[0070] Y′ i,j =max(0,Y i,j )

[0071] where Y′ i,jRepresents the element at position (i, j) of the activated feature map.

[0072] S233, perform a maximum pooling operation on the activated feature map to obtain a pooled output feature map:

[0073]

[0074] Where Y″ i,j Represents the element at the (i, j) position of the output feature map after pooling, N represents the pooling window, b and c represent the element index in the pooling window N, the value range of b and c is (0, a-1), and a represents the pooling window size.

[0075] S234, perform residual block calculation on the output feature map after pooling to obtain a residual block output feature map Z:

[0076] Z=F(Y″ i，j ,W i )+Y″ i,j

[0077] Where F(·) represents the residual mapping, W i Represents a set of weights.

[0078] S235. Classify or regress the residual block output feature map Z through the fully connected layer to obtain the river and lake encroachment range:

[0079] y=f(WZ+b)

[0080] Where y represents the output of the fully connected layer, f(·) represents the activation function, such as ReLU, sigmoid or softmax, W represents the weight matrix, and b represents the bias vector. The remote sensing image is divided into a binary image through y, where 0 represents the non-occupied area and 1 represents the occupied area. According to the coordinate range of the occupied area, the central coordinate position (X0, Y0) of the river and lake occupied range is extracted, and the image time before and after the image occupation is obtained as the occupation start time (T1, T2).

[0081] S3. Obtain the shore-based video surveillance image of the river and lake supervision range according to the central coordinate position of the river and lake encroachment range and the start and end time of the encroachment, and obtain the alarm list information of river and lake encroachment based on the shore-based video surveillance image and the LSP graph convolutional network.

[0082] Step S3 includes the following sub-steps S31 to S35:

[0083] S31. According to the central coordinate position of the river and lake encroachment range and the start and end time of the encroachment, the shore-based video surveillance image of the river and lake supervision range is retrieved so that it can completely cover the river and lake encroachment area.

[0084] S32. Mark the shore-based video surveillance images with encroachment and non-encroachment data labels according to the encroachment type, and obtain a video image feature set Y of river and lake encroachment data labels.

[0085] In the embodiment of the present invention, the expert experience is combined to determine the basis for river and lake encroachment, and the shore-based video surveillance images are marked with encroachment and non-encroachment data labels according to the four types of behaviors: illegal occupation, illegal mining, illegal dumping and illegal construction.

[0086] S33. Construct a multimodal video image training sample library Dataset = {X, Y} according to the training sample set X and the river and lake encroachment data label video image feature set Y.

[0087] S34. Train the LSP graph convolutional network through the multimodal video image training sample library Dataset to obtain the river and lake encroachment alarm model.

[0088] Step S34 includes the following sub-steps S341 to S345:

[0089] S341, each sample point x in the multimodal video image training sample library Dataset i It is connected with its neighborhood sample points to form a graph G(υ,ε), where υ represents the data set composed of nodes on the graph, and ε represents the set of edges, that is, the neighborhood point x j With sample point x i distance.

[0090] S342, calculate sample point x i With neighboring point x j The local structure vector LS i,j :

[0091]

[0092] Where (j,i) represents the connected neighborhood point x j With sample point x i , SIM(·) represents a function that quantifies the similarity of a pair of nodes, which can be defined as the Euclidean distance between two features, and ‖·‖2 represents the 2-norm. LSP (local structure preserving) is used to describe graph topology information and maintain local structure information during training. In the embodiment of the present invention, local structure information is represented by a local structure vector.

[0093] S343, according to the local structure vector LS i,j Calculate the similarity S of the local structure of each sample point i :

[0094]

[0095] Where DKL (·) represents relative entropy, which is a distance formula used to measure two probability distributions. In the embodiment of the present invention, it is used to measure the similarity of the local structures of the teacher model and the student model. represents the local structure vector of the student model in the LSP graph convolutional network, Represents the local structure vector of the teacher model in the LSP graph convolutional network.

[0096] In the embodiment of the present invention, the teacher model and the student model have the same input data and are trained simultaneously. During the training process, the knowledge gained from the teacher model training is transferred to the student model to improve operating efficiency and generalization ability.

[0097] S344, based on the similarity S of the local structure i Construct the loss function L of the LSP graph convolutional network:

[0098]

[0099] L=H(y s ,y r )+λL LSP

[0100] Where L LSP represents the local structure loss function, M represents the number of samples, H(·) represents the cross entropy loss function, y r Represents the true sample data label, y s represents the value predicted by the student model, and λ represents the hyperparameter that balances the two losses.

[0101] S345. The LSP graph convolutional network is trained with the goal of minimizing the loss function L to obtain a river and lake encroachment alarm model.

[0102] In the embodiment of the present invention, the influence of sample capacity on model accuracy is quantitatively analyzed experimentally, and the training sample capacity is adjusted accordingly. The teacher model is set to 10 layers, the student model is set to 8 layers, the learning rate is 0.008, and λ is set to 500.

[0103] S35. Input the multimodal video image data within the river and lake supervision range into the river and lake encroachment alarm model, and output the type, location and time of the river and lake encroachment to form an alarm list information of the river and lake encroachment.

[0104] The alarm list information of river and lake encroachment can support the precise supervision of the four disorderly behaviors in rivers and lakes, and improve the efficiency of automated inspection and disposal of rivers and lakes.

[0105] Those skilled in the art will appreciate that the embodiments described herein are intended to help readers understand the principles of the present invention, and should be understood that the protection scope of the present invention is not limited to such specific statements and embodiments. Those skilled in the art can make various other specific variations and combinations that do not deviate from the essence of the present invention based on the technical revelations disclosed by the present invention, and these variations and combinations are still within the protection scope of the present invention.

Claims

1. A method for mining clues and warning of river and lake encroachment based on multimodal images, characterized in that: The following steps are involved: S1. Obtain high-resolution satellite remote sensing data of the river and lake supervision area, and pre-process the high-resolution satellite remote sensing data; S2. Construct a river and lake encroachment clue mining model based on the residual network, and use the river and lake encroachment clue mining model to mine the central coordinate position and encroachment start and end time of the river and lake encroachment range from the pre-processed high-resolution satellite remote sensing data; S3. Obtain shore-based video surveillance images of the river and lake supervision range according to the central coordinate position of the river and lake encroachment range and the start and end time of the encroachment, and obtain the alarm list information of river and lake encroachment based on the shore-based video surveillance images and the LSP graph convolutional network; The step S2 comprises the following sub-steps: S21. Delineate the types and ranges of river and lake encroachment in the preprocessed high-resolution satellite remote sensing data as samples to construct a training sample set ; S22, taking the actual river and lake encroachment range as the target set, the training sample set Input the residual network, use the back propagation algorithm and gradient descent method to train the residual network, and obtain the river and lake encroachment clue mining model; S23, inputting the pre-processed high-resolution satellite remote sensing data into the river and lake encroachment clue mining model, outputting the river and lake encroachment range, and extracting the center coordinate position and the encroachment start and end time of the river and lake encroachment range through the image information; The step S23 comprises the following sub-steps: S231, using the preprocessed high-resolution satellite remote sensing data as an input image, performing a convolution operation on the input image to obtain a convolution output feature map: in Indicates that the convolution output feature map is The elements of the position, Indicates that the input image is The pixel value of the position, Indicates that the convolution kernel is The value of the position; S232. Activate the convolution output feature map through the ReLU function to obtain the activated feature map: in Indicates that the activated feature map is Elements of position; S233, perform a maximum pooling operation on the activated feature map to obtain a pooled output feature map: in Indicates that the output feature map after pooling is The elements of the position, represents the pooling window, and Represents the pooling window The index of the element in and The value range is , Indicates the pooling window size; S234, perform residual block calculation on the output feature map after pooling to obtain a residual block output feature map : in represents the residual mapping, represents a weight set; S235, output feature map of residual block through full connection layer Perform classification or regression to obtain the scope of river and lake encroachment: in represents the output of the fully connected layer, represents the activation function, represents the weight matrix, represents the bias vector; through The remote sensing image is divided into a binary map, where 0 is the non-occupied area and 1 is the occupied area. According to the coordinate range of the occupied area, the center coordinate position of the river and lake occupied range is extracted. , the image time before and after the image occupation is obtained as the occupation start time .

2. The method for mining river and lake encroachment clues and warning based on multimodal images according to claim 1 is characterized in that: The high-resolution satellite remote sensing data in step S1 is optical remote sensing data with a spatial resolution better than 2 m and without cloud or snow cover.

3. The method for mining river and lake encroachment clues and warning based on multimodal images according to claim 1 is characterized in that: The preprocessing of high-resolution satellite remote sensing data in step S1 includes geometric precision correction, atmospheric correction and cloud mask processing.

4. The method for mining river and lake encroachment clues and warning based on multimodal images according to claim 1 is characterized in that: The step S3 comprises the following sub-steps: S31, retrieve the shore-based video surveillance images of the river and lake supervision area according to the central coordinate position of the river and lake encroachment area and the start and end time of the encroachment, so that it can completely cover the river and lake encroachment area; S32, marking the shore-based video surveillance images with encroachment and non-encroachment data labels according to the encroachment type, and obtaining the river and lake encroachment data label video image feature set ; S33, according to the training sample set and river and lake encroachment data label video image feature set Constructing a multimodal video image training sample library ; S34. Training sample library through multimodal video images The LSP graph convolutional network is trained to obtain a river and lake encroachment warning model; S35. Input the multimodal video image data within the river and lake supervision range into the river and lake encroachment alarm model, and output the type, location and time of the river and lake encroachment to form an alarm list information of the river and lake encroachment.

5. The method for mining river and lake encroachment clues and warning based on multimodal images according to claim 4 is characterized in that: The step S34 includes the following sub-steps: S341, multimodal video image training sample library Each sample point in Connect with its neighborhood sample points to form a graph ,in Represents a dataset consisting of nodes on the graph, Represents the set of edges, i.e., the neighborhood points With sample points distance; S342, calculate sample points With neighboring points The local structure vector : in Represents connecting neighboring points With sample points The edge, represents the function that quantifies the similarity of node pairs, represents the 2-norm; S343, according to the local structure vector Calculate the similarity of the local structure of each sample point : in represents the relative entropy, represents the local structure vector of the student model in the LSP graph convolutional network, Represents the local structure vector of the teacher model in the LSP graph convolutional network; S344, based on the similarity of local structure Constructing the loss function of LSP graph convolutional network : in represents the local structure loss function, represents the number of samples, represents the cross entropy loss function, represents the true value sample data label, represents the value predicted by the Student model, represents the hyperparameter that balances the two losses; S345. Loss function The LSP graph convolutional network is trained with the minimum as the goal to obtain the river and lake encroachment alarm model.

Images