A method for selecting key frames of geotemporal data

Abstract

The application discloses a kind of key frame selection methods of geospatial-temporal data, it is related to geographic information technology and deep learning field.The method includes: using the auto-encoder based on convolutional neural network and generative adversarial neural network to learn the semantic embedding of geospatial-temporal data, obtain the low-dimensional vector representation containing data semantic features;The cosine distance between the low-dimensional vector representation of data is calculated, the semantic similarity of data is measured, and an association graph is constructed;Consider the proximity distance penalty and the aggregate statistical information of data, construct an enhanced association graph;Based on the number of key frames given by user, find the shortest path of fixed length on the enhanced association graph, so as to select key frame.The method can quickly select key time points from large-scale geospatial-temporal data, has good data summarization and expression ability, and can provide data trend and feature information by preferentially transmitting key frames under the condition that user network bandwidth is limited.

Description

Technical Field

[0001] This invention relates to the fields of geographic information technology and deep learning, specifically to a method for selecting keyframes in geographic time-series data. Background Technology

[0002] Remote sensing satellites, physical monitoring equipment, and urban sensors constantly generate massive amounts of geographic time-series data, which is crucial for meteorology, environment, transportation, and urban planning. However, this geographic time-series data is enormous and is often provided to users via web-based data portals. Unfortunately, users' network bandwidth is often insufficient for real-time exploration, forcing them to wait considerable time to download the data locally before observation and analysis. This severely impacts the efficiency of data exploration and analysis, hindering the maximization of the data's value.

[0003] Therefore, by selecting keyframes, users can quickly download portions of data to gain a data overview and insights into trends and characteristics. The problem of keyframe selection has been studied in areas such as geographic time-series data, scientific voxel data, and video summarization. Commonly used keyframe selection methods include those based on dimensionality reduction and clustering, those based on Dynamic Time Wrapping (DTW), and algorithms based on information entropy and dynamic programming. These traditional methods struggle to capture semantic information in the data, and the evaluation criteria for keyframe selection are relatively singular, failing to meet the diverse needs of users in geographic time-series data analysis. To address these issues, this invention proposes a method for selecting keyframes in geographic time-series data. Summary of the Invention

[0004] The purpose of this invention is to propose a novel keyframe selection method for geographic temporal data to address the problems mentioned in the background art. This invention uses an autoencoder based on convolutional neural networks and generative adversarial neural networks to capture the semantic features of the data, and constructs an enhanced association graph that considers proximity penalties and data aggregation information. Then, a shortest path algorithm is used to select more general and representative keyframes, improving the efficiency of data exploration for users.

[0005] To achieve the above objectives, the present invention adopts the following technical solution: A method for selecting keyframes from geographic time-series data includes the following steps: S1. Semantic feature extraction: Low-dimensional vector representation of geographic spatiotemporal data is obtained through an autoencoder based on convolutional neural networks and generative adversarial neural networks; S2. Association Graph Generation: Calculate the cosine distance between pairs of data using low-dimensional vector representations to measure the semantic similarity of the data and construct an association graph; S3. Enhanced Association Graph Generation: Based on the association graph, further consider the proximity penalty and statistical trend information after data aggregation to construct an enhanced association graph; S4. Fixed-length shortest path calculation: Based on the number of keyframes k given by the user, find the fixed-length shortest path on the enhanced association graph to select keyframes.

[0006] Preferably, S1 specifically includes the following: S1.1, An autoencoder based on convolutional neural networks and generative adversarial neural networks is used as the model; the model specifically includes an encoder based on convolutional neural networks, a decoder based on convolutional neural networks, and a discriminator based on convolutional neural networks; S1.2 Use the data that requires keyframe selection to train the neural network; S1.3, The data recovered by the decoder from the latent space code generated by the encoder and the original image are made as close as possible as possible as the training target of the encoder and decoder, and the mean squared error is used as the loss function. S1.4. The objective of the discriminator is to correctly determine whether a certain data is the original data or the data recovered by the decoder from the latent space encoding. The discriminant loss function used is: ; S1.5. Using the outputs of convolutional neural networks at different layers in the encoder, the outputs are concatenated into a single vector as the semantic encoding of a single frame of data. Preferably, the association graph in S2 specifically includes the following characteristics: a) The association graph is a complete graph, where each node is a frame of data, i.e., a single point in time; the weight of each edge is a specific floating-point number, called the cost; b) The cost of each edge in the association graph is obtained by calculating the cosine similarity of the semantic encoding vectors of two data frames. The specific calculation formula is as follows:

[0007] in, Z i , Z j Representing a frame i , j The implicit encoding.

[0008] Preferably, the enhanced association graph in S3 specifically includes the following characteristics: a) The enhanced association graph has the same structure as the association graph, but the edge weights are different; b) The weights of the edges in the enhanced association graph additionally consider proximity penalty, that is, for frames that are too close in time, the cost between the corresponding data frames is increased inversely proportional to the distance, specifically expressed as:

[0009] in, σ This is a hyperparameter used to control the decay rate of the distance penalty. n The total number of data frames within the selected time range. k Specify the number of frames to select for the user; c) The weights of the edges in the enhanced association graph additionally consider the differences in aggregated information between pairwise data frames; the aggregation includes three aggregation methods: average, maximum, and minimum. The absolute difference in the aggregated values ​​between pairwise data frames is calculated and normalized, and the cost between corresponding data frames is increased inversely proportional to this difference. The specific function is expressed as follows:

[0010] in, and Representing a frame and frame Normalized aggregate value; d) The weights of the edges in the enhanced association graph, i.e. the cost, are a linear combination of the cosine distance of the semantic encoding vector, the difference in aggregated information, and the proximity penalty.

[0011] Preferably, S4 specifically includes the following: S4.1 The user specifies the number of keyframes k and the time range T to be focused on; where k is an integer, less than the number of data frames contained in T; S4.2 Calculate the shortest path of length k from the start data frame to the end data frame using the Dijkstra shortest path algorithm; S4.3 For selection tasks that exceed the threshold U, first perform uniform sampling on the original sequence to obtain a subsequence, and then perform shortest path search.

[0012] Compared with existing technologies, the present invention provides a method for selecting keyframes of geographic time-series data, which has the following beneficial effects: The method for selecting keyframes in geographic time-series data described in this invention fully captures the semantic information of the data, enabling the selection of keyframes with good generalization capabilities and applicability to various data exploration tasks. By adjusting the cost weights in the enhanced association graph, users can make keyframes more likely to summarize spatial semantics and feature change information, or more likely to summarize the changing trends of aggregated data values. The proximity penalty term effectively avoids the keyframe collapse phenomenon introduced by the global optimality of the shortest path (i.e., the situation where keyframes are concentrated in a specific time period due to drastic changes in data within a short period). This invention is applicable to keyframe selection of any raster geographic data, suitable for common data exploration tasks in geographic data systems, and has broad application prospects. Attached Figure Description

[0013] Figure 1 This is a flowchart of a method for selecting keyframes in geographic time-series data according to the present invention.

[0014] Figure 2 This is a flowchart of semantic feature extraction in a method for selecting keyframes of geographic temporal data according to the present invention; Figure 3 This is a schematic diagram of the weights of each cost function and the linear mixing result in the fully connected enhanced association graph of the geographic time series data key frame selection method described in this invention, represented by an adjacency matrix; Figure 4 To utilize the keyframe selection method for geographic time-series data described in this invention, the selection results of 12 keyframes are obtained from 240 frames of atmospheric cloud brightness temperature data. Detailed Implementation

[0015] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0016] Example 1 The flowchart of the keyframe selection method for geographic time series data described in this invention is as follows: Figure 1 As shown, the specific steps are as follows: Step 1: Semantic feature extraction: such as Figure 2 As shown, a low-dimensional vector representation of geospatial data is obtained through an autoencoder based on a convolutional neural network and a generative adversarial neural network: a) Neural Network Architecture: An autoencoder based on convolutional neural networks and generative adversarial neural networks is used as the model. Specifically, it includes an encoder containing multiple convolutional neural networks, a decoder based on convolutional neural networks, and a discriminator based on convolutional neural networks. The encoder, decoder, and discriminator all contain multiple convolutional layers, which respectively encode data into a low-dimensional latent space, restore data from the low-dimensional latent space to its original size, and evaluate the reconstruction quality of the data. b) Neural Network Training: The neural network is trained using data from which keyframe selection is required. The training objective for the encoder and decoder is to make the data recovered by the decoder from the latent space encoding generated by the encoder as close as possible to the original image, using mean squared error (MSE) as the loss function. The optimization objective for the discriminator is to correctly determine whether a given data point is the original data or data recovered by the decoder from the latent space encoding, using the discriminative loss function: This network has a certain degree of generalization and can be used on data not in the training set; c) Semantic Feature Extraction: The outputs of different layers of the convolutional neural network in the encoder are concatenated into a single vector, which serves as the semantic code for a single frame of data. The dimensionality of the semantic code is determined by the complexity of the data. Step Two: Association graph generation: Calculate the low-dimensional vector representation of the data to represent the cosine distance between each pair of data, measure the semantic similarity of the data, and construct an association graph. a) Construction of the association graph: The association graph is a complete graph, where each node represents a frame of data, i.e., a single point in time. The weight of each edge is a specific floating-point number, called the cost; b) Cost Calculation: The cost of each edge in the association graph is obtained by calculating the cosine similarity of the semantic encoding vectors of two data frames, using the following formula: in and For frames and The implicit encoding; Step 3: Enhanced Association Graph Generation: Building upon the association graph, further considering proximity penalty and statistical trend information after data aggregation, an enhanced association graph is constructed. a) Construction of augmented association graph: The augmented association graph has the same structure as the association graph, but the edge weights are different; b) Proximity Penalty: Edge weights are additionally penalized based on proximity. That is, for frames that are too close in time, the cost between the corresponding data frames is increased inversely proportional to the distance. Specifically, the formula is: in This is a hyperparameter used to control the decay rate of the distance penalty. The total number of data frames within the selected time range. Specify the number of frames to select for the user; c) Aggregate Statistical Changes: Edge weights additionally consider the differences in aggregated information between pairwise data frames. Aggregation includes three methods: average, maximum, and minimum. The absolute difference in aggregated values ​​between pairwise data frames is calculated and normalized. The cost between corresponding data frames is increased inversely proportional to this difference. Specifically, the formula is: in and Representing a frame and frame The normalized aggregate value.

[0017] d) Linear combination of costs: The weights of edges in the enhanced association graph, i.e., the costs, are a linear combination of the cosine distance of the semantic encoding vectors, the difference in aggregated information, and the proximity penalty. The weight of the proximity penalty should be set by the system, while the weight combination of the semantic cost and the statistical cost should be determined by the user. The generated enhanced association graph is a fully connected graph, and the weights between any two nodes are as follows: Figure 3 As shown; from right to left, they represent the spatial semantic difference costs ( C struc ), cost of aggregated statistical results ( C stat ), distance penalty cost ( C dist The linear weighted result of the three costs; Step Four: Fixed-length shortest path calculation: Based on the number of keyframes k given by the user, a fixed-length shortest path is found on the enhanced association graph to select keyframes. a) Define the search scope: The user needs to specify the number of keyframes k and the time range T to be focused on. k is an integer and should be less than the total number of data frames included in T. b) Shortest Path Calculation: The Dijkstra's shortest path algorithm is used to calculate the shortest path of length k from the start data frame to the end data frame. During this process, a priority queue is used to record the path search status, containing state nodes. Each state node includes the currently relaxing graph node, the length of the current path, and the total cost of the current time frame. A preorder array is used to record the relaxation history, and the user backtracks to the shortest path when a stopping condition is met. The stopping condition is that the currently relaxing graph node in the top state node of the priority queue is the end node, and the total length of the current sequence is k. c) Long-range uniform sampling: Considering the case of excessively large data frames. A specific threshold U is set. When the time length in T exceeds U, a subsequence of length U is first uniformly sampled over T, and then the aforementioned shortest path search algorithm is executed on the subsequence. Since the time complexity of this method is O(n log n),... Real-time interactive performance can usually be achieved when U < 2000.

[0018] In summary, to obtain such Figure 4 The results shown Figure 4 This paper demonstrates the selection of 12 keyframes from 240 frames of cloud brightness temperature data using the method described above. The data was collected during a hurricane event (August-September 2019), and the keyframes effectively reveal the hurricane's formation and northward movement.

[0019] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for selecting keyframes in geographic time-series data, characterized in that, Includes the following steps: S1. Semantic Feature Extraction: Low-dimensional vector representations of geospatial data are obtained through an autoencoder based on convolutional neural networks and generative adversarial neural networks; specifically including the following: S1.1, An autoencoder based on convolutional neural networks and generative adversarial neural networks is used as the model; the model specifically includes an encoder based on convolutional neural networks, a decoder based on convolutional neural networks, and a discriminator based on convolutional neural networks; S1.2 Use the data that requires keyframe selection to train the neural network; S1.3, The data recovered by the decoder from the latent space code generated by the encoder and the original image are made as close as possible as possible as the training target of the encoder and decoder, and the mean squared error is used as the loss function. S1.

4. The objective of the discriminator is to correctly determine whether a certain data is the original data or the data recovered by the decoder from the latent space encoding. The discriminant loss function used is: ； S1.

5. Using the outputs of convolutional neural networks at different layers in the encoder, the outputs are concatenated into a single vector as the semantic encoding of a single frame of data. S2. Association Graph Generation: Calculate the cosine distance between pairs of data using low-dimensional vector representations to measure the semantic similarity of the data and construct an association graph; S3. Enhanced Association Graph Generation: Based on the association graph, further considering proximity penalty and statistical trend information after data aggregation, an enhanced association graph is constructed; the enhanced association graph specifically includes the following characteristics: a) The enhanced association graph has the same structure as the association graph, but the edge weights are different; b) The weights of the edges in the enhanced association graph additionally consider proximity penalty, that is, for frames that are too close in time, the cost between the corresponding data frames is increased inversely proportional to the distance, specifically expressed as: in, σ This is a hyperparameter used to control the decay rate of the distance penalty. n The total number of data frames within the selected time range. k Specify the number of frames to select for the user; c) The weights of the edges in the enhanced association graph additionally consider the differences in aggregated information between pairwise data frames; the aggregation includes three aggregation methods: average, maximum, and minimum. The absolute value of the difference between the aggregated values ​​of each pair of data frames is calculated and normalized. The cost between the corresponding data frames is increased inversely proportional to this difference. The specific function is expressed as follows: in, and Representing a frame and frame Normalized aggregate value; d) The weights of the edges in the enhanced association graph, i.e. the cost, are a linear combination of the cosine distance of the semantic encoding vector, the difference in aggregated information, and the proximity penalty. S4. Fixed-length shortest path calculation: Based on the number of keyframes k given by the user, find the fixed-length shortest path on the enhanced association graph to select keyframes.

2. The method for selecting keyframes in geographic time-series data according to claim 1, characterized in that, The association graph described in S2 specifically includes the following characteristics: a) The association graph is a complete graph, where each node is a frame of data, i.e., a single point in time; the weight of each edge is a specific floating-point number, called the cost; b) The cost of each edge in the association graph is obtained by calculating the cosine similarity of the semantic encoding vectors of two data frames. The specific calculation formula is as follows: in, Z i , Z j Representing a frame i , j The implicit encoding.

3. The method for selecting keyframes in geographic time-series data according to claim 1, characterized in that, S4 specifically includes the following: S4.1 The user specifies the number of keyframes k and the time range T to be focused on; where k is an integer, less than the number of data frames contained in T; S4.2 Calculate the shortest path of length k from the start data frame to the end data frame using the Dijkstra shortest path algorithm; S4.3 For selection tasks that exceed the threshold U, first perform uniform sampling on the original sequence to obtain a subsequence, and then perform shortest path search.

Images