Methods for detecting strong winds

Abstract

The application discloses a detection method for strong wind weather, comprising: acquiring a plurality of image frames; determining a feature matrix based on the plurality of image frames; determining a time sequence feature matrix of two different time scales based on the plurality of image frames and the feature matrix; fusing the time sequence feature matrix of different time scales to obtain a fusion feature matrix and obtaining a strong wind detection result based on the fusion feature matrix. The scheme provided by the application can more comprehensively and accurately identify the strong wind state, and effectively improves the reliability and robustness of detection, especially in a complex environment.

Description

Technical Field

[0001] This invention relates to the field of weather detection, specifically to a method for detecting windy weather. Background Technology

[0002] In real-world applications, strong winds can easily cause significant global motion changes in camera equipment, such as shaking and swaying. These motion changes not only directly reduce image clarity and degrade picture quality, but also create dynamic features with clear time dependence in the video sequence. This means that the motion states between adjacent frames or even multiple frames are closely correlated in time, exhibiting regular changes that evolve over time.

[0003] The video-based wind detection task aims to accurately identify features caused by strong winds. Therefore, this task not only needs to focus on the appearance information of a single frame image (such as the degree of blur caused by shaking, changes in the static features of objects in the scene, etc.), but more importantly, it needs to focus on the motion patterns caused by wind that evolve over time and the structured relationships between different frames.

[0004] However, traditional methods often rely on manual feature extraction of image changes or deep learning classification methods based on single-frame images. These methods struggle to accurately capture multi-scale motion features that change over time. For example, they cannot effectively identify sudden jerks in a short period of time, nor can they capture the gradual trend of motion intensity over a long time sequence. Especially in complex environments (such as local disturbances such as swaying trees or moving vehicles, or occlusions caused by buildings or obstacles), they are easily affected by interference and prone to misjudgment. Summary of the Invention

[0005] In view of this, in order to overcome at least one aspect of the above-mentioned problems, embodiments of the present invention propose a method for detecting strong winds, comprising the following steps:

[0006] Acquire multiple image frames;

[0007] Determine the feature matrix based on multiple image frames;

[0008] Two temporal feature matrices with different time scales are determined based on multiple image frames and the feature matrix;

[0009] The time-series feature matrices at different time scales are fused to obtain a fused feature matrix, and the wind detection result is obtained based on the fused feature matrix.

[0010] In some embodiments, obtaining two temporal feature matrices at different time scales based on multiple image frames and the feature matrix further includes:

[0011] Several adjacency matrices are determined based on multiple image frames;

[0012] Based on the feature matrix and several of the adjacency matrices, two time-scale temporal feature matrices are obtained.

[0013] In some embodiments, determining a plurality of adjacency matrices based on a plurality of image frames further includes:

[0014] The first optical flow feature is calculated based on each pair of adjacent image frames, and the first adjacency matrix is ​​determined based on the first optical flow feature;

[0015] The second optical flow feature is calculated based on two image frames with an interval of N frames, and the second adjacency matrix is ​​determined based on the second optical flow feature.

[0016] In some embodiments, calculating a first optical flow feature based on every two adjacent image frames and determining a first adjacency matrix based on the first optical flow feature further includes:

[0017] The first optical flow feature of each pixel is calculated based on each pair of adjacent image frames;

[0018] The first amplitude of each pixel is determined based on the first optical flow feature of each pixel, and the first average amplitude of all pixels is calculated. The first adjacency matrix is ​​determined based on the first average amplitude.

[0019] The calculation of a second optical flow feature based on two image frames with an interval of N frames, and the determination of a second adjacency matrix based on the second optical flow feature, further includes:

[0020] The second optical flow feature of each pixel is calculated based on two image frames with an interval of N frames.

[0021] The second magnitude of each pixel is determined based on the second optical flow feature of each pixel, and the second average magnitude of all pixels is calculated. The second adjacency matrix is ​​then determined based on the second average magnitude.

[0022] In some embodiments, obtaining time-series feature matrices at different time scales based on the feature matrix and the adjacency matrix further includes:

[0023] A first temporal feature matrix is ​​obtained based on the feature matrix and the first adjacency matrix;

[0024] The second temporal feature matrix is ​​obtained based on the feature matrix and the second adjacency matrix.

[0025] In some embodiments, obtaining a first temporal feature matrix based on the feature matrix and the first adjacency matrix further includes:

[0026] Based on a predetermined number of rows in the first adjacency matrix and a corresponding number of rows in the feature matrix, multiple first feature vectors are obtained, and a first temporal feature matrix is ​​obtained based on the multiple first feature vectors;

[0027] The second time-series feature matrix is ​​obtained based on the feature matrix and the second adjacency matrix, and further includes:

[0028] Multiple second feature vectors are obtained based on a preset number of rows in the second adjacency matrix and a corresponding number of rows in the feature matrix, and a second temporal feature matrix is ​​obtained based on the multiple second feature vectors.

[0029] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix further includes:

[0030] The first enhancement matrix and the second enhancement matrix are obtained based on the first time series feature matrix and the second time series feature matrix;

[0031] The first enhancement matrix and the second enhancement matrix are concatenated to obtain the fused feature matrix.

[0032] In some embodiments, obtaining a first enhancement matrix and a second enhancement matrix based on the first time-series feature matrix and the second time-series feature matrix further includes:

[0033] Each row in the first time-series feature matrix is ​​used as a query vector, and the corresponding row in the second time-series feature matrix is ​​used as a key vector and a value vector to obtain the first enhancement vector corresponding to each row of the first time-series feature matrix;

[0034] The first enhancement matrix is ​​obtained based on each row of the first temporal feature matrix and the corresponding first enhancement vector;

[0035] Each row in the second time-series feature matrix is ​​used as a query vector, and the corresponding row in the first time-series feature matrix is ​​used as a key vector and a value vector to obtain the second enhancement vector corresponding to each row of the second time-series feature matrix.

[0036] The second enhancement matrix is ​​obtained based on each row of the second time-series feature matrix and the corresponding second enhancement vector.

[0037] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix and obtaining a strong wind detection result based on the fused feature matrix further includes:

[0038] A pseudo-space feature matrix is ​​obtained by performing a linear transformation based on the fused feature matrix.

[0039] Based on the pseudo-spatial feature matrix, temporal attention processing and spatial attention processing are performed sequentially to obtain spatiotemporal joint features;

[0040] The wind detection result is obtained by decoding based on the spatiotemporal joint features.

[0041] In some embodiments, the method further includes:

[0042] Based on the wind detection results, the time period with strong winds was determined;

[0043] A strong wind alert is generated based on the time period and sent to the corresponding recipient, and the video data collected during the time period is marked as a special dataset.

[0044] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a system for detecting strong winds, comprising:

[0045] The acquisition module is configured to acquire multiple image frames;

[0046] The first calculation module is configured to encode a feature matrix from multiple image frames and construct several adjacency matrices based on the optical flow of the multiple image frames.

[0047] The second calculation module is configured to obtain time-series feature matrices at different time scales based on the feature matrix and the adjacency matrix;

[0048] The fusion detection module is configured to fuse the time-series feature matrices at different time scales to obtain a fusion feature matrix and obtain a strong wind detection result based on the fusion feature matrix.

[0049] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer device, comprising:

[0050] At least one processor; and

[0051] The memory stores a computer program that can run on the processor, which, when executing the program, performs the steps of any of the windy weather detection methods described above.

[0052] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the windy weather detection methods described above.

[0053] This invention offers one of the following beneficial technical effects: The proposed solution first acquires multiple consecutive image frames as raw data for wind detection. Then, the acquired image frames are encoded to extract a feature matrix containing deep semantic features, enabling the matrix to effectively characterize the visual features of each frame. Next, an adjacency matrix is ​​constructed based on the optical flow calculation results between multiple image frames. Optical flow reflects the motion trajectory of objects between different frames, and the adjacency matrix constructed through optical flow introduces inter-frame motion information, thus accurately describing temporal motion correlations. Subsequently, the constructed feature matrix and adjacency matrix are used to extract temporal feature matrices from different time scales, obtaining high-frequency and low-frequency temporal feature matrices. This achieves full capture of multi-scale dynamic information in complex wind fields, thereby improving adaptability to complex environments. Finally, the high-frequency and low-frequency time-series feature matrices are fused to obtain a fused feature matrix that includes multi-scale dynamic information. At the same time, a spatiotemporal separation attention mechanism is introduced to process the fused feature matrix, achieving effective decoupling and joint encoding of time and space dimensions. This further enhances the ability to distinguish complex motion patterns such as high-frequency jitter and low-frequency swaying. Thus, when performing strong wind detection based on this fused matrix, the strong wind state can be identified more comprehensively and accurately, especially in complex environments, effectively improving the reliability and robustness of the detection. Attached Figure Description

[0054] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0055] Figure 1 A flowchart illustrating a method for detecting strong winds provided in an embodiment of the present invention;

[0056] Figure 2 A flowchart illustrating a method for detecting strong winds provided in an embodiment of the present invention;

[0057] Figure 3 A flowchart of a spatiotemporal separation attention encoder provided for an embodiment of the present invention;

[0058] Figure 4 A schematic diagram of the structure of a windy weather detection system provided in an embodiment of the present invention;

[0059] Figure 5 A schematic diagram of the structure of a computer device provided for an embodiment of the present invention;

[0060] Figure 6A schematic diagram of the structure of a computer-readable storage medium provided for an embodiment of the present invention. Detailed Implementation

[0061] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0062] It should be noted that all uses of "first" and "second" in the embodiments of the present invention are for the purpose of distinguishing two entities or parameters with the same name but different names. It is clear that "first" and "second" are only for the convenience of expression and should not be construed as limiting the embodiments of the present invention. Subsequent embodiments will not explain this in detail.

[0063] According to one aspect of the present invention, embodiments of the present invention provide a method for detecting strong winds, such as... Figure 1 As shown, it may include the following steps:

[0064] S1, acquire multiple image frames;

[0065] S2, determine the feature matrix based on multiple image frames;

[0066] S3, determine two temporal feature matrices with different time scales based on multiple image frames and the feature matrix; S4, fuse the temporal feature matrices with different time scales to obtain a fused feature matrix and obtain the wind detection result based on the fused feature matrix.

[0067] Specifically, the process begins by acquiring multiple consecutive image frames. These consecutive image frames are serialized data with fixed time intervals and uniform size, recording the dynamic changes of objects in the scene under wind conditions. Next, the acquired image frames are encoded to extract a feature matrix containing deep semantic features, enabling this matrix to effectively represent the visual features of each frame. Then, using the constructed feature matrix and the multiple image frames, temporal feature matrices are extracted from different time scales, achieving a comprehensive capture of multi-scale dynamic information in complex wind fields, thereby improving adaptability to complex environments. Finally, the temporal feature matrices from different time scales are fused to obtain a fused feature matrix that includes multi-scale dynamic information. When performing strong wind detection based on this fused matrix, the wind state can be identified more comprehensively and accurately, especially in complex environments, effectively improving the reliability and robustness of the detection.

[0068] In some embodiments, such as Figure 2As shown, for multiple consecutive image frames (assuming a total of T frames, each frame being H×W in size), feature extraction is performed on each frame using a pre-trained CNN network (e.g., ResNet-18). Specifically, a mature ResNet-18 model trained on a large-scale image dataset (such as ImageNet) can be used, removing its original last fully connected (FC) classification layer and retaining only the preceding convolutional layers and global average pooling (GAP) layers. When a single frame image is input into this network, after layer-by-layer feature extraction by the convolutional layers and processing by the GAP layers, a D-dimensional feature vector is finally output. This vector condenses the deep features with semantic information in the image frame, demonstrating the powerful image feature extraction capability of the pre-trained CNN network. After performing the above processing on each of the T frames, the D-dimensional features corresponding to each frame are arranged in chronological order, forming a T×D (T rows, D columns) feature matrix X, where each row of the matrix corresponds to the D-dimensional features of one frame image.

[0069] In some embodiments, obtaining two temporal feature matrices at different time scales based on multiple image frames and the feature matrix further includes:

[0070] Several adjacency matrices are determined based on multiple image frames;

[0071] Based on the feature matrix and several of the adjacency matrices, two time-scale temporal feature matrices are obtained.

[0072] Specifically, an adjacency matrix can be constructed based on optical flow features between multiple consecutive image frames. Optical flow reflects the motion trajectory of an object between different frames. The adjacency matrix constructed through optical flow features introduces motion information between frames, thereby accurately describing temporal motion correlations. Then, by using the constructed feature matrix and multiple adjacency matrices, temporal feature matrices are extracted from different time scales, achieving full capture of multi-scale dynamic information in complex wind fields, thus improving the adaptability to complex environments.

[0073] In some embodiments, determining a plurality of adjacency matrices based on a plurality of image frames further includes:

[0074] The first optical flow feature is calculated based on each pair of adjacent image frames, and the first adjacency matrix is ​​determined based on the first optical flow feature;

[0075] The second optical flow feature is calculated based on two image frames with an interval of N frames, and the second adjacency matrix is ​​determined based on the second optical flow feature.

[0076] Specifically, such as Figure 2As shown, in constructing the adjacency matrix, temporal correlation information is obtained through image frames with two different intervals. First, for multiple consecutive image frames, the first optical flow feature between every two adjacent image frames is calculated. The first optical flow feature focuses on the instantaneous motion state of objects between adjacent frames, reflecting subtle motion changes over a short period of time. The first adjacency matrix constructed based on these first optical flow features can accurately characterize the close temporal correlation between adjacent frames, thus helping to capture high-frequency, localized, rapid motion features. Simultaneously, for these image frames, a second optical flow feature can also be calculated between two image frames with an interval of N frames (e.g., intervals of 2 frames, 3 frames, etc.). The second optical flow feature focuses on the motion trend over a longer time interval, reflecting the global change pattern and slow trend of object motion. The second adjacency matrix constructed based on the second optical flow feature describes the temporal dependency between frames at a more macroscopic time scale, helping to capture low-frequency, global motion features.

[0077] In some embodiments, calculating a first optical flow feature based on every two adjacent image frames and determining a first adjacency matrix based on the first optical flow feature further includes:

[0078] The first optical flow feature of each pixel is calculated based on each pair of adjacent image frames;

[0079] The first amplitude of each pixel is determined based on the first optical flow feature of each pixel, and the first average amplitude of all pixels is calculated. The first adjacency matrix is ​​determined based on the first average amplitude.

[0080] Specifically, such as Figure 2 As shown, optical flow methods (such as the Farneback algorithm) can be used to calculate the first optical flow feature of each pixel by applying optical flow to adjacent image frames t and t+1. Based on the first optical flow feature, the corresponding first amplitude value of each pixel can be calculated. :

[0081] , where (x, y) represents the pixel position.

[0082] After obtaining the amplitude of each pixel, the average amplitude of all pixel amplitudes (the first average amplitude) can be calculated and used as the edge weight parameter between adjacent frames:

[0083]

[0084] After obtaining the first average amplitude, based on only when hour, = The rule of setting all other positions to 0 determines the first adjacency matrix. , where i represents the number of rows in the first adjacency matrix and is equal to t, and j represents the number of columns in the first adjacency matrix and is equal to t+1. If the total number of image frames is T, then the first adjacency matrix can be a T×T dimensional square matrix.

[0085] For example, the calculations obtained from frame 1 and frame 2 Then it can be filled into The first row and second column of the image, with the remaining positions in the first row set to 0, are calculated from the second and third frames. Then it can be filled into The value in the second row and third column is 0, and the rest of the positions in the second row are 0. This process continues until the value of the corresponding position in each of the first T-1 rows is calculated, and the vector of all zeros is taken as the Tth row, thus obtaining a T×T dimensional square matrix.

[0086] In some embodiments, calculating a second optical flow feature based on two image frames with an interval of N frames, and determining a second adjacency matrix based on the second optical flow feature, further includes:

[0087] The second optical flow feature of each pixel is calculated based on two image frames with an interval of N frames.

[0088] The second magnitude of each pixel is determined based on the second optical flow feature of each pixel, and the second average magnitude of all pixels is calculated. The second adjacency matrix is ​​then determined based on the second average magnitude.

[0089] Specifically, such as Figure 2 As shown, the second optical flow feature of each pixel can also be obtained by using optical flow methods (such as the Farneback algorithm) to calculate the second optical flow feature of image frame t and image frame t+N+1 at intervals of N frames. Based on the second optical flow feature, the corresponding second magnitude of each pixel can be calculated. :

[0090] , where (x, y) represents the pixel position.

[0091] After obtaining the amplitude of each pixel, the average amplitude of all pixel amplitudes (the second average amplitude) can be calculated and used as the edge weight parameter between the two frames:

[0092]

[0093] After obtaining the second average amplitude, based on only when hour, = The rule of setting all other positions to 0 determines the second adjacency matrix. , where i represents the number of rows in the second adjacency matrix and is equal to t, and j represents the number of columns in the second adjacency matrix and is equal to t+N+1. If the total number of image frames is T, then the first adjacency matrix can be a T×T dimensional square matrix.

[0094] For example, if N is 3, it can be calculated based on the 1st and 5th frames. Then it can be filled into The value is in the 1st row and 5th column, while the rest of the 1st row is 0. This value is calculated from the 2nd and 6th frames. Then it can be filled into The value in the 2nd row and 6th column is 0, and the rest of the positions in the 2nd row are 0. This process continues until the value of each corresponding position in the first TN-1 rows is calculated, and the vector of all 0s is used as the last N+1 rows, thus obtaining a T×T dimensional square matrix.

[0095] In some embodiments, obtaining time-series feature matrices at different time scales based on the feature matrix and the adjacency matrix further includes:

[0096] A first temporal feature matrix is ​​obtained based on the feature matrix and the first adjacency matrix;

[0097] The second temporal feature matrix is ​​obtained based on the feature matrix and the second adjacency matrix.

[0098] Specifically, such as Figure 2 As shown, T frames of images can be treated as T frame nodes. Each frame node is then connected only to its directly adjacent frame nodes, using a dense single-chain connection (1-2-3-4-5…). For example, the first frame node is connected to the second frame node, the second frame node is connected to the third frame node, and so on. The calculated results are then used to further refine the connection. As the weight of each edge, a high-frequency path GNN graph is obtained. Similarly, each frame node is connected to a distant frame node with a fixed step size N, i.e., a cross-frame multi-chain connection (e.g., 1-5-9-… / 2-6-10-… / 3-7-11-… / 4-8-12…). For example, the 1st frame node is connected to the 5th frame node, the 5th frame node is connected to the 9th frame node, and so on; the 2nd frame node is connected to the 6th frame node, the 6th frame node is connected to the 10th frame node, and so on. The above calculations are then used to obtain the... As the weight of each edge, a low-frequency path GNN graph is obtained.

[0099] Graph Attention Network (GAT) is a graph neural network architecture that incorporates an attention mechanism. It adaptively assigns weights to neighboring nodes, thus more effectively modeling the influence of different neighbors on the target node. Compared to traditional fully connected attention mechanisms, GAT only performs attention calculations on adjacent nodes within the graph structure, preserving the graph's structural priors and adaptively measuring the importance of neighboring nodes through learnable attention weights. Therefore, GAT can be used to extract features from both high-frequency and low-frequency path GNN graphs. In particular, the edge weights in both high-frequency and low-frequency path GNN graphs are calculated using optical flow, which reflects the inter-frame physical motion intensity relationships. Thus, GAT can accurately model these dynamic relationships, ultimately improving the extraction of multi-scale wind features.

[0100] In this embodiment, each row of the T×D feature matrix X corresponds to the D-dimensional features of a frame image, while each row of the first adjacency matrix / second adjacency matrix corresponds to the weights of the edges in the high-frequency path GNN graph / low-frequency path GNN graph. Therefore, for ease of calculation, the feature matrix and the first adjacency matrix can be used as the actual calculation parameters of the high-frequency path GNN graph. The high-frequency feature matrix, i.e., the first temporal feature matrix, is extracted by calculating the feature matrix and the first adjacency moments using GAT. Similarly, the feature matrix and the second adjacency matrix are used as the actual calculation parameters of the low-frequency path GNN graph. The low-frequency feature matrix, i.e., the second temporal feature matrix, is extracted by calculating the feature matrix and the second adjacency moments using GAT. Since the first adjacency matrix is ​​constructed based on the first optical flow of adjacent frames, reflecting the inter-frame motion state within a short time interval, the obtained first temporal feature matrix includes rapidly changing local dynamic information (such as high-frequency leaf shaking). Since the second adjacency matrix is ​​constructed based on the second optical flow at intervals of N frames, it reflects the inter-frame motion trend over a longer time interval (such as tree trunk swaying). Therefore, the resulting second temporal feature matrix includes global dynamic information of slow trends.

[0101] Through the two independent GAT processing flows described above, the first and second temporal feature matrices corresponding to different time scales are extracted from the feature matrix and the two adjacency matrices, respectively. The two temporal feature matrices effectively preserve the key correlation information of inter-frame motion at different scales, thereby improving the feature representation capability of complex wind scenes.

[0102] In some embodiments, obtaining a first temporal feature matrix based on the feature matrix and the first adjacency matrix further includes:

[0103] Multiple first feature vectors are obtained based on a preset number of rows in the first adjacency matrix and a corresponding number of rows in the feature matrix, and a first temporal feature matrix is ​​obtained based on the multiple first feature vectors.

[0104] Specifically, such as Figure 2 As shown, when using GAT to calculate the first adjacency matrix and feature matrix, GAT's multi-head attention mechanism can aggregate features from adjacent frames, outputting a D-dimensional first feature vector for every M consecutive frames. Then, multiple D-dimensional first feature vectors are fused to finally obtain the first temporal feature matrix that fuses the local dynamics. Its size is T×D.

[0105] The specific calculation process can be as follows: each consecutive M frames of images is used as a sliding window. Correspondingly, each consecutive M-1 rows in the first adjacency matrix and each consecutive M rows in the feature matrix are used as sliding windows. GAT outputs a D-dimensional first feature vector based on each sliding window in the first adjacency matrix and the corresponding sliding window in the feature matrix. This yields the first feature vector corresponding to the center image within the M-frame image window. After multiple calculations, the first feature vectors corresponding to all frames except edge frames are obtained. Then, the edge frames can be completed by copying the feature vectors corresponding to the nearest neighbor frames. The more image frames there are, the smaller the processing error will be. Finally, all the first feature vectors are combined according to the time dimension to obtain the T×D first temporal feature matrix H_all.

[0106] For example, if M is 3, then every 3 consecutive frames are considered a sliding window. Correspondingly, every 2 rows of the first adjacency matrix and every 3 rows of the feature matrix are considered a sliding window. Thus, the first 1-2 rows of the first adjacency matrix (where the first row represents the edge weights between the 1st and 2nd frames, and the second row represents the edge weights between the 2nd and 3rd frames) and the first 1-3 rows of the feature matrix (where the first row represents the features of the 1st frame, the second row represents the features of the 2nd frame, and the third row represents the features of the 3rd frame) are used for... The first feature vector corresponding to the second frame is calculated. Then, using rows 2-3 of the first adjacency matrix (where row 2 represents the edge weights between frames 2 and 3, and row 3 represents the edge weights between frames 3 and 4) and rows 2-4 of the feature matrix (where row 2 represents the features of frame 2, row 3 represents the features of frame 3, and row 4 represents the features of frame 4), the first feature vector corresponding to the third frame is obtained. This process is repeated for all frames except the first and T-th frames. Finally, the nearest neighbor frames are sampled and copied, with the first feature vector corresponding to the second frame used as the first feature vector corresponding to the first frame, and the first feature vector corresponding to the (T-1)-th frame used as the first feature vector corresponding to the T-th frame.

[0107] In some embodiments, obtaining a second temporal feature matrix based on the feature matrix and the second adjacency matrix further includes:

[0108] Multiple second feature vectors are obtained based on a preset number of rows in the second adjacency matrix and a corresponding number of rows in the feature matrix, and a second temporal feature matrix is ​​obtained based on the multiple second feature vectors.

[0109] Specifically, such as Figure 2 As shown, when using GAT to calculate the second adjacency matrix and feature matrix, GAT's multi-head attention mechanism can aggregate features from adjacent frames, outputting a D-dimensional first feature vector for every M frames. Then, multiple D-dimensional first feature vectors are fused to finally obtain the second temporal feature matrix with fused global dynamics. Its size is T×D.

[0110] The specific calculation process can be as follows: M frames at preset intervals are used as a sliding window; correspondingly, M-1 rows at preset intervals in the second adjacency matrix and M rows at preset intervals in the feature matrix are used as sliding windows. GAT outputs a D-dimensional second feature vector based on each sliding window in the second adjacency matrix and the corresponding sliding window in the feature matrix. This yields the second feature vector corresponding to the center image within the M-frame image window. After multiple calculations, the second feature vectors corresponding to all frames except edge frames are obtained. Then, the edge frames can be completed by copying the feature vectors corresponding to the nearest neighbor frames. The more image frames there are, the smaller the processing error becomes. Finally, all the second feature vectors are combined along the time dimension to obtain a T×D second temporal feature matrix. .

[0111] For example, if M is 3, and N is 3 when calculating the second adjacency matrix, then based on taking one image every three frames and a total of three frames as a sliding window, correspondingly, one row every three rows in the second adjacency matrix and a total of two rows as a sliding window, and one row every three rows in the feature matrix and a total of three rows as a sliding window, this utilizes the first and fifth rows of the second adjacency matrix (where the first row represents the edge weight between the first and fifth frames, and the fifth row represents the edge weight between the fifth and ninth frames) and the first, fifth, and ninth rows of the feature matrix (where the first row represents the feature of the first frame, the fifth row represents the feature of the ninth frame, and the fifth row represents the feature of the eleventh frame). The second feature vector corresponding to the 5th frame is calculated using rows 2 and 6 of the second adjacency matrix (where row 2 represents the edge weight between the 2nd and 6th frames, and row 6 represents the edge weight between the 6th and 10th frames) and rows 2, 6, and 10 of the feature matrix (where row 2 represents the feature of the 2nd frame, row 6 represents the feature of the 6th frame, and row 10 represents the feature of the 10th frame). This process is repeated to obtain the first feature vectors for all frames except for frames 1 to 4 and frames T-3 to T. Finally, the second feature vector corresponding to the 5th frame is used as the second feature vector for the 1st frame, the second feature vector corresponding to the 6th frame is used as the second feature vector for the 2nd frame, and so on, to obtain the second feature vectors for frames 1 to 4 and frames T-3 to T.

[0112] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix further includes:

[0113] The first enhancement matrix and the second enhancement matrix are obtained based on the first time series feature matrix and the second time series feature matrix;

[0114] The first enhancement matrix and the second enhancement matrix are concatenated to obtain the fused feature matrix.

[0115] Specifically, such as Figure 2As shown, since the first and second temporal features are extracted independently from different time scales, they need to be fused before subsequent detection. Before fusion, both need to be enhanced to strengthen the temporal expressive power of the final features. When fusing temporal features from different time scales, the first and second temporal feature matrices can be interactively enhanced. By interleaving the complementary dynamic information in the two matrices, the first temporal feature matrix retains local high-frequency characteristics while incorporating global trend information, forming the first enhancement matrix. Similarly, the second temporal feature matrix maintains the global low-frequency trend while supplementing locally rapidly changing information, forming the second enhancement matrix.

[0116] Finally, the first and second enhancement matrices, after cross-enhancement, are concatenated to obtain the fused feature matrix. This fused matrix includes interactively enhanced high-frequency dynamic features and low-frequency trend features, integrating information on rapid local changes and slow global trends. This significantly enhances the feature matrix's ability to comprehensively express temporal changes, enabling more comprehensive and accurate identification of wind conditions in subsequent recognition processes. Especially in complex environments, it effectively improves the reliability and robustness of detection.

[0117] In some embodiments, obtaining a first enhancement matrix and a second enhancement matrix based on the first time-series feature matrix and the second time-series feature matrix further includes:

[0118] Each row in the first time-series feature matrix is ​​used as a query vector, and the corresponding row in the second time-series feature matrix is ​​used as a key vector and a value vector to obtain the first enhancement vector corresponding to each row of the first time-series feature matrix;

[0119] The first enhancement matrix is ​​obtained based on each row of the first temporal feature matrix and the corresponding first enhancement vector;

[0120] Each row in the second time-series feature matrix is ​​used as a query vector, and the corresponding row in the first time-series feature matrix is ​​used as a key vector and a value vector to obtain the second enhancement vector corresponding to each row of the second time-series feature matrix.

[0121] The second enhancement matrix is ​​obtained based on each row of the second time-series feature matrix and the corresponding second enhancement vector.

[0122] Specifically, such as Figure 2As shown, in the process of interactive enhancement between the first and second temporal feature matrices, cross-attention feature enhancement operations can be performed independently for each frame t. First, for the first temporal feature matrix, each row is used as a query vector, and the corresponding row in the second temporal feature matrix is ​​used as the key and value vectors, respectively. The first enhancement vector corresponding to each row in the first temporal feature matrix is ​​calculated through an attention mechanism. This process allows each row of features in the first temporal feature matrix to obtain supplementary information about the global trend from the corresponding row in the second temporal feature matrix, achieving a precise correlation between local features and the global trend. Subsequently, each row in the first temporal feature matrix is ​​added to its corresponding first enhancement vector to obtain the enhanced features. Multiple enhanced features are then concatenated along the time dimension to obtain the first enhancement matrix. This matrix retains the high-frequency dynamic characteristics of the original first temporal feature matrix while incorporating the low-frequency trend information of the second temporal feature matrix, enhancing the perception ability of local features to the global background.

[0123] Similarly, a symmetrical approach is used to enhance the second temporal feature matrix: each row of the second temporal feature matrix is ​​used as a query vector, and the corresponding rows of the first temporal feature matrix are used as key and value vectors, respectively. The same attention mechanism is used to calculate the second enhancement vector corresponding to each row of the second temporal feature matrix. This allows each row of features in the second temporal feature matrix to obtain supplementary details of rapid local changes from the corresponding rows of the first temporal feature matrix, achieving accurate capture of local dynamics by global features. Each row of the second temporal feature matrix is ​​added to its corresponding second enhancement vector to obtain the enhanced features. Multiple enhanced features are concatenated along the time dimension to obtain the second enhancement matrix. This matrix retains the low-frequency trend characteristics of the original second temporal feature matrix while incorporating the high-frequency dynamic information of the first temporal feature matrix, enhancing the ability of global features to characterize local details. Finally, the obtained first and second enhancement matrices are concatenated along the feature dimension to generate the final fused features.

[0124] For example, for frame t, first from the first temporal feature matrix Extract the D-dimensional high-frequency features corresponding to the t-th frame. : And from the second time-series feature matrix Extract the D-dimensional low-frequency features corresponding to the t-th frame. : Then calculate. and Cross attention: using As a query, As Key / Value → Output the first augmentation vector .use As a query, As Key / Value → Output Second Enhanced Vector Next, use and Perform feature enhancement: Repeat the above process for each frame to obtain multiple enhanced h values. e and multiple enhanced l e , will multiple h e The first enhancement matrix H is obtained by concatenating along the time dimension. e t , to have multiple l e The second enhancement matrix L is obtained by concatenating along the time dimension. e t H e t and L e t The size is Finally, H is concatenated along the feature dimension. e t and L e t This generates the final fusion features: Its size is .

[0125] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix and obtaining a strong wind detection result based on the fused feature matrix further includes:

[0126] A pseudo-space feature matrix is ​​obtained by performing a linear transformation based on the fused feature matrix.

[0127] Based on the pseudo-spatial feature matrix, temporal attention processing and spatial attention processing are performed sequentially to obtain spatiotemporal joint features;

[0128] The wind detection result is obtained by decoding based on the spatiotemporal joint features.

[0129] Specifically, such as Figure 2 and Figure 3 As shown, in order to adapt to spatial attention computation, the fused features can be... Projecting the linear transformation into pseudo-space features The size of the pseudo-space feature is ,in It should be noted that the fusion feature Temporal features are converted into pseudo-spatial tensors by linear projection. Their spatial dimensions H'×W' have no direct correspondence with the original image, but the correlation between regions can be learned adaptively through training.

[0130] Finally, the pseudo-space features will be... The input is fed into a spatiotemporal separation attention encoder, which sequentially performs temporal attention processing and spatial attention processing, and performs spatiotemporal joint modeling. This encodes multi-scale motion patterns such as high-frequency jitter and low-frequency swaying into a unified temporal feature representation, providing discriminative features for subsequent wind force classification or strong wind detection tasks.

[0131] In particular, when performing temporal attention calculations, based on the same spatial location exist Feature extraction is performed to obtain feature slices of all frames at that spatial location. Its size is Since the extraction is based on the same spatial location (i, j), therefore In the sequence (i, j), (i, j) remains constant, while the other two colons represent the extracted features. Then, self-attention is calculated based on the feature slices: = Attention(Q, K, V), where Q / K / V come from... [:, i, j, :], through a size of Learnable parameter matrix Generation. The final time features are obtained. Size is .

[0132] When performing spatial attention calculations, the features corresponding to each frame t need to be processed separately to calculate all spatial locations within that frame. attention, first from Extract the feature slice corresponding to frame t. The size is H'×W'×D'. Then, self-attention is calculated based on the feature slices: Q / K / V come from Through size Learnable parameter matrix Generate. The final result is of size [size missing]. spatiotemporal joint characteristics .

[0133] Finally, as Figure 2 As shown, for Simultaneous average pooling (GAP) is performed on both the time and spatial dimensions to obtain a D'-dimensional feature vector. This D'-dimensional feature vector is then input into a multilayer perceptron (MLP) to obtain the wind feature classification result. In some embodiments, a binary cross-entropy loss or a cross-entropy loss optimization model can be selected based on the task type.

[0134] By introducing a spatiotemporal separation attention mechanism to process the fused pseudo-temporal-space features, the effective decoupling and joint encoding of time and space dimensions can be achieved, which can effectively enhance the model's ability to distinguish complex motion patterns such as high-frequency jitter and low-frequency swaying.

[0135] In some embodiments, the method further includes:

[0136] Based on the wind detection results, the time period with strong winds was determined;

[0137] A strong wind alert is generated based on the time period and sent to the corresponding recipient, and the video data collected during the time period is marked as a special dataset.

[0138] Specifically, through embodiments of the present invention, wind detection results can be obtained. Based on these results, it can be determined whether strong winds exist and their intensity level, thereby pinpointing the specific time period during which the strong winds will persist. Once the time period is determined, a strong wind alert can be generated and sent to the appropriate recipients. This alert may include key information such as the duration of the strong winds and the estimated wind intensity. Recipients can cover various categories of objects based on the application scenario, thereby helping relevant parties avoid the risks brought by strong winds through timely warnings.

[0139] Additionally, video data collected during this time period can be labeled as a special dataset. Because strong winds significantly affect the movement of objects and the state of the scene in videos (e.g., causing objects to sway and the image to shake), the video data from this time period differs fundamentally from data collected under normal weather conditions. On one hand, this type of data can serve as "strong wind scene samples" for subsequent optimization of the strong wind detection model. On the other hand, in other applications that rely on normal scene data (such as conventional object recognition and scene state monitoring), it is necessary to explicitly label its special attributes to avoid data distortion caused by strong winds affecting the accuracy of other sensory recognition results. For example, the application of data from this time period can be abandoned, or the output of other sensory recognition results from this time period can be turned off. Furthermore, labeling it as a special dataset facilitates subsequent data retrieval and management. For example, if it is necessary to trace scene changes under strong winds, the video data for the corresponding time period can be quickly located.

[0140] Based on the same inventive concept, according to another aspect of the present invention, embodiments of the present invention also provide a windy weather detection system 400, such as... Figure 4 As shown, it includes:

[0141] The acquisition module 401 is configured to acquire multiple image frames;

[0142] The first calculation module 402 is configured to encode a feature matrix from multiple image frames and construct several adjacency matrices based on the optical flow of the multiple image frames.

[0143] The second calculation module 403 is configured to obtain time-series feature matrices at different time scales based on the feature matrix and the adjacency matrix;

[0144] The fusion detection module 404 is configured to fuse the time-series feature matrices at different time scales to obtain a fusion feature matrix and obtain a strong wind detection result based on the fusion feature matrix.

[0145] In some embodiments, constructing several adjacency matrices based on the optical flow of multiple image frames further includes:

[0146] The first optical flow is calculated based on each pair of adjacent image frames, and the first adjacency matrix is ​​determined based on the first optical flow;

[0147] The second optical flow is calculated based on two image frames with an interval of N frames, and the second adjacency matrix is ​​determined based on the second optical flow.

[0148] In some embodiments, calculating a first optical flow based on every two adjacent image frames and determining a first adjacency matrix based on the first optical flow further includes:

[0149] The first optical flow of each pixel is calculated based on every two adjacent image frames;

[0150] The first magnitude of each pixel is determined based on the first optical flow of each pixel, and the first average magnitude of all pixels is calculated. The first adjacency matrix is ​​determined based on the first average magnitude.

[0151] The calculation of the second optical flow based on two image frames with an interval of N frames, and the determination of the second adjacency matrix based on the second optical flow, further includes:

[0152] The second optical flow for each pixel is calculated based on two image frames with an interval of N frames.

[0153] The second magnitude of each pixel is determined based on the second optical flow of each pixel, and the second average magnitude of all pixels is calculated. The second adjacency matrix is ​​then determined based on the second average magnitude.

[0154] In some embodiments, obtaining time-series feature matrices at different time scales based on the feature matrix and the adjacency matrix further includes:

[0155] A first temporal feature matrix is ​​obtained based on the feature matrix and the first adjacency matrix;

[0156] The second temporal feature matrix is ​​obtained based on the feature matrix and the second adjacency matrix.

[0157] In some embodiments, obtaining a first temporal feature matrix based on the feature matrix and the first adjacency matrix further includes:

[0158] Based on a predetermined number of rows in the first adjacency matrix and a corresponding number of rows in the feature matrix, multiple first feature vectors are obtained, and a first temporal feature matrix is ​​obtained based on the multiple first feature vectors;

[0159] The second time-series feature matrix is ​​obtained based on the feature matrix and the second adjacency matrix, and further includes:

[0160] Multiple second feature vectors are obtained based on a preset number of rows in the second adjacency matrix and a corresponding number of rows in the feature matrix, and a second temporal feature matrix is ​​obtained based on the multiple second feature vectors.

[0161] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix further includes:

[0162] The first enhancement matrix and the second enhancement matrix are obtained based on the first time series feature matrix and the second time series feature matrix;

[0163] The first enhancement matrix and the second enhancement matrix are concatenated to obtain the fused feature matrix.

[0164] In some embodiments, obtaining a first enhancement matrix and a second enhancement matrix based on the first time-series feature matrix and the second time-series feature matrix further includes:

[0165] Each row in the first time-series feature matrix is ​​used as a query vector, and the corresponding row in the second time-series feature matrix is ​​used as a key vector and a value vector to obtain the first enhancement vector corresponding to each row of the first time-series feature matrix;

[0166] The first enhancement matrix is ​​obtained based on each row of the first temporal feature matrix and the corresponding first enhancement vector;

[0167] Each row in the second time-series feature matrix is ​​used as a query vector, and the corresponding row in the first time-series feature matrix is ​​used as a key vector and a value vector to obtain the second enhancement vector corresponding to each row of the second time-series feature matrix.

[0168] The second enhancement matrix is ​​obtained based on each row of the second time-series feature matrix and the corresponding second enhancement vector.

[0169] In some embodiments, fusing the time-series feature matrices at different time scales to obtain a fused feature matrix and obtaining a strong wind detection result based on the fused feature matrix further includes:

[0170] A pseudo-space feature matrix is ​​obtained by performing a linear transformation based on the fused feature matrix.

[0171] Based on the pseudo-spatial feature matrix, temporal attention processing and spatial attention processing are performed sequentially to obtain spatiotemporal joint features;

[0172] The wind detection result is obtained by decoding based on the spatiotemporal joint features.

[0173] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 5 As shown, embodiments of the present invention also provide a computer device 501, comprising:

[0174] At least one processor 520; and

[0175] The memory 510 stores a computer program 511 that can run on a processor. When the processor 520 executes the program, it performs the steps of any of the above-described methods for detecting strong winds.

[0176] Based on the same inventive concept, according to another aspect of the present invention, such as Figure 6 As shown, embodiments of the present invention also provide a computer-readable storage medium 601, which stores a computer program 610. When the computer program 610 is executed by a processor, it performs the steps of any of the above-described methods for detecting strong winds.

[0177] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above methods.

[0178] Furthermore, it should be understood that the computer-readable storage medium (e.g., memory) described herein may be volatile memory or non-volatile memory, or may include both volatile memory and non-volatile memory.

[0179] Those skilled in the art will also understand that the various exemplary logic blocks, modules, circuits, and algorithm steps described in conjunction with the disclosure herein can be implemented as electronic hardware, computer software, or a combination of both. To clearly illustrate this interchangeability between hardware and software, the functionality of various illustrative components, blocks, modules, circuits, and steps has been generally described. Whether this functionality is implemented as software or as hardware depends on the specific application and the design constraints imposed on the system as a whole. Those skilled in the art can implement the functionality in various ways for each specific application, but such implementation decisions should not be construed as departing from the scope of the embodiments disclosed herein.

[0180] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0181] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0182] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0183] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0184] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A method of detecting a strong wind weather, characterized by, The method comprises the following steps: obtaining a plurality of image frames; determining a feature matrix based on the plurality of image frames; determining a time sequence feature matrix of two different time scales based on the plurality of image frames and the feature matrix; fusing the time sequence feature matrix of different time scales to obtain a fused feature matrix and obtaining a gale detection result based on the fused feature matrix; determining a plurality of adjacency matrices based on the plurality of image frames; determining a time sequence feature matrix of two different time scales based on the feature matrix and the plurality of adjacency matrices; determining a plurality of adjacency matrices based on the plurality of image frames, further comprising: calculating a first optical flow feature based on every two adjacent image frames, and determining a first adjacency matrix based on the first optical flow feature; calculating a second optical flow feature based on every two image frames with an interval of N frames, and determining a second adjacency matrix based on the second optical flow feature; determining a time sequence feature matrix of different time scales based on the feature matrix and the adjacency matrix, further comprising: determining a first time sequence feature matrix based on the feature matrix and the first adjacency matrix; determining a second time sequence feature matrix based on the feature matrix and the second adjacency matrix. calculating a first optical flow feature based on every two adjacent image frames, and determining a first adjacency matrix based on the first optical flow feature, further comprising:

2. The method of claim 1, wherein, calculating a first optical flow feature of each pixel based on every two adjacent image frames; determining a first amplitude of each pixel based on the first optical flow feature of each pixel, and calculating a first average amplitude of all pixels, and determining a first adjacency matrix based on the first average amplitude; calculating a second optical flow feature of each pixel based on every two image frames with an interval of N frames; determining a second amplitude of each pixel based on the second optical flow feature of each pixel, and calculating a second average amplitude of all pixels, and determining a second adjacency matrix based on the second average amplitude. determining a first time sequence feature matrix based on the first adjacency matrix and the feature matrix, further comprising: obtaining a plurality of first feature vectors based on each preset number of rows in the first adjacency matrix and a corresponding number of rows in the feature matrix, and obtaining a first time sequence feature matrix based on the plurality of first feature vectors; 3. The method of claim 1, wherein, determining a second time sequence feature matrix based on the second adjacency matrix and the feature matrix, further comprising: obtaining a plurality of second feature vectors based on each preset number of rows in the second adjacency matrix and a corresponding number of rows in the feature matrix, and obtaining a second time sequence feature matrix based on the plurality of second feature vectors. fusing the time sequence feature matrix of different time scales to obtain a fused feature matrix, further comprising: obtaining a first enhancement matrix and a second enhancement matrix based on the first time sequence feature matrix and the second time sequence feature matrix; 4. The method of claim 1, wherein, splicing the first enhancement matrix and the second enhancement matrix to obtain a fused feature matrix. ​ ​ 5. The method of claim 4, wherein, The first enhancement matrix and the second enhancement matrix are obtained based on the first time sequence feature matrix and the second time sequence feature matrix, and further comprising: Each row in the first time sequence feature matrix is taken as a query vector, and a corresponding row in the second time sequence feature matrix is taken as a key vector and a value vector to obtain a corresponding first enhancement vector of each row in the first time sequence feature matrix; A first enhancement matrix is obtained based on each row in the first time sequence feature matrix and the corresponding first enhancement vector; Each row in the second time sequence feature matrix is taken as a query vector, and a corresponding row in the first time sequence feature matrix is taken as a key vector and a value vector to obtain a corresponding second enhancement vector of each row in the second time sequence feature matrix; A second enhancement matrix is obtained based on each row in the second time sequence feature matrix and the corresponding second enhancement vector.

6. The method of claim 1, wherein, The time sequence feature matrices of different time scales are fused to obtain a fused feature matrix, and a gale detection result is obtained based on the fused feature matrix, and further comprising: A pseudo-space feature matrix is obtained based on linear transformation of the fused feature matrix; Temporal attention processing and spatial attention processing are sequentially performed based on the pseudo-space feature matrix to obtain a time-space joint feature; The gale detection result is obtained based on decoding of the time-space joint feature.

7. The method of claim 1, wherein, Further comprising: A time period with gale weather is determined based on the gale detection result; A gale reminder is generated based on the time period and sent to a corresponding receiver, and video data collected in the time period is marked as a special data set.

Images