A typhoon multi-task estimation method guided by multi-modal information mixing
Through the multi-modal information hybrid guidance typhoon multi-task estimation method, combined with time-series multi-channel satellite data and development information, the problem that the existing technology cannot estimate the typhoon intensity and wind circle size at the same time is solved, and the typhoon intensity and wind circle size is achieved simultaneously Estimation improves the accuracy and practical application value of the estimation.
Patent Information
- Application Number
- CN202311209274.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-18
- Publication Date
- 2024-11-26
- Estimated Expiration
- 2043-09-18
AI Technical Summary
The existing deep learning-based typhoon estimation method cannot effectively estimate the typhoon intensity and wind circle size at the same time, and ignores the importance of typhoon development information.
The multi-task estimation method of multi-modal information guided by multi-modal information is adopted. By integrating the time-series multi-channel satellite data with position characteristics, development time characteristics, intensity weight characteristics and wind circle weight characteristics, input the shared network and development network for feature extraction, and The typhoon intensity and wind ring radius are estimated using a decoder.
It has achieved the extraction of typhoon characteristics from multimodal meteorological data, and learned the commonalities and characteristics between intensity and wind circle estimation, and can accurately estimate the typhoon intensity and wind circle size at the same time, which has improved the practical significance of disaster prevention and mitigation.
Smart Images

Figure CN117332364B_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the technical field of typhoon real-time estimation, and in particular, relates to a typhoon multi-task estimation method guided by multi-modal information mixing. Background Art
[0002] A typhoon is a low-pressure vortex in tropical or subtropical regions, so it is also called a tropical cyclone. A typhoon is a complex weather system that mainly occurs on the ocean surface and is usually accompanied by low pressure, strong winds and heavy rains. In addition, strong typhoons can also cause other natural disasters, such as floods and mudslides, which seriously endanger people's lives and property. In order to prevent such weather disasters, real-time estimation of the intensity and size of the typhoon is of great significance to social development. Since the factors affecting the development of typhoons are both diverse and rich, this has caused certain difficulties in the real-time estimation of typhoons. Therefore, it is very necessary to establish a typhoon estimation system that can utilize the multimodal development information of typhoons.
[0003] With the development of satellite observation technology and deep learning, most of the current typhoon estimation methods based on deep learning have surpassed traditional methods in performance. Therefore, compared with traditional methods with certain subjectivity, deep learning methods with strong autonomous learning capabilities have begun to receive widespread attention. At present, most of the methods based on deep learning only realize real-time estimation of intensity, ignoring the importance of the size of the wind circle. In practical applications, the radius of a typhoon's wind circle is usually used to measure the scale of a typhoon. It represents the impact range of strong winds brought by the main circulation of the typhoon and is an important reference indicator for typhoon defense. In addition, the current typhoon estimation methods based on deep learning also ignore the importance of typhoon development information to typhoon estimation. Summary of the invention
[0004] The purpose of this application is to provide a typhoon multi-task estimation method guided by multi-modal information mixing, so as to solve the problem that the existing technical solutions cannot effectively estimate the typhoon intensity and wind circle size at the same time, and cannot better extract typhoon development information.
[0005] In order to achieve the above purpose, the technical solution of this application is as follows:
[0006] A typhoon multi-task estimation method guided by multi-modal information mixing, comprising:
[0007] After fusing the time series multi-channel satellite data with the location features extracted by the location extraction module, the data is input into the shared network module for feature extraction. Then, the output features of the shared network module are fused with the development duration features extracted by the time extraction module to obtain shared features.
[0008] The time series multi-channel satellite data difference is input into the development network module to extract the development features, and then multiplied with the intensity weight features and wind circle weight features extracted by the category extraction module to obtain the intensity development features and wind circle development features;
[0009] The shared features and the intensity development features are spliced together and input into the intensity decoder to estimate the intensity of the typhoon. The shared features and the wind circle development features are spliced together and input into the wind circle decoder to estimate the radius of the typhoon's wind circle.
[0010] Furthermore, the position extraction module is composed of four fully connected layers, and the longitude and latitude data of the typhoon center are input into the position extraction module to extract the position features of the typhoon.
[0011] Furthermore, the time extraction module is composed of two fully connected layers, and the typhoon existence duration data is input into the time extraction module to extract the typhoon development duration characteristics.
[0012] Furthermore, the category extraction module is composed of two identical multilayer perceptrons, each of which includes two fully connected layers. The typhoon intensity level data is input into the category extraction module to extract intensity weight features and wind circle weight features.
[0013] Furthermore, the shared network module includes four feature extraction layers and two channel residual connections, each feature extraction layer includes a 2D convolution and maximum pooling layer, wherein:
[0014] The output of the second feature extraction layer is connected to the output of the third feature extraction layer through a channel residual connection, and then input to the fourth feature extraction layer;
[0015] The input of the fourth feature extraction layer is connected to the output of the fourth feature extraction layer through a channel residual connection as the output feature of the shared network module.
[0016] Furthermore, the development network module consists of three feature extraction layers, each of which includes a 3D convolution and maximum pooling layer.
[0017] This application proposes a typhoon multi-task estimation method guided by multi-modal information mixing. The data required for estimation include one-dimensional data, two-dimensional satellite data, and additional typhoon development information. The one-dimensional data is the central longitude and latitude of the typhoon, the two-dimensional satellite data contains the bright temperature data of multiple infrared channels, and the development information includes the existence duration of the typhoon, the intensity level at the previous moment, and the difference data of multiple channel satellite data. Therefore, this application can extract features from multi-modal meteorological data and fuse data, and finally give the estimation results of typhoon intensity and wind circle radius at the same time.
[0018] The beneficial effects of the present application are: being able to extract the characteristics of the typhoon itself and the development characteristics of the typhoon from multimodal meteorological data, and being able to use these characteristics to learn the commonalities and characteristics between the two tasks of intensity and wind circle estimation, and finally achieving simultaneous estimation of intensity and wind circle. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] Figure 1 This is a flow chart of the typhoon multi-task estimation method guided by mixed multi-modal information in this application.
[0020] Figure 2 Schematic diagram of the estimated network model for this application.
[0021] Figure 3 This is a schematic diagram of a shared network module according to an embodiment of the present application.
[0022] Figure 4 A schematic diagram of a network module developed in accordance with an embodiment of the present application. DETAILED DESCRIPTION
[0023] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0024] An embodiment of the present application, such as Figure 1 As shown, a typhoon multi-task estimation method guided by multi-modal information mixing is provided, including:
[0025] Step S1: After fusing the time series multi-channel satellite data with the location features extracted by the location extraction module, the data are input into the shared network module for feature extraction, and then the output features of the shared network module are fused with the development duration features extracted by the time extraction module to obtain shared features.
[0026] The typhoon multi-task estimation method guided by multi-modal information mixing in this embodiment is to construct an estimation network model through training, and then use the trained estimation network model to process real-time typhoon data to estimate the intensity and wind circle at the current moment.
[0027] The estimation network model constructed in this embodiment is as follows Figure 2 As shown, it includes a multimodal information hybrid guidance structure, a shared network module, a development network module and two output decoders. Among them, the multimodal information hybrid guidance structure consists of three modules: position extraction module, category extraction module and time extraction module.
[0028] Specifically, the location extraction module is composed of four fully connected layers, and the longitude and latitude data of the typhoon center are input into the location extraction module to extract the location features of the typhoon. In a specific embodiment, the number of neurons in the four fully connected layers is set to 5, 25, 58 and 156 respectively, the four weights are W_P_1 to W_P_4, defined as floating-point variables, and the biases are b_p_1 to b_p_4, defined as floating-point variables.
[0029] The category extraction module is composed of two identical multilayer perceptrons. The typhoon intensity level data is input into the category extraction module to extract the typhoon intensity level features. In a specific embodiment, the category extraction module is composed of two identical multilayer perceptrons, each of which includes two fully connected layers, which are used to extract the typhoon's intensity level information at the previous moment and output two weight features C_1 and C_2, respectively referred to as intensity weight features and wind circle weight features, which are embedded in the development network as auxiliary features, thereby obtaining development features suitable for two estimation tasks. The two multilayer perceptrons have the same structure, and the number of neurons in the two internal fully connected layers is set to 2 and 4 respectively, and the weights and biases of the two fully connected layers are defined as floating-point variables.
[0030] It should be noted that the wind circle estimated in this application is the size of the wind circle corresponding to the maximum wind speed. There is a corresponding relationship between the wind circle and the intensity. The input intensity level can enable the network model to know the current development stage of the typhoon. The output of the sensor is obtained by mapping the intensity level, and the result of the sensor output is related to the intensity and wind circle.
[0031] The time extraction module consists of two fully connected layers. The typhoon's existence duration data is input into the time extraction module to extract the typhoon's development duration characteristics. In a specific embodiment, the number of neurons in the two fully connected layers is set to 2 and 4 respectively, the two weights are W_T_1 and W_T_2, defined as floating-point variables, and the biases are b_t_1 and b_t_2, defined as floating-point variables.
[0032] It should be noted that all fully connected layers in the above modules are followed by the Tanh activation function.
[0033] In a specific embodiment, Figure 3As shown in the figure, the shared network module mainly consists of four feature extraction layers and two channel residual connections. Each feature extraction layer includes a 2D convolution and a maximum pooling layer. All convolution layers in the shared network module are followed by a Relu activation function. The convolution layer is used to extract features from two-dimensional satellite data, and the pooling layer is used for feature downsampling. The output of the second feature extraction layer is connected to the output of the third feature extraction layer through a channel residual connection, and then input to the fourth feature extraction layer; the input of the fourth feature extraction layer is connected to the output of the fourth feature extraction layer through a channel residual connection, which serves as the output feature of the shared network module.
[0034] The weight of the entire shared network module is W_S, defined as a floating-point variable without bias. The (number of input and output channels, convolution kernel size, convolution step and padding width) of the four 2D convolutional layers are set to (9, 16, 7, 3, 0), (16, 32, 5, 1, 0), (32, 64, 3, 1, 1) and (96, 192, 3, 1, 1) respectively. The convolution kernel size, convolution step and padding width of the first three maximum pooling layers are all 3, 2, 0; the convolution kernel size, convolution step and padding width of the last pooling layer are set to 2, 2, 0.
[0035] In this embodiment, before the time series multi-channel satellite data is sent to the shared network for feature extraction, the position features generated by the position extraction module will be early fused with it, so that the shared network pays more attention to the key position during the feature extraction process. After the shared network finishes extracting features, the development duration features extracted by the time extraction model will be late fused with its output features to form shared features for multi-task estimation. The two different fusion methods are implemented through addition and splicing operations, that is, early fusion is achieved through addition, and late fusion is achieved through splicing.
[0036] Step S2: input the time series multi-channel satellite data difference into the development network module to extract the development features, and then multiply them with the intensity weight features and wind circle weight features extracted by the category extraction module to obtain the intensity development features and wind circle development features.
[0037] Specifically, Figure 4As shown in the figure, the development network module mainly consists of three feature extraction layers, each of which includes a 3D convolution and a maximum pooling layer. All convolution layers in the development network module are followed by the Relu activation function. Since its input data is the difference of two-dimensional satellite data, the convolution layer can extract the change features in the satellite data from it, and the pooling layer is also used for feature downsampling. The weight of the entire development network is W_D, defined as a floating-point variable without bias. The (number of input and output channels) of the three 3D convolution layers are set to (1, 4), (4, 8) and (8, 16) respectively; the convolution kernel size of the convolution layer is set to (1, 5, 5), (1, 3, 3) and (1, 3, 3) respectively; the convolution step size is set to (1, 2, 2), (1, 1, 1) and (1, 1, 1); the padding width is 0. The convolution kernel size, convolution step size and padding width of the maximum pooling layer are all set to (1, 3, 3), (1, 2, 2) and 0.
[0038] After the development network module extracts the development features, they are fused with the intensity weight features and wind circle weight features extracted by the category extraction module to form the intensity development features and wind circle development features used to estimate the specific tasks of the typhoon. The fusion method is achieved through matrix multiplication operations.
[0039] Step S3: splice the shared features with the intensity development features and input them into the intensity decoder to estimate the intensity of the typhoon; splice the shared features with the wind circle development features and input them into the wind circle decoder to estimate the radius of the typhoon's wind circle.
[0040] Specifically, the output features of the shared network module and the development network module are used to integrate and feed into the corresponding decoders to simultaneously estimate the typhoon's intensity and wind circle radius. This method obtains simultaneous estimation of typhoon intensity and size under the guidance of development information, which is more practical in disaster prevention and mitigation applications.
[0041] The decoder consists of four fully connected layers followed by a Relu activation function, which is used to output two regression values of the intensity and size of the typhoon. The number of neurons in the four fully connected layers is set to 1024, 512, 128 and 1 respectively. The four weights of the two decoders are W_DI_1 to W_DI_4 and W_DS_1 to W_DS_4, defined as floating-point variables, and the biases are b_di_1 to b_di_4 and b_ds_1 to b_ds_4, defined as floating-point variables.
[0042] For the estimation network model in this application, the training process is as follows:
[0043] First, the dataset is preprocessed. The dataset is divided into three parts: training set, validation set, and test set. The training set contains 160 typhoon data that occurred in the longitude and latitude range of 60S-60N, 80E-160W from 2015 to 2020. The validation set and test set contain 46 typhoon data that occurred in 2021-2022, which are randomly divided in a 1:1 ratio. The one-dimensional data, two-dimensional data, and environmental data in the dataset are standardized to make the data more conducive to model training. The data of the training set are rotated clockwise by 90 degrees, 180 degrees, and 270 degrees for data enhancement.
[0044] While training:
[0045] The parameters of the convolutional and fully connected layers in the model were initialized with random weights, and a total of 450 complete trainings were performed; the learning rate of the model optimizer SGD was set to 0.0005 and the momentum was set to 0.4.
[0046] Part of the typhoon's one-dimensional data (typhoon longitude and latitude) and development information (typhoon's existence duration, intensity level at the previous moment) are input into the multimodal hybrid guidance structure; the two-dimensional satellite data are input into the shared network; and the difference data of the two-dimensional data are input into the development network.
[0047] The multimodal information mixing guidance structure and two convolutional networks extract features from the input multimodal data respectively, and mix and fuse the features of the three modules.
[0048] Based on the design idea of soft sharing in multi-task learning, the shared representation and the development features suitable for intensity and wind circle estimation are fused and sent to two decoders respectively, and the estimated values of the typhoon's intensity and size are output at the same time.
[0049] The error between the estimated typhoon and the true value of the typhoon is calculated to obtain two L1 losses, which are weightedly added and the total error is fed back to the network to adjust the parameters of the network and optimize the estimation performance.
[0050] Use the SGD algorithm to minimize the loss function, and update the loss weight of the task every 100 epochs of training, so that the model can balance the learning process of the two tasks in iterative training. The dynamic update of weights is based on the loss value of a single task. By increasing the weight of tasks with large losses, the model can pay more attention to tasks with larger estimation errors, thereby balancing the performance of the two tasks. The specific weight update rules are as follows:
[0051] For tasks with larger loss values, update their weights to the L1 loss of the task / the L1 loss of another task.
[0052] The model was trained until convergence and the best performance model was obtained for typhoon estimation.
[0053] This application also provides experimental data, and compares the technical solution of this application with some traditional technical solutions (ADT, MTCSWA) and other deep learning methods (CNN for intensity estimation, CNN for sizeestimation, TC-MTLNet and DeepTCNet). The performance indicators used for comparison are mean absolute error MAE and root mean square error RMSE. The comparison results are shown in Table 1:
[0054] Table 1
[0055]
[0056] From the comparison results in Table 1, it can be seen that the performance of the technical solution of the present application is better than that of the traditional technical solution.
[0057] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A typhoon multi-task estimation method guided by multi-modal information mixing, characterized in that: The typhoon multi-task estimation method guided by multi-modal information mixing includes: After fusing the time series multi-channel satellite data with the location features extracted by the location extraction module, the data is input into the shared network module for feature extraction. Then, the output features of the shared network module are fused with the development duration features extracted by the time extraction module to obtain shared features. The time series multi-channel satellite data difference is input into the development network module to extract the development features, and then multiplied with the intensity weight features and wind circle weight features extracted by the category extraction module to obtain the intensity development features and wind circle development features; The shared features and the intensity development features are spliced and input into the intensity decoder to estimate the intensity of the typhoon. The shared features and the wind circle development features are spliced and input into the wind circle decoder to estimate the radius of the wind circle of the typhoon. The position extraction module is composed of four fully connected layers, and the longitude and latitude data of the typhoon center are input into the position extraction module to extract the position features of the typhoon; The time extraction module is composed of two fully connected layers. The typhoon existence duration data is input into the time extraction module to extract the typhoon development duration characteristics; The category extraction module is composed of two identical multilayer perceptrons, each of which includes two fully connected layers. The typhoon intensity level data is input into the category extraction module to extract the intensity weight feature and the wind circle weight feature; The shared network module includes four feature extraction layers and two channel residual connections, each feature extraction layer includes a 2D convolution and maximum pooling layer, where: The output of the second feature extraction layer is connected to the output of the third feature extraction layer through a channel residual connection, and then input to the fourth feature extraction layer; The input of the fourth feature extraction layer is connected to the output of the fourth feature extraction layer through a channel residual connection as the output feature of the shared network module; The development network module consists of three feature extraction layers, each of which includes a 3D convolution and maximum pooling layer.
