Typhoon identification and eye positioning multi-task system based on mask stream key point detection

Abstract

The application relates to a typhoon identification and wind eye positioning multi-task system based on mask flow key point detection, which comprises the following steps: collecting an infrared cloud picture, dividing the infrared cloud picture into an initial training set and an initial test set in a preset mode; performing picture division processing on the initial training set based on a Mask algorithm; training a model of the system by using the training set after the picture division processing, inferring three key point coordinates of a typhoon, comparing the three key point coordinates with real three key point coordinates, and optimizing the model; and testing the model of the system by using the initial test set. The application has the beneficial effects that the problem of complicated calculation faced by a traditional meteorological method is overcome, the speed and efficiency of the detection process are improved, the accuracy of the final result is improved, and the detection precision of the model is obviously improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of target detection and key point detection in computer vision technology, and particularly relates to a typhoon identification and wind eye positioning multi-task system based on mask stream key point detection. BACKGROUND

[0002] With the progress of satellite remote sensing monitoring technology in recent years, a large amount of real-time electromagnetic wave radiation data can be monitored and infrared remote sensing images can be generated, and how to quickly and efficiently utilize a large amount of data has become a difficult problem. Traditional meteorologists study typhoons from the texture and statistical characteristics of infrared cloud images, and use mathematical methods such as fractal geometry and fuzzy technology to identify. However, these methods are complex and not flexible enough, and lack robustness in complex weather scenarios.

[0003] In recent years, artificial intelligence based on deep learning has developed rapidly, and researchers have begun to try to apply convolutional neural networks (CNN) to typhoon target detection and wind eye positioning tasks, and have achieved good results, but have ignored the optimization of the typhoon scene, thus leading to a tedious typhoon detection process and a lack of an end-to-end typhoon detection task model. This is because existing deep learning-based typhoon detection work mostly follows the general target detection and key point detection paradigm. According to the traditional general paradigm, most researchers split the detection task into two relatively independent tasks: task one is the target detection task of the typhoon, and task two is the wind eye positioning task.

[0004] Specifically, the target detection task of the typhoon belongs to a subtask of the general target detection task, and it is one of the most important and most challenging tasks in computer vision, and is also an upstream task of key point detection. One of the difficulties of target detection is the uncertainty of the number of targets to be detected in the image.

[0005] And the wind eye detection task belongs to a subtask of key point detection, and the existing mainstream framework is a top-down framework, which adopts a two-stage form, first detects and crops the image of a single object through a target detection model, and then uses a key point detection model to detect a fixed number of key points for each object frame. The top-down framework is very complex in both training and inference processes, and requires very high computing resources.

[0006] In summary, for these two tasks: the target detection task of the typhoon and the wind eye positioning task, it can be seen that:

[0007] 1) The problem faced by the existing traditional meteorological method is that it is complex and not flexible enough, and lacks robustness in the face of complex weather scenarios.

[0008] 2) Existing deep learning-based work mostly adopts a two-stage detection process, and for complex typhoon scenes, a model needs to be set up for each task, including a target detection model for the target detection task of the typhoon and a key point detection model for the eye positioning task, which lacks optimization for the typhoon scene and ignores the internal connection between tasks in this special scene, which is insufficient in detection accuracy and leads to complexity and time-consuming in processing the two tasks at the same time.

[0009] Therefore, when the scene is a complex meteorological environment, both the traditional method and the current deep learning-based artificial intelligence method have shortcomings and do not have higher detection accuracy and reasoning speed. SUMMARY

[0010] In order to overcome the problems exposed in the prior art when simultaneously solving the target detection task and the eye positioning task of the typhoon:

[0011] 1) The existing traditional meteorological method faces the problem that its calculation is complicated and not flexible enough, and its robustness is not enough when facing complex meteorological scenes.

[0012] 2) Existing deep learning-based work mostly adopts a two-stage detection process, and for complex typhoon scenes, a model needs to be set up for each task, including a target detection model for the target detection task of the typhoon and a key point detection model for the eye positioning task, which lacks optimization for the typhoon scene and ignores the internal connection between tasks in this special scene, which is insufficient in detection accuracy and leads to complexity and time-consuming in processing the two tasks at the same time.

[0013] The present application proposes a typhoon recognition and eye positioning multi-task system based on mask flow key point detection, including collecting infrared cloud images, performing typhoon position labeling and eye position labeling on the infrared cloud images containing typhoon positions and eye positions, taking part of the infrared cloud images as an initial training set, and taking the remaining infrared cloud images containing typhoon positions and eye positions as an initial test set, further comprising the following steps:

[0014] Step 1: performing mask algorithm-based sub-image processing on the initial training set;

[0015] Step 2: training the model of the system using the sub-image processed training set, inferring the 3 key point coordinates of the typhoon and comparing them with the real 3 key point coordinates to optimize the model;

[0016] Step 3: testing the model of the system using the initial test set.

[0017] Preferably, the step 1 comprises:

[0018] Step 1.1: Copying the infrared cloud image containing N typhoons into N identical infrared cloud image copies;

[0019] Step 1.2: Keeping one typhoon in each infrared cloud image copy, which is completely different from the typhoon in other infrared cloud image copies, and covering the remaining typhoons in the infrared cloud image copy with a mask algorithm;

[0020] Step 1.3: Randomly shuffling the input order of all infrared cloud image copies in the initial training set.

[0021] Preferably, the step 2 comprises:

[0022] Step 2.1: Scaling the infrared cloud image after the image segmentation processing, and normalizing the scaled infrared cloud image, the normalized matrix is represented as V i，j ∈[0, 1];

[0023] Step 2.2: Inputting the preprocessed infrared cloud image into the CNN backbone model part of the system model to extract low-level image features;

[0024] Step 2.3: Dividing the image containing the low-level image features into multiple patches, mapping the patches through a linear projection function into a d-dimensional embedding layer, outputting visual tokens, and adding the patch position embedding layer p_e to the visual tokens to form a 1D token embedding sequence T = [keypoint tokens], [visual tokens];

[0025] Step 2.4: Inputting the 1D token embedding sequence into the Transformer model part of the system model, processing it by the Transformer model part of the system model, and inferring the position coordinates of the 3 key points of the typhoon in the infrared cloud image, the coordinates of the 3 key points being the upper left corner coordinate p min =(x min , y min ) of the typhoon, the lower right corner coordinate p max =(x min +w, y min +h) of the typhoon, and the eye coordinate p eye =(x eye , y eye ) of the typhoon, wherein w represents the length of the typhoon in the image, and h represents the height of the typhoon in the image;

[0026] Step 2.5: Using the heat map regression method to calculate the loss and optimize the system model.

[0027] Preferably, the step 2.4 comprises:

[0028] Step 2.4.1: the multi-head attention mechanism in the Transformer model part of the model of the system processes the 1D token embedding sequence;

[0029] Step 2.4.2: the three-channel thermal map proportional to the infrared cloud image is mapped by the MLP module in the Transformer model part of the model of the system.

[0030] Step 2.4.3: the positions of the maximum values of the three channels in the thermal map correspond to the positions of the left upper corner, the right lower corner and the eye of the typhoon respectively.

[0031] Preferably, the step 2.5 comprises:

[0032]

[0033] Wherein: h(x, y) is the real coordinates of the three special points of the typhoon in the infrared cloud image, (u, v) is the predicted coordinates of the three special points inferred by the model, and α and σ are the scaling factor and the standard deviation respectively. When the generated thermal map is the same size as the input image, α = 1, and the value range of each point in the thermal map h is (0, 1]. The maximum value 1 is obtained at the special point position.

[0034] Using the MSE loss calculation method:

[0035]

[0036] Wherein s represents the number of pixel points in the thermal map, and α is the weight coefficient to balance the contributions of the typhoon corner point and the typhoon eye.

[0037] Preferably, the step 3 comprises:

[0038] Step 3.1: scaling processing is performed on the infrared cloud image of the test set, and normalization processing is performed on the scaled infrared cloud image, and the normalized matrix is represented as V i，j ∈[0, 1];

[0039] Step 3.2: input the preprocessed infrared cloud image into the CNN backbone model part of the model of the system to extract low-level image features;

[0040] Step 3.3: split the picture containing the low-level image features into a plurality of patches, map the patches to a d-dimensional embedding layer by a linear projection function, output visual tokens, and add the patch position embedding layer p e to the visual tokens to form a 1D token embedding sequence T = [keypoint tokens], [visual tokens];

[0041] Step 3.4: input the 1D token embedding sequence into the Transformer model part of the system model, process it by the Transformer model part of the system model, and infer the position coordinates of the three key points of a certain typhoon in the infrared cloud image. The coordinates of the three key points are the upper left corner coordinate p min = (x min , y min ) of the typhoon, the lower right corner coordinate p max = (x min +w, y min +h) of the typhoon, and the eye coordinate p eye = (x eye , y eye ) of the typhoon, where w represents the length of the typhoon in the image, and h represents the height of the typhoon in the image.

[0042] Step 3.5: perform confidence judgment. When the result is higher than the preset value, go to step 3.6. When the result is equal to or lower than the preset value, go to step 3.7.

[0043] Step 3.6: perform Mask algorithm operation, perform Mask operation on the typhoon area formed by the three special points, cover it with a mask, and return the covered picture to step 3.2. Perform the loop operation of steps 3.2-3.5 again to obtain new three special points in the picture. The number of loop operations is controlled by a preset control mechanism.

[0044] Step 3.7: the test is completed.

[0045] Preferably, the step 3.4 comprises:

[0046] Step 3.4.1: process the 1D token embedding sequence by the multi-head attention mechanism in the Transformer model part of the system model.

[0047] Step 3.4.2: obtain the heat map of the infrared cloud image by the MLP module in the Transformer model part of the system model.

[0048] Step 3.4.3: taking the position coordinates of the three key points of a typhoon in the infrared cloud image as the position coordinates of the three key points of a typhoon in the thermal map.

[0049] Preferably, the step 3.5 comprises:

[0050] The confidence judgment mode is that a threshold t is preset by artificial experience, when the confidence is greater than the threshold t, it indicates that the confidence is higher than the preset value, and when the confidence is less than or equal to the threshold t, it indicates that the confidence is lower than the preset value.

[0051] Preferably, the step 3.6 comprises:

[0052] The pixels in the region formed by the position coordinates of the three key points of a typhoon in the inferred infrared cloud image are subjected to a Mask operation, and the pixel value V of the region image is modified to:

[0053]

[0054] wherein random(1) randomly returns a random number between 0 and 1.

[0055] Preferably, the step 3.6 comprises:

[0056] The preset control mechanism is a confidence and cycle number joint control mechanism.

[0057] The control mechanism comprises that a threshold t is preset by artificial experience, when the confidence is greater than the threshold t and the cycle number is less than r, the cycle starts, and when the confidence is less than or equal to the threshold t or the cycle number is equal to r, the cycle ends.

[0058] The beneficial technical effects of the present application are:

[0059] 1) The model of the system of the present application is an artificial intelligence algorithm comprising a CNN backbone network model, a Transformer model and a Mask algorithm, which overcomes the problem of complicated calculation faced by traditional meteorological methods with powerful computing power, and overcomes the problem of insufficient flexibility of traditional meteorological methods with the self-learning performance of artificial intelligence, so that the artificial intelligence algorithm can produce sufficient robustness when facing complex meteorological scenes.

[0060] 2) Compared with the two models used by the current artificial intelligence algorithm to solve the target detection task and the eye positioning task of a typhoon, the present application uses a Mask process framework to replace the original two models and uses a two-stage process step, omits the process of using a target detection model to perform the target detection task of a typhoon in the first stage, simplifies the total process of the two tasks, saves the resources of the model training and inference process, and improves the speed and efficiency of the detection process.

[0061] 3) By finding the internal relationship between the original two tasks, the reasoning scene is optimized, the target detection task of the typhoon is converted into the detection task of "finding special points", the target detection task is converted into the key point detection task, and the wind eye positioning task is combined, the two tasks which are independent in the field of artificial intelligence in the past are connected internally through this conversion, not only the two tasks become one task, the complexity of the task is reduced, more importantly, strong correlation appears between the two tasks, the two tasks provide reasoning basis for each other, the accuracy of the final result is improved, compared with the traditional scheme, the two tasks are almost independent of each other, it is difficult to have the beneficial effects of the present application;

[0062] 4) Through the pre-processing of model training based on the Mask algorithm-based subgraph processing method, the traditional attention dispersion problem of the model is effectively avoided, the interference between multiple typhoon instances in the same infrared cloud image is eliminated, and the detection accuracy of the model is obviously improved. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0064] Figure 1 Figure 1: Comparison diagram of traditional two-stage detection process and single-stage detection process of the present application.

[0065] Figure 2 Figure 2: Cycle detection flowchart of the present application based on Mask operation.

[0066] Figure 3 Figure 3: Subgraph processing flowchart of the present application based on Mask algorithm.

[0067] Figure 4 Figure 4: Heat map generated by infrared cloud image and labeled eye coordinates of the present application.

[0068] Figure 5 Figure 5: Comparison diagram of traditional subgraph processing without using Mask algorithm and subgraph processing using Mask algorithm of the present application. DETAILED DESCRIPTION

[0069] With reference to the drawings and embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0070] Additional aspects and advantages of the present application will be described in the following description, which will become apparent from the following description, or will be learned by practice of the present application.

[0071] As Figures 1-2 shown, in order to solve the problems in the background art:

[0072] 1) The existing traditional meteorological method is complicated and not flexible enough, and its robustness is not enough when facing complex weather scenarios.

[0073] 2) The existing deep learning-based work mostly adopts a two-stage detection process, and needs to set a model for each task for complex typhoon scenarios, including a target detection model for the target detection task of the typhoon and a key point detection model for the eye positioning task, which lacks optimization for the typhoon scene and ignores the internal connection between tasks in this special scene, which is insufficient in detection accuracy and also leads to complexity and time-consuming in processing the two tasks at the same time.

[0074] As Figures 1-5 shown, the present application proposes a typhoon recognition and eye positioning multi-task system based on mask stream key point detection, which includes:

[0075] First, the infrared cloud image needs to be collected and divided into an initial training set and an initial test set in a predetermined manner, and the ratio of the former to the latter is about 7:3 to ensure the training of the model. In practice, the meteorological detection system will collect a large amount of data every day, and these data need to be monitored for a long time. If this process is always handled in the traditional way, it will lead to low efficiency, and the assistance of artificial intelligence is needed.

[0076] The initial training set refers to the infrared cloud image containing the position of the typhoon and the position of the eye, which is labeled with the position of the typhoon and the position of the eye. One part of the infrared cloud image is used as the initial training set, and the purpose is to use the model of the system of the present application to infer the position of the typhoon and the position of the eye in the infrared cloud image, and then compare it with the known position to train the model.

[0077] The initial test set refers to the remaining infrared cloud image containing the position of the typhoon and the position of the eye, which is used as the initial test set. These test sets are used to test the model of the system of the present application.

[0078] AsFigure 1 As shown, the present application changes the way of solving the target detection task and the eye positioning task of the typhoon by using the traditional two-stage process steps to a single-stage process step. Specifically, the target bounding box [x min , y min , w, h] in the target detection task of the typhoon is expressed in the form of a set of diagonal points [p min , p max ], p min =(x min , y min ), p max =(x min +w, y min +h), w, h respectively represent the length and width of the typhoon in the picture, so that p min and p max respectively represent the upper left corner and the lower right corner of the typhoon instance.

[0079] Therefore, the model of the system of the present application achieves the following purposes: finding three points in an infrared cloud picture, and the three points are internally related, that is, they belong to the same typhoon, that is, the three special point positions of the typhoon: the lower left corner of the typhoon, the upper right corner of the typhoon, and the eye of the typhoon, that is, [p min , p max , p eye ].

[0080] Before training the model of the system, since more than one typhoon may appear in the training infrared cloud picture, and each typhoon has its own three special point positions, more than three special point positions may appear in the same infrared cloud picture, causing the new model to have a problem of attention dispersion. In order to solve the problem, the present application uses a Mask algorithm to perform picture division processing on the initial training set, as shown in Figure 3 , the infrared cloud picture containing N typhoons is copied into N infrared cloud copy pictures, each infrared cloud copy picture selects only one typhoon which is different from the typhoons of other infrared cloud copy pictures, and the remaining typhoons in the infrared cloud copy picture are all covered by the Mask algorithm. After the operation, an infrared cloud picture containing N typhoons will become N infrared cloud copy pictures containing only one typhoon and the order is disturbed, so that the new model only needs to find a typhoon in an infrared cloud copy picture during training, ensuring the attention of the new model.

[0081] After the picture division processing based on the Mask algorithm is completed, the present application uses these infrared cloud copy pictures containing only one typhoon and the order is disturbed as a training set to train the model of the system. The training content includes:

[0082] The infrared cloud picture copy is preprocessed, including scaling processing and normalization processing. Specifically, the scaling processing refers to that the image size of the infrared cloud picture copy involved in the application is 1800*1800, but scaling processing is performed when the model is input, and the obtained image size is 256*256.

[0083] The normalization processing refers to that the infrared cloud picture is represented by a pixel matrix H*w*C, and the pixel value V of the image, when the infrared cloud picture is a gray image, the channel number C=1, and the normalized matrix is represented as V i，j ∈[0, 1], at this time, the heat map obtained after normalization has a size of 64*64, as shown in formula (2). Figure 4

[0084] The preprocessed heat map is input into the convolutional neural network cNN backbone model part of the model of the system, and low-level image features are extracted. The purpose and effect of this process are to provide a more concise input for the next step of the Transformer model, reduce the operation amount, and enable the Transformer model to calculate faster to obtain the result.

[0085] However, the result obtained by the CNN processing cannot be directly input into the Transformer model part of the system model, and the low-level image features need to be converted into a 1D token embedding sequence that can be used by the Transformer model part of the system model. The specific method is as follows: the picture containing the low-level image features is divided into multiple patches, the patch is mapped to a d-dimensional embedding layer through a linear projection function, and the output is a visual token. The patch position embedding layer p_e is added to the visual token to form a 1D token embedding sequence T=[keypoint tokens], [visual tokens].

[0086] The 1D token embedding sequence is input into the Transformer model part of the model of the system, and the position coordinates of the three key points of the typhoon in the infrared cloud picture are inferred by the Transformer model part of the system model. The coordinates of the three key points are the upper left corner coordinate p min =(x min , y min ) of the typhoon, the lower right corner coordinate p max =(x min +w, y min +h) of the typhoon, and the eye coordinate p eye =(x eye , y eye ) of the typhoon. ​

[0087] The specific reasoning process is as follows: the multi-head attention mechanism in the Transformer model part of the model of the system processes the 1D token embedding sequence, and the three-channel thermal map proportional to the infrared cloud image is obtained by mapping by the MLP module in the Transformer model part The positions of the maximum values of the three channels in the thermal map correspond to the positions of the three key points of the typhoon, i.e., the upper left corner, the lower right corner, and the eye of the typhoon, respectively.

[0088] Specifically, taking the thermal map of the present application as an example, its value is composed of 64x64 normalized 0-1 values. The maximum value is the maximum value in the 64x64 value matrix, and the position of the key point is the matrix position index corresponding to the maximum value, for example, if the maximum value in the matrix is obtained at the 4th row and the 5th column, then the position of the key point is (4, 5) in the matrix, and through the proportion of the thermal map and the original image, the coordinate position in the original image can be mapped back.

[0089] After the model of the system is trained and the three special points of the infrared cloud copy image are inferred, the loss is calculated and the model of the system is optimized by using the thermal map regression method.

[0090]

[0091] Wherein: h(x, y) is the true coordinates of the three special points of the typhoon in the infrared cloud image, (u, v) is the predicted coordinates of the three special points inferred by the model, and alpha and sigma are the scaling factor and the standard deviation, respectively. When the generated thermal map is the same size as the input image, alpha = 1, the value range of each point in the thermal map h is (0, 1], and the maximum value 1 is obtained at the special point position.

[0092] Using the MSE loss calculation method:

[0093]

[0094] Where s represents the number of pixel points in the thermal map, alpha is the weight coefficient, used to balance the contributions of the typhoon corner point and the typhoon eye. The experimental results of the present application show that when alpha1 = alpha2 = 1 and alpha3 = 1.2, better performance can be obtained.

[0095] The model of the system of the present application contains an improved Transformer model part. Due to the performance of the Transformer model itself, i.e. the connection between points, because of the attention mechanism in it, tokens exchange information with each other. After training, the Transformer model part in the model of the system of the present application will enable the entire model of the system of the present application to master the connection between the three special points, i.e. the three special points belong to the same typhoon, rather than any combination of three special points of different typhoons. This strong internal connection will change the disadvantage of weak connection between the two tasks in the two stages of the traditional two-stage method, because the two tasks are two independent recognition contents and the correlation is weak, but the Transformer model part in the model of the system of the present application greatly enhances the connection between the two tasks, i.e. the connection between points.

[0096] After training, the model of the system can be tested using the initial test set. It includes:

[0097] The infrared cloud images of the test set are preprocessed, and the division of the images during training is no longer performed.

[0098] Specifically, the preprocessing includes scaling the infrared cloud images in the test set, i.e. scaling the image size of the infrared cloud images in the test set from 1800x1800 to 256x256. Then, the scaled infrared cloud images are normalized, including: representing the infrared cloud images as a pixel matrix HxWxC, and the pixel value of the image is V. When the infrared cloud images are grayscale images, the number of channels C=1, and the normalized matrix is represented as V i，j ∈[0, 1], at this time, the heat map obtained after normalization is shown as Figure 4 .

[0099] The preprocessed heat map is input into the convolutional neural network CNN backbone model part in the model of the system in the same way as the aforementioned training method, and low-level image features are extracted. The purpose and effect of this process are: to provide a more concise input for the next Transformer model, reduce its computational load, and enable the Transformer model to calculate faster to obtain the result.

[0100] However, the results obtained by the CNN cannot be directly input into the Transformer model part of the system model, and the low-level image features need to be converted into a 1D token embedding sequence that can be used by the Transformer model part of the system model. The specific method is as follows: the picture containing the low-level image features is divided into multiple patches, and the patches are mapped into a d-dimensional embedding layer by a linear projection function, outputting visual tokens. The patch position embedding layer p_e is added to the visual tokens to form a 1D token embedding sequence T = [keypoint tokens], [visual tokens].

[0101] The 1D token embedding sequence is input into the Transformer model part of the system model, and the Transformer model part of the system model is processed to infer the position coordinates of the three key points of the typhoon in the infrared cloud image. The coordinates of the three key points are the upper left corner coordinate p min = (x min , y min ) of the typhoon, the lower right corner coordinate p max = (x min +w, y min +h) of the typhoon, and the eye coordinate p eye = (x eye , y eye ) of the typhoon.

[0102] The specific inference process is as follows: the multi-head attention mechanism in the Transformer model part of the system model processes the 1D token embedding sequence, and the MLP module in the Transformer model part maps to a three-channel thermal map proportional to the infrared cloud image. The positions of the maximum values in the three channels of the thermal map correspond to the positions of the upper left corner, lower right corner, and eye of the typhoon, respectively.

[0103] Specifically, taking the thermal map of the present application as an example, its value is composed of 64x64 normalized 0-1 values. The maximum value is the maximum value in the 64x64 value matrix, and the position of the key point is the matrix position index corresponding to the maximum value, such as the 4th row and 5th column of the matrix. The maximum value is obtained, and the position of the key point is (4, 5) in the matrix. Through the proportion of the thermal map and the original image, the coordinate position of the original image can be mapped back.

[0104] After the above steps, the model of the system of the application has found 3 special points from an infrared cloud image, and determined the positions of the upper left corner, the lower right corner and the eye of a certain typhoon.

[0105] At this time, confidence judgment needs to be performed, as shown in the formula (1) : Figure 2 The three special points may not represent a typhoon. Due to the use of different key point models, the generation of confidence is slightly different, but it is generated before and after the generation of the heat map. Some models add a branch to map the heat map to obtain the confidence after obtaining the heat map. Some models add a branch to map the feature map to obtain the confidence before the heat map. Some models directly take the maximum value of the heat map as the confidence (the value range of the maximum value is also between 0 and 1). The model of the system used in the application directly selects the maximum value as the confidence.

[0106] As shown in the formula (2) : Figure 2 If the confidence is higher than the threshold t, it means that the three special points found in this cycle are a typhoon and its eye, rather than other meteorological entities, so other typhoons and their eyes that may exist can be searched for. However, if the confidence is lower than or equal to the threshold t, it means that the three special points found in this cycle are not a typhoon and its eye, and this cycle ends, indicating that the infrared cloud image has been processed. A large number of experiments have proved that the threshold t is preset by artificial experience, and the threshold t = 0.5 has achieved good test results.

[0107] Specifically, when the infrared cloud image may have other typhoons and their eyes, Mask algorithm operation needs to be performed to mask cover the typhoon area formed by the three special points, that is, to perform Mask operation on the pixels in the area formed by the position coordinates of the three key points of a certain typhoon in the inferred infrared cloud image, and modify the pixel value V of the area image to:

[0108]

[0109] Wherein random(1) randomly returns a random number between 0 and 1.

[0110] The covered picture is returned to the preprocessing part in the test process, and is processed again by the convolutional neural network (CNN) backbone model part and the Transformer model part in the system of the application. New special points of a typhoon may be found again, but the new special point may not be a typhoon, and thus a confidence judgment needs to be performed again. If the confidence is higher than a threshold t, it means that the three special points found in this cycle are a typhoon and its eye, rather than other meteorological entities, and the above cycle can be repeated. If the confidence is lower than or equal to the threshold t, the cycle ends, and the infrared cloud image processing is completed, and all typhoons in the infrared cloud image have been found.

[0111] Meanwhile, for a general typhoon data set, there are not more than three typhoons in one infrared cloud image, and thus the application adopts a joint control mode of confidence and cycle number, where r represents the cycle number, and the application sets r = 5.

[0112] Since the application abandons the traditional two-stage process for processing two tasks and adopts a single-stage process for simultaneously processing two tasks, obvious progress and beneficial effects are brought.

[0113] Specifically, from the training perspective, the two-stage process needs to train two independent models, and the training of the target detection model in the first stage often requires a high cost. The application only needs to train one model.

[0114] Specifically, from the inference perspective, the mask stream in the application refers to a Mask operation, which is a very fast algorithm rather than a model, and the time cost is almost negligible. The application uses an algorithm to replace a model, which is equivalent to saving the inference time of the first stage in the traditional two-stage process. More specifically, in the two-stage process, the first stage uses one model to infer an image, which takes 0.05 s and detects two typhoon instances. In the second stage, the two typhoon instances are sequentially sent to another model, and each image detection also takes 0.05 s. Therefore, the two-stage process takes a total of 0.05 s + 0.05 s x 2 = 0.15 s. In the application, since only a single-stage process is used, the time required is only 0.05 s x 2 = 0.1 s. The inference speed of each typhoon is improved by 25%, and the more the number of typhoons in the infrared cloud image, the more obvious the improvement.

[0115] In addition to the improvement in speed, the application also significantly improves the accuracy and precision.

[0116] Specifically, the two-stage process traditionally used to process the two tasks has obvious defects in error propagation, i.e. in the first stage process, there may be cases of missed detection or false detection (the detected area does not contain a typhoon or the overlapping area containing the typhoon is too small), and these errors will be passed on to the subsequent second stage process. This will result in poor final typhoon and typhoon eye results. However, the single-stage process of the present application simultaneously processes the two tasks, which to some extent avoids the problem of error propagation.

[0117] Specifically, simultaneously processing the two tasks also brings the benefit of mutual promotion between tasks. Because for the model of the system of the present application, compared with the traditional way, the label of the key point is additionally added in the input information, which is equivalent to additional information, which is more helpful for the model to learn. This means that when the model of the system of the present application locates the left upper corner and the right lower corner of the typhoon, it will consider the position of the eye of the typhoon and correct it according to the position of the eye, and when locating the position of the eye, it will also adjust according to the positions of the left upper corner and the right lower corner of the typhoon, so that the simultaneous completion of the two tasks will have some improvement in accuracy compared with the separate completion of one of the tasks.

[0118] At the same time, the present application uses Mask algorithm-based subgraph processing, which greatly improves the accuracy of the inference of the model of the system compared with the traditional subgraph processing without using Mask algorithm, and also improves the accuracy of the final typhoon prediction. As shown in Figure 5 The figure shows the difference between the two groups of infrared cloud images with and without Mask algorithm-based subgraph processing. Specifically, (b) and (c) are heat map visualization graphs in the inference stage, where the higher the attention of the model of the system to the image, the more the image is biased towards red, and vice versa. The model (b) does not use Mask algorithm-based subgraph operation in the training process, and the model (c) uses Mask algorithm-based subgraph operation. For the two given infrared cloud images containing two typhoon instances, it can be seen that the model (b) locates the right lower corner of typhoon (1) and the left upper corner of typhoon (2). After the first round of inference, the model (b) masks the red box part of the image, and since the two target typhoons have been blocked, the model (b) cannot continue to infer the correct result in the second inference, so there is only one result (the confidence of the second inference does not reach the threshold). Model (c) can accurately give the prediction result because the model only sees a single typhoon instance during training, so the attention is not distracted.

[0119] As used herein, and unless otherwise indicated, all technical terms, including terms with acronyms, have the same meaning as commonly understood by one of ordinary skill in the art in the field of the application. As used herein, unless indicated otherwise, the term "about" means ± 10 %.

[0120] It is to be understood that the components of the devices in the embodiments can be distributed in the devices of the embodiments as described in the embodiments, or can be changed to be located in one or more devices different from the embodiments. The components of the above embodiments can be combined into one component, or can be further split into multiple sub-components.

[0121] Finally, it should be noted that the above embodiments are only used to illustrate but not to limit the technical solutions of the present application. Although the present application is described in detail with reference to the above embodiments, it should be understood by those skilled in the art that the present application can still be modified or replaced equivalently without departing from the spirit and scope of the present application. Any modification or partial replacement should be covered in the scope of the claims of the present application.

Claims

1. A multi-task system for typhoon identification and eye location based on masked stream keypoint detection, comprising collecting infrared cloud images, labeling the typhoon location and eye location of the infrared cloud images containing the typhoon location and eye location, using a portion of the infrared cloud images as an initial training set, and using the remaining infrared cloud images containing the typhoon location and eye location as an initial test set, characterized in that: It also includes the following steps: Step 1: Perform graph partitioning on the initial training set based on the Mask algorithm; Step 2: Use the training set after the image splitting process to train the model of the system, infer the coordinates of the three key points of the typhoon and compare them with the actual coordinates of the three key points to optimize the model; Step 3: Test the model of the system using the initial test set; Step 1 includes: Step 1.1: Copy the infrared cloud image containing N typhoons into N identical copies of the infrared cloud image; Step 1.2: For each infrared cloud image copy, retain one typhoon, and the typhoon is completely different from the typhoons in other infrared cloud image copies. Then, use the Mask algorithm to mask all the other typhoons in the infrared cloud image copy. Step 1.3: Randomly shuffle the input order of all infrared cloud image copies in the initial training set; Step 2 includes: Step 2.1: Scale the infrared cloud image after image segmentation, and normalize the scaled infrared cloud image. The normalized matrix is ​​represented as V. i,j ∈[0,1]; Step 2.2: Input the infrared cloud image preprocessed in Step 2.1 into the CNN backbone model of the system model and extract low-level image features; Step 2.3: Segment the image containing the low-level image features into multiple patches, map the patches to a d-dimensional embedding layer through a linear projection function, and output as visual tokens. The patch position embedding layer p_e is added to the visual tokens to form a 1D token embedding sequence T = [keypoint tokens], [visual tokens]. Step 2.4: Embed the 1D token into the Transformer model part of the sequence input system model. The Transformer model part of the system model processes the token and infers the position coordinates of the three key points of the typhoon in the infrared cloud image. The coordinates of the three key points are the coordinates of the upper left corner of the typhoon, p. min =(x min ,y min The lower right corner coordinates of the typhoon are p. max =(x min +w,y min +h) and the eye coordinates of the typhoon p eye =(x eye ,y eye ), where w represents the length of the typhoon in the image and h represents the height of the typhoon in the image; Step 2.5: Use heatmap regression to calculate the loss and optimize the system model.

2. A multi-task system for typhoon identification and eye location based on masked stream key point detection as described in claim 1, characterized in that: Step 2.4 includes: Step 2.4.1: The multi-head attention mechanism in the Transformer model part of the system model processes the 1D token embedding sequence; Step 2.4.2: Obtain a three-channel thermal image proportional to the infrared cloud image by mapping the MLP module in the Transformer model part of the system model. Step 2.4.3: The positions of the maximum values ​​of the three channels in the heat map correspond to the positions of the upper left corner, lower right corner and eye of the typhoon's three key points, respectively.

3. The multi-task system for typhoon identification and eye location based on masked stream key point detection according to claim 1, characterized in that: Step 2.5 includes: Where: h(x,y) are the true coordinates of the three special points of the typhoon in the infrared cloud image, (u,v) are the inferred coordinates of the three special points inferred by the model, α and σ are the scaling factor and standard deviation, respectively. When the generated heat map is the same size as the input image, α = 1, and the value range of each point in the heat map h is (0,1], with the maximum value of 1 obtained at the special point position; Using the MSE loss calculation method: Where s represents the number of pixels in the heatmap, and α is a weighting coefficient used to balance the contributions of the typhoon corner and the typhoon eye.

4. The multi-task system for typhoon identification and eye location based on masked stream key point detection according to claim 1, characterized in that: Step 3 includes: Step 3.1: Scale the infrared cloud images of the test set, and then normalize the scaled infrared cloud images. The normalized matrix is ​​represented as V. i,j ∈[0,1]; Step 3.2: Input the infrared cloud image preprocessed in Step 3.1 into the CNN backbone model of the system model and extract low-level image features; Step 3.3: Segment the image containing the low-level image features into multiple patches, map the patches to a d-dimensional embedding layer through a linear projection function, and output as visual tokens. The patch position embedding layer p_e is added to the visual tokens to form a 1D token embedding sequence T = [keypoint tokens], [visual tokens]. Step 3.4: Embed the 1D token into the Transformer model part of the sequence input system model. The Transformer model part of the system model processes the token and infers the position coordinates of three key points of a typhoon in the infrared cloud image. The coordinates of the three key points are the coordinates of the upper left corner of the typhoon. min =(x min ,y min The lower right corner coordinates of the typhoon are p. max =(x min +w,y min +h) and the eye coordinates of the typhoon p eye =(x eye ,y eye ), where w represents the length of the typhoon in the image and h represents the height of the typhoon in the image; Step 3.5: Perform a confidence level judgment. If the result is higher than the preset value, proceed to step 3.

6. If the result is equal to or lower than the preset value, proceed to step 3.

7. Step 3.6: Perform the Mask algorithm operation to mask the typhoon area formed by the three key points. Return the masked image to step 3.2 and repeat the loop operation of steps 3.2-3.5 to obtain three new key points in the image. The number of loop operations is controlled by a preset control mechanism. Step 3.7: This test is now complete.

5. A multi-task system for typhoon identification and eye location based on mask stream key point detection according to claim 4, characterized in that: Step 3.4 includes: Step 3.4.1: The 1D token embedding sequence is processed by the multi-head attention mechanism in the Transformer model part of the system model; Step 3.4.2: Obtain the heat map of the infrared cloud image by mapping the MLP module in the Transformer model part of the system model; Step 3.4.3: Use the position of the maximum value in the heat map as the position coordinates of the three key points of a typhoon in the infrared cloud map.

6. The multi-task system for typhoon identification and eye location based on masked stream key point detection according to claim 4, characterized in that: Step 3.5 includes: The confidence level is determined by a preset threshold t based on human experience. When the confidence level is greater than the threshold t, it indicates that the confidence level is higher than the preset value. When the confidence level is less than or equal to the threshold t, it indicates that the confidence level is lower than the preset value.

7. A multi-task system for typhoon identification and eye location based on masked stream key point detection according to claim 4, characterized in that: Step 3.6 includes: The pixels in the region formed by the coordinates of three key points of a typhoon in the inferred infrared cloud image are masked, and the pixel value V of the image in that region is modified as follows: in random(1) returns a random number between 0 and 1.

8. A multi-task system for typhoon identification and eye location based on masked stream key point detection according to claim 4, characterized in that: Step 3.6 includes: The preset control mechanism is a joint control mechanism of confidence level and number of iterations; The control mechanism includes: a threshold t is preset by human experience; when the confidence level is greater than the threshold t and the number of iterations is less than r, the loop starts; when the confidence level is less than or equal to the threshold t or the number of iterations is equal to r, the loop ends.

Images