Multi-source meteorological observation information classification method and system

Abstract

The invention discloses a multi-source meteorological observation information classification method and system, and relates to the field of meteorological observation information, and the method comprises the steps: converting all to-be-classified multi-source meteorological observation information into a standard format; generating a corresponding mapping vector for each piece of multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each piece of multi-source meteorological observation information into an improved multi-layer perceptron model, and generating the probability that each piece of multi-source meteorological observation information belongs to each category; and according to the probability that each piece of multi-source meteorological observation information belongs to each category, judging the category to which each piece of multi-source meteorological observation information belongs. The invention provides a classification method specially suitable for a multi-source meteorological observation information scene, classification is carried out by fully utilizing the similarity contained in the semantic space of the meteorological observation information, accurate classification can be carried out aiming at a plurality of categories, the structure is simple, the training speed is high, and the convergence efficiency is high.

Description

Technical Field

[0001] The present invention relates to the field of meteorological observation information, and in particular to a multi-source meteorological observation information classification method and system. Background Art

[0002] With the rapid development of meteorological observation systems, the quality of various meteorological observation data is also continuously improving, and various industries have higher and higher requirements for meteorological data products. In order to meet this requirement, the use of data fusion and assimilation technology to integrate multi-source meteorological observation information from different sources is an effective way to obtain high-precision and high-quality meteorological products. At present, China and foreign countries have achieved rich research results in multi-source meteorological observations. These results cover multiple fields such as land, ocean, and atmosphere, and have played an important role in weather and climate research and operations, as well as disaster prevention and mitigation applications.

[0003] Although there are many research results in multi-source meteorological observation at this stage, there are still some problems. In order for the meteorological data processing center to efficiently process and utilize multi-source meteorological observation information, it is necessary to classify it by category. The current methods for meteorological data classification do not fully utilize the semantics of meteorological data itself for classification, and the methods for meteorological target classification cannot be transferred to the scene of multi-source meteorological observation information classification. At the same time, other large-scale classification models have complex structures and large number of parameters, and are not suitable for the scene of multi-source meteorological observation information classification. Summary of the invention

[0004] In view of the above-mentioned technical deficiencies, the purpose of the present invention is to provide a multi-source meteorological observation information classification method and system, aiming to solve the problem that the current method for meteorological data classification does not fully utilize the semantics of the meteorological data itself for classification, and the method for meteorological target classification cannot be migrated to the scenario of multi-source meteorological observation information classification.

[0005] In order to solve the above technical problems, the present invention provides a multi-source meteorological observation information classification method and system.

[0006] The first aspect disclosed in the present application provides a method for classifying multi-source meteorological observation information, the method comprising: converting all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; generating a corresponding mapping vector for each multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and determining the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0007] Preferably, the method of generating a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format includes: after removing symbols and numerical data from all multi-source meteorological observation information converted into a standard format, performing a word segmentation operation on the remaining text content, and recording the set of all words as O; calculating each word according to the set O. The word frequency in the m-th multi-source meteorological observation information ,in For each word in O The number of times it appears in the m-th multi-source meteorological observation information, where , , d is the number of multi-source meteorological observation information; calculate the inverse text frequency of each word in the set O ,in, To contain words The number of multi-source meteorological observation information; the word frequency With the inverse text frequency Multiply them together to get the mapping vector of the m-th multi-source meteorological observation information ,in, , .

[0008] Preferably, the improved multi-layer perceptron model is defined as follows:

[0009] in, for classifier, GELU is the Gaussian error linear unit function, and is the projection matrix, and is the offset vector, is the mapping vector of the m-th multi-source meteorological observation information, The components of are the probabilities that the m-th multi-source meteorological observation information belongs to each category.

[0010] Preferably, the training and testing of the improved multi-layer perceptron model includes the following steps: establishing a multi-source meteorological observation information data set, and labeling the sample label of the category to which each multi-source meteorological observation information belongs, wherein the sample label is in the form of a one-hot vector; dividing the multi-source meteorological observation information data set into a training set and a test set in a ratio of 8 to 2; using the training set to train the multi-layer perceptron model by back propagation and mini-batch gradient descent until the loss function converges, wherein the loss function uses a multivariate cross entropy loss function, defined as , is the i-th sample label of the j-th multi-source meteorological observation information in the training set, is the probability that the j-th multi-source meteorological observation information in the training set belongs to the i-th category, J and I are the size of the training set and the number of categories respectively; the multi-layer perceptron model that has completed the training is tested using the test set to determine whether the accuracy of the multi-layer perceptron model that has completed the training reaches more than 99.97%. If not, retrain; if so, obtain the improved multi-layer perceptron model.

[0011] Preferably, the categories include: ground meteorological information, high-altitude meteorological information, marine meteorological information, meteorological radiation information, atmospheric composition information, weather radar information and satellite meteorological information.

[0012] The second aspect disclosed in the present application provides a multi-source meteorological observation information classification system, which is used for the above-mentioned multi-source meteorological observation information classification method, and the system includes: a standardization module, which is used to convert all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; a mapping vector module, which is used to generate a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format; a probability generation module, which is used to input the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and a classification module, which is used to determine the category to which each multi-source meteorological observation information belongs according to the probability that each multi-source meteorological observation information belongs to each category.

[0013] The third aspect disclosed in the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor implements the following steps when executing the computer program: converting all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; generating a corresponding mapping vector for each multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and determining the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0014] The fourth aspect disclosed in the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the following steps when executed by a processor: converting all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; generating a corresponding mapping vector for each multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and determining the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0015] One or more technical solutions provided in this application have at least the following technical effects or advantages: A classification method specifically suitable for multi-source meteorological observation information scenarios is provided, and the similarity contained in the semantic space of the meteorological observation information itself is fully utilized for classification, and multiple categories can be accurately classified, with a simple structure, fast calculation speed, and no complex neural network model structure. In addition, the Gaussian error linear unit function used in the multi-layer perceptron of the present invention is continuous and smooth over the entire real number domain, which is helpful for gradient propagation during the training process, thereby improving training efficiency and convergence speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1 The figure is a flow chart of a multi-source meteorological observation information classification method.

[0018] Figure 2 This is a schematic diagram of the structure of a multi-source meteorological observation information classification system. DETAILED DESCRIPTION

[0019] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] The embodiment of the present application provides a multi-source meteorological observation information classification method to solve the problem that the current method for meteorological data classification does not fully utilize the semantics of the meteorological data itself for classification, and the method for meteorological target classification cannot be migrated to the scenario of multi-source meteorological observation information classification.

[0021] like Figure 1 As shown, an embodiment of the present application provides a multi-source meteorological observation information classification method, the method comprising: Convert all the multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; Specifically, all multi-source meteorological observation information to be classified will be regarded as a set, and the data cleaning algorithm in Python language will be used or the data mining framework will be used to convert all multi-source meteorological observation information in this set into a unified key-value pair JSON format. The advantage of JSON format is that it can clearly represent the structure, content and attributes of information, which can not only make classification more accurate, but also facilitate subsequent task processing and operation after classification.

[0022] Generate a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format; Input the mapping vector of each multi-source meteorological observation information into the improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; The category to which each multi-source meteorological observation information belongs is determined based on the probability that each multi-source meteorological observation information belongs to each category.

[0023] Furthermore, generating a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format includes: After removing the symbols and numerical data from all multi-source meteorological observation information converted into a standard format, the remaining text content is segmented, and the set of all words is recorded as O; Calculate each word according to the set O The word frequency in the m-th multi-source meteorological observation information ,in For each word in O The number of times it appears in the m-th multi-source meteorological observation information, where , , d is the number of multi-source meteorological observation information; Calculate the inverse text frequency of each word in set O ,in, To contain words The number of multi-source meteorological observation information; The word frequency With the inverse text frequency Multiply them together to get the mapping vector of the m-th multi-source meteorological observation information ,in, , .

[0024] Furthermore, the improved multi-layer perceptron model is defined as follows:

[0025] in, for Classifier, GELU is the Gaussian error linear unit function, defined as is the cumulative distribution function of the standard normal distribution, , and is the projection matrix, and is the offset vector, and the parameters of the projection matrix and the offset vector are initialized by randomly sampling from a Gaussian distribution N(0,0.01) with a standard deviation of 0.01 and a mean of 0. is the mapping vector of the m-th multi-source meteorological observation information, The components of are the probabilities that the m-th multi-source meteorological observation information belongs to each category.

[0026] Furthermore, the training and testing of the improved multi-layer perceptron model includes the following steps: Establish a multi-source meteorological observation information dataset and annotate each multi-source meteorological observation information with a sample label of its category, where the sample label is in the form of a one-hot vector; The multi-source meteorological observation information dataset is divided into a training set and a test set in a ratio of 8 to 2; The multilayer perceptron model is trained using the training set by back propagation and mini-batch gradient descent until the loss function converges. The loss function uses a multivariate cross entropy loss function, which is defined as , is the i-th sample label of the j-th multi-source meteorological observation information in the training set, is the probability that the j-th multi-source meteorological observation information in the training set belongs to the i-th category, J and I are the size of the training set and the number of categories respectively. The purpose of training is to make Try to get as close as possible to the corresponding sample label ; The test set is used to test the trained multilayer perceptron model to determine whether the accuracy of the trained multilayer perceptron model reaches 99.97% or more. If not, the multilayer perceptron model is retrained. If so, the improved multilayer perceptron model is obtained.

[0027] Further, the categories include: ground meteorological information, high-altitude meteorological information, marine meteorological information, meteorological radiation information, atmospheric composition information, weather radar information and satellite meteorological information.

[0028] In summary, the multi-source meteorological observation information classification method provided in the embodiment of the present application has the following technical effects: 1. A classification method specifically suitable for multi-source meteorological observation information scenarios is provided.

[0029] 2. Make full use of the similarity contained in the semantic space of meteorological observation information itself for classification, and accurately classify multiple categories.

[0030] 3. Simple structure, no complex neural network model structure, fast training speed and high convergence efficiency.

[0031] In summary, the multi-source meteorological observation information classification method realizes automatic classification through a computer, thereby improving the efficiency of multi-source meteorological observation information processing.

[0032] Based on the same inventive concept as the multi-source meteorological observation information classification method in the aforementioned embodiment, Figure 2 As shown, the present application provides a multi-source meteorological observation information classification system, the system comprising: A standardization module, the standardization module is used to convert all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; A mapping vector module, the mapping vector module is used to generate a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format; A probability generation module, wherein the probability generation module is used to input the mapping vector of each multi-source meteorological observation information into the improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; A classification module is used to determine the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0033] Furthermore, the mapping vector module of the system specifically includes the following operation steps: After removing the symbols and numerical data from all multi-source meteorological observation information converted into a standard format, the remaining text content is segmented, and the set of all words is recorded as O; Calculate each word according to the set O The word frequency in the m-th multi-source meteorological observation information ,in For each word in O The number of times it appears in the m-th multi-source meteorological observation information, where , , d is the number of multi-source meteorological observation information; Calculate the inverse text frequency of each word in set O ,in, To contain words The number of multi-source meteorological observation information; The word frequency With the inverse text frequency Multiply them together to get the mapping vector of the m-th multi-source meteorological observation information ,in, , .

[0034] Furthermore, the improved multilayer perceptron in the probability generation module of the system is defined as:

[0035] in, for classifier, GELU is the Gaussian error linear unit function, and is the projection matrix, and is the offset vector, is the mapping vector of the m-th multi-source meteorological observation information, The components of are the probabilities that the m-th multi-source meteorological observation information belongs to each category.

[0036] Furthermore, the training and testing of the improved multi-layer perceptron model includes the following steps: Establish a multi-source meteorological observation information dataset and annotate each multi-source meteorological observation information with a sample label of its category, where the sample label is in the form of a one-hot vector; The multi-source meteorological observation information dataset is divided into a training set and a test set in a ratio of 8 to 2; The multilayer perceptron model is trained using the training set by back propagation and mini-batch gradient descent until the loss function converges. The loss function uses a multivariate cross entropy loss function, which is defined as , is the i-th sample label of the j-th multi-source meteorological observation information in the training set, is the probability that the j-th multi-source meteorological observation information in the training set belongs to the i-th category, J and I are the size of the training set and the number of categories respectively; The test set is used to test the trained multilayer perceptron model to determine whether the accuracy of the trained multilayer perceptron model reaches 99.97% or more. If not, the multilayer perceptron model is retrained. If so, the improved multilayer perceptron model is obtained.

[0037] Further, the categories include: ground meteorological information, high-altitude meteorological information, marine meteorological information, meteorological radiation information, atmospheric composition information, weather radar information and satellite meteorological information.

[0038] In one embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: converting all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; generating a corresponding mapping vector for each multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and determining the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0039] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: converting all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; generating a corresponding mapping vector for each multi-source meteorological observation information converted into the standard format; inputting the mapping vector of each multi-source meteorological observation information into an improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; and determining the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

[0040] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0041] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-source meteorological observation information classification method, characterized in that: The method comprises: Convert all the multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; Generate a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format; Input the mapping vector of each multi-source meteorological observation information into the improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; The category to which each multi-source meteorological observation information belongs is determined based on the probability that each multi-source meteorological observation information belongs to each category.

2. The method according to claim 1, characterized in that The step of generating a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format includes: After removing the symbols and numerical data from all multi-source meteorological observation information converted into a standard format, the remaining text content is segmented, and the set of all words is recorded as O; Calculate each word according to the set O The word frequency in the m-th multi-source meteorological observation information ,in For each word in O The number of times it appears in the m-th multi-source meteorological observation information, where , , d is the number of multi-source meteorological observation information; Calculate the inverse text frequency of each word in set O ,in, To contain words The number of multi-source meteorological observation information; The word frequency With the inverse text frequency Multiply them together to get the mapping vector of the m-th multi-source meteorological observation information ,in, , .

3. The method according to claim 1, characterized in that The improved multi-layer perceptron model is defined as follows: in, for classifier, GELU is the Gaussian error linear unit function, and is the projection matrix, and is the offset vector, is the mapping vector of the m-th multi-source meteorological observation information, The components of are the probabilities that the m-th multi-source meteorological observation information belongs to each category.

4. The method according to claim 1 or 3, characterized in that The training and testing of the improved multi-layer perceptron model includes the following steps: Establish a multi-source meteorological observation information dataset and annotate each multi-source meteorological observation information with a sample label of its category, where the sample label is in the form of a one-hot vector; The multi-source meteorological observation information dataset is divided into a training set and a test set in a ratio of 8 to 2; The multilayer perceptron model is trained using the training set by back propagation and mini-batch gradient descent until the loss function converges. The loss function uses a multivariate cross entropy loss function, which is defined as , is the i-th sample label of the j-th multi-source meteorological observation information in the training set, is the probability that the j-th multi-source meteorological observation information in the training set belongs to the i-th category, J and I are the size of the training set and the number of categories respectively; The test set is used to test the trained multi-layer perceptron model to determine whether the accuracy of the trained multi-layer perceptron model reaches 99.97% or more. If not, retrain the model. If so, the improved multi-layer perceptron model is obtained.

5. The method according to claim 1 or 3, characterized in that: The categories include: ground meteorological information, high-altitude meteorological information, marine meteorological information, meteorological radiation information, atmospheric composition information, weather radar information and satellite meteorological information.

6. A multi-source meteorological observation information classification system, characterized in that: The system comprises: A standardization module, the standardization module is used to convert all multi-source meteorological observation information to be classified into a standard format, wherein the standard format is in the form of key-value pairs; A mapping vector module, the mapping vector module is used to generate a corresponding mapping vector for each multi-source meteorological observation information converted into a standard format; A probability generation module, wherein the probability generation module is used to input the mapping vector of each multi-source meteorological observation information into the improved multi-layer perceptron model to generate the probability that each multi-source meteorological observation information belongs to each category; A classification module is used to determine the category to which each multi-source meteorological observation information belongs based on the probability that each multi-source meteorological observation information belongs to each category.

7. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of a multi-source meteorological observation information classification method described in any one of claims 1 to 5 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of a multi-source meteorological observation information classification method described in any one of claims 1 to 5 are implemented.

Images