Meteorological ocean database access control method and device

Abstract

The invention discloses an access control method and device for a meteorological ocean database. The method comprises the following steps: acquiring to-be-accessed data information and the meteorological ocean database; the meteorological ocean database comprises index information and meteorological ocean data; the meteorological ocean data comprises radar data, satellite inversion data and meteorological ocean raster data; preprocessing the meteorological ocean database to obtain a preprocessed meteorological ocean database; performing information matching processing on the to-be-accessed data information and the preprocessed meteorological ocean database to obtain matching index information; and based on the matching index information, retrieving from the meteorological ocean database to obtain corresponding meteorological ocean data, and taking the obtained meteorological ocean data as access result information of the meteorological ocean database.

Description

Technical Field

[0001] This invention relates to the fields of meteorological and oceanographic data processing and data modeling, specifically to an access control method and apparatus for a meteorological and oceanographic database. Background Technology

[0002] Meteorological and oceanographic data are characterized by multi-source heterogeneity, strong spatiotemporal correlation, and rapid dynamic changes. Traditional database access technologies face the following challenges when processing this type of data: 1. Low data organization and retrieval efficiency: Traditional relational databases struggle to efficiently store and query multidimensional meteorological and oceanographic data (such as data including time, space, and barosphere dimensions). When users need to retrieve multiple dimensions simultaneously (e.g., "a certain time period + a specific latitude and longitude range + a specific forecast model"), complex multi-table join queries are required, resulting in long response times.

[0003] 2. Insufficient data quality and preprocessing: Meteorological and oceanographic data often contain noise, missing values, or inconsistent formats, directly affecting retrieval accuracy. For example, radar data from different sources may have different sampling frequencies, and without preprocessing, this can lead to matching errors.

[0004] 3. Lack of comprehensive consideration of forecast model values, spatiotemporal continuity, and spatial correlation.

[0005] In view of the aforementioned problems with existing meteorological and oceanographic databases, there is an urgent need to develop an efficient and fast access control method and device for meteorological and oceanographic databases. Summary of the Invention

[0006] This invention primarily addresses the problems of low data organization and retrieval efficiency, insufficient data quality and preprocessing, and lack of comprehensive consideration of forecast model values, spatiotemporal continuity, and spatial correlation in existing meteorological and oceanographic databases. This invention discloses an access control method and device for meteorological and oceanographic databases.

[0007] In a first aspect, this invention discloses an access control method for a meteorological and oceanographic database, comprising: S1, acquire the data information to be accessed and the meteorological and oceanographic database; the meteorological and oceanographic database includes index information and meteorological and oceanographic data; the meteorological and oceanographic data includes radar data, satellite inversion data, and meteorological and oceanographic grid data; S2, preprocess the meteorological and oceanographic database to obtain the preprocessed meteorological and oceanographic database; S3, perform information matching processing on the data information to be accessed and the preprocessed meteorological and oceanographic database to obtain matching index information; S4. Based on the matching index information, retrieve the corresponding meteorological and oceanographic data from the meteorological and oceanographic database, and use the obtained meteorological and oceanographic data as the access result information of the meteorological and oceanographic database.

[0008] The meteorological and oceanographic database is implemented using a KEY-VALUE distributed in-memory database; the index information serves as the KEY value of the database; the meteorological and oceanographic data corresponding to each index information serves as the VALUE value corresponding to the KEY value; the index information includes forecast model value, forecast start time, forecast lead time, pressure layer, spatial longitude range, and spatial latitude range; The data information to be accessed includes the forecast model value to be retrieved, the start time, the forecast lead time, the pressure layer, the spatial longitude range, and the spatial latitude range.

[0009] The preprocessing of the meteorological and oceanographic database to obtain a preprocessed meteorological and oceanographic database includes: S21, Perform data cleaning and processing on the meteorological and oceanographic database to obtain the first database; S22, Perform category detection processing on the first database to obtain the second database; S23, perform pattern reduction on the second database to obtain a preprocessed meteorological and oceanographic database.

[0010] The step of matching the data to be accessed with the preprocessed meteorological and oceanographic database to obtain matching index information includes: S31, For the data information to be accessed, perform classification similarity calculations on each index information in the preprocessed meteorological and oceanographic database to obtain the corresponding total similarity vector; S32, calculate the similarity metric for each total similarity vector to obtain the corresponding similarity metric value; S33, determine the index information with the smallest similarity metric value, which is the matching index information.

[0011] The expression for calculating the similarity metric is: , in, It is the ReLU activation function. Let zs be the i-th element of the total similarity vector, and zs be the similarity metric.

[0012] The step involves calculating the classification similarity between the data to be accessed and each index in the preprocessed meteorological and oceanographic database to obtain a corresponding total similarity vector, including: S311, perform a first similarity calculation on the forecast model value to be retrieved in the data information to be accessed and the corresponding element in each index information in the preprocessed meteorological and oceanographic database to obtain the first similarity value of each index information; S312, the start time and forecast validity of the data to be retrieved in the data to be accessed are compared with the corresponding elements in each index information of the preprocessed meteorological and oceanographic database to calculate the second similarity, and the second similarity value of each index information is obtained. S313, For the pressure layer, spatial longitude range, and spatial latitude range to be retrieved in the data information to be accessed, and the corresponding elements in each index information in the preprocessed meteorological and oceanographic database, a third similarity calculation is performed to obtain the third similarity value of each index information. S314, using the first similarity value, second similarity value and third similarity value of each index information as vector elements, construct the total similarity vector corresponding to the index information.

[0013] The first similarity calculation includes: The forecast pattern value to be retrieved in the data information to be accessed is represented as a first vector; The forecast model values ​​in the index information of the preprocessed meteorological and oceanographic database are represented as a second vector; Subtract the first vector from the second vector to obtain the difference vector; Perform an EEMD transformation on the difference vector to obtain the transformed vector; A first fusion difference calculation is performed on the transformed vector, the first vector, and the second vector to obtain a first similarity value of the index information; The expression for calculating the first fusion difference is: , in, The first similarity value is given by N, where N is the length of the transformed vector. and These are the i-th elements of the first vector and the second vector, respectively. Let i be the i-th element of the transformation vector.

[0014] A second aspect of the present invention discloses an access control device for meteorological and oceanographic data, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the access control method for the meteorological and oceanographic database.

[0015] In a third aspect of this invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions, and when the computer instructions are invoked by a computer, they are used to execute the access control method for the meteorological and oceanographic database.

[0016] A fourth aspect of this invention discloses an information data processing terminal, which is used to implement the access control method for the meteorological and oceanographic database.

[0017] The beneficial effects of this invention are as follows: This invention employs a distributed in-memory database architecture, offering high concurrency advantages. By using a key-value distributed in-memory database, index information (forecast patterns, spatiotemporal ranges, etc.) is used as the key, and meteorological and oceanographic data as the value, avoiding the disk I / O bottleneck of traditional databases and supporting tens of thousands of concurrent queries. Based on a multidimensional index structure (forecast pattern values ​​+ spatiotemporal ranges), it supports rapid location; for example, by using spatial filling curves (such as Hilbert curves) to convert multidimensional space into a one-dimensional index, it improves the efficiency of range queries.

[0018] A three-tiered preprocessing mechanism improves the efficiency and accuracy of data access. Data cleaning removes noisy data and fills in missing values; for example, spatiotemporal interpolation algorithms address missing points in radar data, improving data quality. Category detection automatically identifies data types (radar, satellite inversion, etc.), providing a foundation for subsequent model reduction, such as distinguishing sampling frequency differences between different sensors. Model reduction standardizes data formats and units; for example, it normalizes barosphere data from different forecast models to standard units, eliminating format heterogeneity. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention. Detailed Implementation

[0020] To better understand the content of this invention, an embodiment is provided here.

[0021] Figure 1 This is a flowchart illustrating the implementation of the method of the present invention.

[0022] In a first aspect, this invention discloses an access control method for a meteorological and oceanographic database, comprising: S1, acquire the data information to be accessed and the meteorological and oceanographic database; the meteorological and oceanographic database includes index information and meteorological and oceanographic data; the meteorological and oceanographic data includes radar data, satellite inversion data, and meteorological and oceanographic grid data; S2, preprocess the meteorological and oceanographic database to obtain the preprocessed meteorological and oceanographic database; S3, perform information matching processing on the data information to be accessed and the preprocessed meteorological and oceanographic database to obtain matching index information; S4. Based on the matching index information, retrieve the corresponding meteorological and oceanographic data from the meteorological and oceanographic database, and use the obtained meteorological and oceanographic data as the access result information of the meteorological and oceanographic database.

[0023] Spatial latitude and longitude division values ​​are the degree intervals used in the latitude and longitude directions when dividing the Earth's surface. They determine the spatial resolution of meteorological data and play an important role in weather forecasting.

[0024] The meteorological and oceanographic database is implemented using a KEY-VALUE distributed in-memory database; the index information serves as the KEY value of the database; the meteorological and oceanographic data corresponding to each index information serves as the VALUE value corresponding to the KEY value; the index information includes forecast model value, forecast start time, forecast lead time, pressure layer, spatial longitude range, and spatial latitude range; The data information to be accessed includes the forecast model value to be retrieved, the start time, the forecast lead time, the pressure layer, the spatial longitude range, and the spatial latitude range. The meteorological and oceanographic database extracts corresponding meteorological data from meteorological and oceanographic data based on index information, and stores the meteorological data in a table system in binary form. The KEY-VALUE distributed in-memory database can be built using the Apache Cassandra storage system; The preprocessing of the meteorological and oceanographic database to obtain a preprocessed meteorological and oceanographic database includes: S21, Perform data cleaning and processing on the meteorological and oceanographic database to obtain the first database; S22, Perform category detection processing on the first database to obtain the second database; S23, perform pattern reduction on the second database to obtain a preprocessed meteorological and oceanographic database.

[0025] The category detection process involves judging the attributes of each data in the first database and determining whether its attributes are consistent with preset attributes. Data that is inconsistent is then deleted from the first database.

[0026] The data cleaning process includes filling in missing values, smoothing noisy data, and smoothing or deleting outlier points. The pattern reduction process involves taking each type of meteorological and oceanographic data in the second database and performing linear function fitting on the data sequence, using the acquisition time of each data point in the data sequence as the independent variable and the value of each data point as the dependent variable, to obtain the pattern function for the meteorological and oceanographic data. The pattern function for the meteorological and oceanographic data then calculates the acquisition time of each data point in the data sequence to obtain the corresponding pattern value. The absolute value of the difference between each data point and its corresponding pattern value is calculated, and data points whose absolute difference exceeds a preset threshold are deleted from the meteorological and oceanographic data. This process is repeated for each type of meteorological and oceanographic data to obtain a preprocessed meteorological and oceanographic database.

[0027] The step of matching the data to be accessed with the preprocessed meteorological and oceanographic database to obtain matching index information includes: S31, For the data information to be accessed, perform classification similarity calculations on each index information in the preprocessed meteorological and oceanographic database to obtain the corresponding total similarity vector; S32, calculate the similarity metric for each total similarity vector to obtain the corresponding similarity metric value; S33, determine the index information with the smallest similarity metric value, which is the matching index information.

[0028] The expression for calculating the similarity metric is: , in, It is the ReLU activation function. Let zs be the i-th element of the total similarity vector, and zs be the similarity metric.

[0029] The step involves calculating the classification similarity between the data to be accessed and each index in the preprocessed meteorological and oceanographic database to obtain a corresponding total similarity vector, including: S311, perform a first similarity calculation on the forecast model value to be retrieved in the data information to be accessed and the corresponding element in each index information in the preprocessed meteorological and oceanographic database to obtain the first similarity value of each index information; S312, the start time and forecast validity of the data to be retrieved in the data to be accessed are compared with the corresponding elements in each index information of the preprocessed meteorological and oceanographic database to calculate the second similarity, and the second similarity value of each index information is obtained. S313, For the pressure layer, spatial longitude range, and spatial latitude range to be retrieved in the data information to be accessed, and the corresponding elements in each index information in the preprocessed meteorological and oceanographic database, a third similarity calculation is performed to obtain the third similarity value of each index information. S314, using the first similarity value, second similarity value and third similarity value of each index information as vector elements, construct the total similarity vector corresponding to the index information; By calculating class similarity, the similarity calculation is decomposed into forecast model (first similarity), spatiotemporal dimension (second similarity), and spatial range (third similarity), with targeted calculation methods: First similarity: The intrinsic mode function (IMF) of the vector is extracted through EEMD transformation to capture the periodic characteristics of the forecast model, such as identifying the similarity between different numerical models in precipitation forecasts. Second similarity: The dynamic changes of the time series are considered, for example, the similarity between the forecast start time and the forecast lead time is calculated using the DTW (Dynamic Time Warping) algorithm, allowing flexible matching on the time axis. Third similarity: Geographic Information System (GIS) technology is combined to calculate the overlap rate of latitude and longitude ranges and the distance to the center point, for example, using the Haversine formula to calculate spherical distance.

[0030] The first similarity calculation includes: The forecast pattern value to be retrieved in the data information to be accessed is represented as a first vector; The forecast model values ​​in the index information of the preprocessed meteorological and oceanographic database are represented as a second vector; Subtract the first vector from the second vector to obtain the difference vector; Perform an EEMD transformation on the difference vector to obtain the transformed vector; A first fusion difference calculation is performed on the transformed vector, the first vector, and the second vector to obtain the first similarity value of the index information.

[0031] The expression for calculating the first fusion difference is: , in, The first similarity value is given by N, where N is the length of the transformed vector. and These are the i-th elements of the first vector and the second vector, respectively. Let i be the i-th element of the transformation vector.

[0032] The first fusion difference calculation enhances feature extraction through EEMD transformation, decomposing the difference vector into IMF components of different scales to capture multi-scale features of the forecast model, such as short-term fluctuations and long-term trends; it uses a combination of trigonometric and exponential functions to reflect the periodic differences between vectors through phase changes; and it enhances robustness by applying exponential decay to outliers. The first fusion difference calculation is applicable to the similarity calculation of meteorological model vectors, such as capturing the similarity of different numerical models in temperature field forecasts.

[0033] The second similarity calculation includes: ， in, The difference between the start time of the report to be retrieved in the data to be accessed and the start time in the index information of the preprocessed meteorological and oceanographic database, where t is a time variable. The overlap length between the forecast lead time to be retrieved in the data to be accessed and the forecast lead time in the index information of the preprocessed meteorological and oceanographic database; The overlap time length is the length of the overlap time between two forecasts. The formula measures the consistency of the data's time base by the difference in reporting times, while This limits the overlap of forecast lead times, and combining the two can avoid biases caused by simply comparing the start time or forecast lead time. For example, when the start times of two sets of data are close but the forecast lead times overlap little, the calculation results will reflect the difference in the actual usable time overlap. Integral operations enhance the cumulative effect of time intervals. exist Integrating within an interval essentially involves weighted accumulation of differences within a time overlap range. Compared to simple difference or ratio calculations, integration more comprehensively reflects the similarity over the entire overlapping period, avoiding interference from single-point errors.

[0034] The third similarity calculation includes: The pressure layer, spatial longitude range, and spatial latitude range to be retrieved in the data information to be accessed are respectively represented by text vectors to obtain the first retrieval text vector, the second retrieval text vector, and the third retrieval text vector; For each index information in the preprocessed meteorological and oceanographic database, the pressure layer, spatial longitude range, and spatial latitude range are represented by text vectors to obtain the first index text vector, the second index text vector, and the third index text vector. A third similarity value is obtained by performing a fusion correlation calculation on the retrieved first text vector, retrieved second text vector, retrieved third text vector, indexed first text vector, indexed second text vector, and indexed third text vector; The expression for the fusion-related calculation is: , in, Let the magnitude of the difference vector between the indexed i-th text vector and the retrieved i-th text vector be the modulus. To index the i-th text vector and retrieve the similarity value of the i-th text vector obtained under the j-th similarity calculation method. For the preset deviation, the first to fourth similarity calculation methods are Jaccard similarity value, cosine similarity value, TF-IDF vector similarity value, and Levenshtein distance, respectively.

[0035] The similarity value of the TF-IDF vectors is obtained by converting the two text vectors into TF-IDF vectors respectively, and then taking the modulus of the difference vector between the two TF-IDF vectors.

[0036] The text vector representation can be implemented using the word2vec function; The EEMD transformation refers to Integrated Empirical Mode Decomposition.

[0037] The data information to be accessed includes the forecast model value to be retrieved, the start time, the forecast lead time, the pressure layer, the spatial longitude range, and the spatial latitude range. In all embodiments of the present invention, the variables involved in all computational expressions or mathematical functions have been dimensionlessized before computation.

[0038] In all embodiments of the present invention, the values ​​of the independent variables in the input of all computational expressions or mathematical functions meet the reasonable requirements of the input range of the computational expressions or mathematical functions, and can ensure that the computational expressions or mathematical functions can be calculated smoothly without violating physical laws or mathematical rules.

[0039] A second aspect of the present invention discloses an access control device for meteorological and oceanographic data, the device comprising: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the access control method for the meteorological and oceanographic database.

[0040] In a third aspect of this invention, a computer-storable medium is disclosed, wherein the computer-storable medium stores computer instructions, and when the computer instructions are invoked by a computer, they are used to execute the access control method for the meteorological and oceanographic database.

[0041] A fourth aspect of this invention discloses an information data processing terminal, which is used to implement the access control method for the meteorological and oceanographic database.

[0042] The above description is merely an embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principle of the present invention should be included within the scope of the claims of the present invention.

Claims

1. An access control method for a meteorological and oceanographic database, characterized in that, include: S1, obtain the data information to be accessed and the meteorological and oceanographic database; The meteorological and oceanographic database includes index information and meteorological and oceanographic data; The meteorological and oceanographic data includes radar data, satellite inversion data, and meteorological and oceanographic grid data; S2, preprocess the meteorological and oceanographic database to obtain the preprocessed meteorological and oceanographic database; S3, perform information matching processing on the data information to be accessed and the preprocessed meteorological and oceanographic database to obtain matching index information; S4. Based on the matching index information, retrieve the corresponding meteorological and oceanographic data from the meteorological and oceanographic database, and use the obtained meteorological and oceanographic data as the access result information of the meteorological and oceanographic database.

2. The access control method for a meteorological and oceanographic database as described in claim 1, characterized in that, The meteorological and oceanographic database is implemented using a KEY-VALUE distributed in-memory database; the index information serves as the KEY value of the database; the meteorological and oceanographic data corresponding to each index information serves as the VALUE value corresponding to the KEY value; the index information includes forecast model value, forecast start time, forecast lead time, pressure layer, spatial longitude range, and spatial latitude range; The data information to be accessed includes the forecast model value to be retrieved, the start time, the forecast lead time, the pressure layer, the spatial longitude range, and the spatial latitude range.

3. The access control method for a meteorological and oceanographic database as described in claim 2, characterized in that, The preprocessing of the meteorological and oceanographic database to obtain a preprocessed meteorological and oceanographic database includes: S21, Perform data cleaning and processing on the meteorological and oceanographic database to obtain the first database; S22, Perform category detection processing on the first database to obtain the second database; S23, perform pattern reduction on the second database to obtain a preprocessed meteorological and oceanographic database.

4. The access control method for a meteorological and oceanographic database as described in claim 1, characterized in that, The step of matching the data to be accessed with the preprocessed meteorological and oceanographic database to obtain matching index information includes: S31, For the data information to be accessed, perform classification similarity calculations on each index information in the preprocessed meteorological and oceanographic database to obtain the corresponding total similarity vector; S32, calculate the similarity metric for each total similarity vector to obtain the corresponding similarity metric value; S33, determine the index information with the smallest similarity metric value, which is the matching index information.

5. The access control method for a meteorological and oceanographic database as described in claim 4, characterized in that, The expression for calculating the similarity metric is: ， in, It is the ReLU activation function. Let zs be the i-th element of the total similarity vector, and zs be the similarity metric.

6. The access control method for a meteorological and oceanographic database as described in claim 4, characterized in that, The step involves calculating the classification similarity between the data to be accessed and each index in the preprocessed meteorological and oceanographic database to obtain a corresponding total similarity vector, including: S311, perform a first similarity calculation on the forecast model value to be retrieved in the data information to be accessed and the corresponding element in each index information in the preprocessed meteorological and oceanographic database to obtain the first similarity value of each index information; S312, the start time and forecast validity of the data to be retrieved in the data to be accessed are compared with the corresponding elements in each index information of the preprocessed meteorological and oceanographic database to calculate the second similarity, and the second similarity value of each index information is obtained. S313, For the pressure layer, spatial longitude range, and spatial latitude range to be retrieved in the data information to be accessed, and the corresponding elements in each index information in the preprocessed meteorological and oceanographic database, a third similarity calculation is performed to obtain the third similarity value of each index information. S314, using the first similarity value, second similarity value and third similarity value of each index information as vector elements, construct the total similarity vector corresponding to the index information.

7. The access control method for a meteorological and oceanographic database as described in claim 6, characterized in that, The first similarity calculation includes: The forecast pattern value to be retrieved in the data information to be accessed is represented as a first vector; The forecast model values ​​in the index information of the preprocessed meteorological and oceanographic database are represented as a second vector; Subtract the first vector from the second vector to obtain the difference vector; Perform an EEMD transformation on the difference vector to obtain the transformed vector; A first fusion difference calculation is performed on the transformed vector, the first vector, and the second vector to obtain a first similarity value of the index information; The expression for calculating the first fusion difference is: ， in, The first similarity value is given by N, where N is the length of the transformed vector. and These are the i-th elements of the first vector and the second vector, respectively. Let be the i-th element of the transformation vector.

8. An access control device for meteorological and oceanographic data, characterized in that, The device includes: Memory containing executable program code; A processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the access control method for the meteorological and oceanographic database as described in any one of claims 1 to 7.

9. A computer-storable medium, characterized in that, The computer storage medium stores computer instructions, which, when invoked by the computer, are used to execute the access control method for the meteorological and oceanographic database as described in any one of claims 1 to 7.

10. An information data processing terminal, characterized in that, The information data processing terminal is used to implement the access control method for the meteorological and oceanographic database as described in any one of claims 1 to 7.

Images