Multi-source heterogeneous data processing method, device and system

Abstract

The present invention provides a multi-source heterogeneous data processing method, device and system, which relates to the field of data processing technology, including: monitoring multiple data directories, and creating a data scanning task when a change is detected in a multi-source heterogeneous data file stored in the data directory; using the nodes included in the data scanning cluster, executing the data scanning task for the target data directory, and obtaining the multi-source heterogeneous data file that has changed in the target data directory; using the nodes included in the multi-level data processing cluster, based on a preset general configuration information table, performing standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file to obtain a data processing result file. The present invention can realize the unified processing of multi-source heterogeneous data, not only significantly improve the processing efficiency of multi-source heterogeneous data, but also effectively alleviate the problems of task congestion, data loss, service delay, etc. in the multi-source heterogeneous data processing process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multi-source heterogeneous data processing method, device and system. Background Art

[0002] The typical characteristics of multi-source heterogeneous data are diversified sources, non-uniform formats, and huge data volumes. Related technologies mostly use timed scheduling, folder monitoring, and other methods to process multi-source heterogeneous data. However, when faced with large-scale multi-source heterogeneous data, it is easy to cause problems such as task congestion, data loss, and service delays. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide a multi-source heterogeneous data processing method, device and system, which can realize the unified processing of multi-source heterogeneous data, not only significantly improve the processing efficiency of multi-source heterogeneous data, but also effectively alleviate the problems of task congestion, data loss, service delay, etc. in the process of multi-source heterogeneous data processing.

[0004] In a first aspect, the present invention provides a multi-source heterogeneous data processing method, which is applied to a system, wherein a data scanning cluster and a multi-level data processing cluster are configured in the system, and the method comprises:

[0005] Monitor multiple data directories, and when changes are detected in the multi-source heterogeneous data files stored in the data directories, create data scanning tasks for the changed target data directories, with the target data directories corresponding to the data scanning tasks one by one;

[0006] Using the nodes included in the data scanning cluster, the data scanning task is performed on the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory;

[0007] By utilizing the nodes contained in the multi-level data processing cluster and based on a preset general configuration information table, the fields corresponding to each data dimension in the multi-source heterogeneous data files are standardized and converted to obtain the data processing result files corresponding to the multi-source heterogeneous data files.

[0008] In one implementation, using nodes included in a data scanning cluster to perform a data scanning task on a target data directory to obtain multi-source heterogeneous data files that have changed in the target data directory includes:

[0009] Monitor the CPU utilization, memory usage, and network bandwidth of each node in the data scanning cluster to determine the load of each node in the data scanning cluster;

[0010] Determine the first call priority of the data scanning task on each node in the data scanning cluster according to the weight of the data scanning task, the time interval of the last execution of the data scanning task for the target data directory, and the load of each node in the data scanning cluster;

[0011] Allocate a scanning node for the data scanning task according to the load corresponding to each node in the data scanning cluster and the first call priority of the data scanning task on each node in the data scanning cluster;

[0012] The scanning node is used to perform data scanning tasks on the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory.

[0013] In one embodiment, before using the nodes included in the multi-level data processing cluster to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file based on a preset general configuration information table to obtain a data processing result file corresponding to the multi-source heterogeneous data file, the method further includes:

[0014] Perform integrity checks on multi-source heterogeneous data files;

[0015] If the multi-source heterogeneous data file passes the integrity check, then the file features of the multi-source heterogeneous data file are extracted;

[0016] Through a machine learning model, a target preprocessing step corresponding to a multi-source heterogeneous data file is determined from a plurality of pre-configured preprocessing steps based on file features, so that the multi-source heterogeneous data file is preprocessed using the target preprocessing step.

[0017] In one embodiment, a target preprocessing step corresponding to a multi-source heterogeneous data file is determined from a plurality of pre-configured preprocessing steps based on file features through a machine learning model, including:

[0018] Based on the historical execution results of multiple pre-configured pre-processing steps, dynamically adjust the weight corresponding to each pre-processing step;

[0019] Through the machine learning model, based on the file features, the historical execution results and weights corresponding to each preprocessing step, the processing time consumed by each preprocessing step to preprocess multi-source heterogeneous data files is predicted;

[0020] Determine the second scheduling priority corresponding to each preprocessing step according to the weight and processing time corresponding to each preprocessing step;

[0021] According to the second scheduling priority, at least one target preprocessing step corresponding to the multi-source heterogeneous data file is determined from each preprocessing step.

[0022] In one embodiment, using the nodes included in the multi-level data processing cluster, based on a preset general configuration information table, the fields corresponding to each data dimension in the multi-source heterogeneous data file are standardized and converted to obtain a data processing result file corresponding to the multi-source heterogeneous data file, including:

[0023] Use the Kafka message queue to push data replication and decompression tasks for multi-source heterogeneous data files to the first-level data processing cluster, and use the nodes contained in the first-level data processing cluster to perform data replication and decompression tasks for multi-source heterogeneous data files to obtain file decompression results;

[0024] Continue to use the Kafka message queue to push data processing and parsing tasks to the second-level data processing cluster for the file decompression results, and use the nodes contained in the second-level data processing cluster to perform data processing and parsing tasks on the file decompression results. The data processing and parsing tasks are used to perform standardized conversions on the fields corresponding to each data dimension in the file decompression results based on the preset general configuration information table to obtain the data processing result file.

[0025] In one embodiment, using nodes included in the first-level data processing cluster, data replication and decompression tasks are performed for multi-source heterogeneous data files, including:

[0026] Determine the third scheduling priority of each node in the first-level data processing cluster for the multi-source heterogeneous data file based on the current resource occupancy rate and current processing capacity corresponding to each node in the first-level data processing cluster;

[0027] Allocating processing nodes to multi-source heterogeneous data files according to a third scheduling priority;

[0028] Use processing nodes to perform data replication and decompression tasks for multi-source heterogeneous data files.

[0029] In one implementation, based on a preset general configuration information table, a field corresponding to each data dimension in the file decompression result is standardized and converted to obtain a data processing result file, including:

[0030] Based on the mapping relationship between the data structure described in the preset general configuration information table and the fields corresponding to multiple data dimensions, the fields corresponding to each data dimension in the file decompression result are standardized and converted to obtain a data processing result file.

[0031] In one embodiment, the method further comprises:

[0032] If a service request for a data processing result file is received, target data information is extracted from the data processing result file based on the query conditions carried in the service request, and the target data information is rendered and displayed;

[0033] The service request includes one or more of a rendering service request, a single-point timing diagram service request, and a profile diagram service request.

[0034] In a second aspect, the present invention further provides a multi-source heterogeneous data processing device, which is applied to a system, wherein a data scanning cluster and a multi-level data processing cluster are configured in the system, and the device comprises:

[0035] The scanning task creation module is used to monitor multiple data directories and create data scanning tasks for the changed target data directories when changes are detected in the multi-source heterogeneous data files stored in the data directories. The target data directories correspond to the data scanning tasks one by one.

[0036] A scanning task execution module is used to use the nodes included in the data scanning cluster to execute the data scanning task for the target data directory and obtain the multi-source heterogeneous data files that have changed in the target data directory;

[0037] The processing task execution module is used to utilize the nodes contained in the multi-level data processing cluster to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data files based on the preset general configuration information table, and obtain the data processing result files corresponding to the multi-source heterogeneous data files.

[0038] In a third aspect, the present invention further provides a multi-source heterogeneous data processing system, comprising a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement any one of the methods provided in the first aspect.

[0039] A multi-source heterogeneous data processing method, device and system provided by the present invention monitor multiple data directories, and when it is monitored that the multi-source heterogeneous data files stored in the data directory have changed, a data scanning task is created for the changed target data directory, and the target data directory corresponds to the data scanning task one by one; then, the nodes included in the data scanning cluster are used to execute the data scanning task for the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory; finally, the nodes included in the multi-level data processing cluster are used to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file based on a preset general configuration information table to obtain the data processing result file corresponding to the multi-source heterogeneous data file. The above method introduces a universal configuration information table to perform standardized conversion on multi-source heterogeneous data from different data sources and with non-uniform formats, thereby achieving unified processing of multi-source heterogeneous data, helping to improve the processing efficiency of multi-source heterogeneous data and reducing the complexity of multi-format data processing. In addition, the use of data scanning clusters and multi-level data processing clusters can efficiently cope with the challenges of processing massive amounts of multi-source heterogeneous data, greatly improving the processing efficiency of multi-source heterogeneous data, thereby effectively alleviating problems such as task congestion, data loss, and service delays in the process of multi-source heterogeneous data processing.

[0040] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention are realized and obtained by the structures particularly pointed out in the description, claims and drawings.

[0041] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0043] Figure 1 A flowchart of a multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0044] Figure 2 A task scheduling diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0045] Figure 3A technical circuit flow chart of a multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0046] Figure 4 A functional radar diagram of a multi-source heterogeneous data processing method provided by an embodiment of the present invention;

[0047] Figure 5 A schematic diagram of the structure of a multi-source heterogeneous data processing device provided by an embodiment of the present invention;

[0048] Figure 6 A schematic diagram of the structure of a multi-source heterogeneous data processing system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described in combination with the embodiments below. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] At present, the existing multi-source heterogeneous data processing technical means are prone to cause problems such as task congestion, data loss, and service delays when faced with large-scale multi-source heterogeneous data; in addition, the front-end rendering and back-end data services of the existing system lack standardization, resulting in inconsistent data processing and display results between different data sources, making it difficult to efficiently meet the needs of real-time query and data visualization.

[0051] Taking meteorological and ocean grid data (referred to as meteorological and ocean grid data) as an example, meteorological and ocean monitoring is a core component of modern environmental science. The temporal and spatial variation patterns of meteorological and ocean grid data are of great significance for scientifically predicting weather and ocean conditions, formulating disaster prevention and mitigation measures, and promoting environmental protection. However, with the development of satellite, ground observation and other technologies, the speed of generating massive, multi-format, and multi-source data has accelerated, which has put forward higher requirements for data processing efficiency, data compatibility, and real-time response capabilities. However, the typical characteristics of meteorological and ocean grid data are diversified sources, non-uniform formats (such as Netcdf, Grib, HDF, etc.), huge data volume, relatively complex data processing, and low data processing efficiency. At the same time, when providing data services, it is also necessary to consider matching different data formats with front-end rendering methods and data transmission efficiency.

[0052] Specifically, the sources of meteorological and ocean grid data mainly include satellite observations, radar detection, numerical simulation and other channels, with various data formats, typical formats include NetCDF (Network Common Data Format), GRIB (General Rules File), HDF (Hierarchical Data Format), etc. The data structure and storage method of different data sources are different, but most data contain similar five-dimensional structures: element dimension (such as temperature, wind speed, air pressure, etc.), time dimension (observation time or forecast time), altitude dimension, longitude dimension and latitude dimension. These data have the following typical characteristics:

[0053] Feature 1: Heterogeneity: The data format is not unified, and there are large differences in physical elements, temporal resolution, spatial resolution, etc. between different data sources, which makes the analysis process complicated and difficult to achieve standardized processing.

[0054] Feature 2: Massiveness: With the advancement of data collection technology, the scale of meteorological and oceanographic data has grown exponentially. The volume of a single meteorological data file may reach tens of GB, containing hundreds of millions of data points. Traditional processing methods are often unable to cope with such massive amounts of data, which can easily lead to low processing efficiency, and even task blockage, data loss, etc.

[0055] Feature 3: Real-time requirements: In applications such as weather forecasting, disaster warning, and ocean condition monitoring, fast response and real-time processing are crucial. However, traditional data processing methods have a slow response speed and cannot meet high-frequency query and visualization requirements.

[0056] Based on the above characteristics, the existing multi-source heterogeneous data processing technologies have the following key problems:

[0057] Problem 1: Data heterogeneity: The main sources of current meteorological and ocean grid data include satellite observations, numerical forecast models, radar detection, etc., and these data are usually stored in different formats, such as NetCDF, GRIB, HDF, etc. This difference in format brings significant difficulties to the parsing and processing of data. The data format, spatiotemporal resolution, and physical element description of each data source are different. Parsing them requires the use of different tools and methods, resulting in complex workflows that are difficult to unify.

[0058] Problem 2: Efficient processing of large-scale data: With the development of observation technology, the scale of meteorological and ocean grid data has grown exponentially. A single weather forecast or ocean monitoring file may contain billions of data points, and the file size may reach tens of GB or even larger. In this context, traditional processing methods often suffer from problems such as low processing speed, task blocking, and uneven allocation of system resources when dealing with the increase in data volume. In addition, many data sources are updated frequently, requiring the system to have real-time data acquisition and processing capabilities, otherwise it will cause data lag and fail to meet application requirements.

[0059] Problem 3: Standardized services and fast response: In practical applications, the query and visualization requirements for meteorological and oceanographic grid data are very diverse. Users hope to quickly obtain the required information from large-scale data sets and analyze them through visualization tools. Due to the large amount of data and complex format, traditional query and rendering methods often cannot meet users' requirements for real-time response when facing a large number of concurrent requests, resulting in slow system response and affecting user experience.

[0060] Based on this, the implementation of the present invention provides a multi-source heterogeneous data processing method, device and system, which can realize the unified processing of multi-source heterogeneous data, not only significantly improving the processing efficiency of multi-source heterogeneous data, but also effectively alleviating the problems of task congestion, data loss, service delay, etc. in the process of multi-source heterogeneous data processing.

[0061] To facilitate understanding of this embodiment, a multi-source heterogeneous data processing method disclosed in an embodiment of the present invention is first introduced in detail. The method is applied to a system, and a data scanning cluster and a multi-level data processing cluster are configured in the system, and the data scanning cluster and the multi-level data processing cluster both contain multiple nodes. Among them, the data scanning cluster is used to implement data scanning tasks; the data processing task is divided into multiple data processing subtasks, and multiple data processing subtasks of the same type will be executed by multiple nodes in a data processing cluster, that is, a type of data processing subtask corresponds to a first-level data processing cluster, and the dependency relationship between different levels of data processing clusters is consistent with the dependency relationship between different types of data processing subtasks.

[0062] See also Figure 1 The flowchart of a multi-source heterogeneous data processing method shown in FIG. 1 mainly includes the following steps S102 to S106:

[0063] Step S102, multiple data directories are monitored, and when changes are detected in the multi-source heterogeneous data files stored in the data directories, a data scanning task is created for the changed target data directory, and the target data directory corresponds to the data scanning task one by one.

[0064] Take the case of multi-source heterogeneous data files, which are meteorological and ocean grid data, as an example. Figure 2 The task scheduling diagram of a multi-source heterogeneous data processing method shown is as follows:

[0065] The types of meteorological and ocean grid data mainly include two categories: numerical forecast data and fusion analysis data. Numerical forecast data includes observation time and forecast time in the time dimension, while fusion analysis data generally only includes observation time in the time dimension.

[0066] Multi-source refers to the diverse sources of data, including CMACast broadcast data, CCTV (China Central Television) broadcast data, Meteorological Bureau dedicated line data, NMEFC (National Marine Environmental Forecast Center) data, National Satellite Ocean Application Center data, etc.

[0067] Heterogeneous: refers to the variety of data formats. Typical formats include NC (Network Common Data Format), grib1 (GRIdded Binary 1), grib2 (GRIdded Binary 2), HDF (Hierarchical Data File), TIF (Tagged Image File Format), etc.

[0068] Five dimensions: refers to element dimension, time dimension, altitude, longitude and latitude, etc.;

[0069] Elements: refers to specific physical quantities. The more well-known ones include temperature, air pressure, humidity, visibility, wind speed and direction, ocean currents and waves, etc. Generally, a single meteorological and oceanographic grid data will have dozens or hundreds of elements.

[0070] In one example, by comparing the status of meteorological and oceanographic grid data (such as hash value, timestamp, file size, etc.), it is possible to automatically detect whether the meteorological and oceanographic grid data stored in the data directory has been added or updated, and the data directory with the new or updated data is used as the target data directory, and a data scanning task is created for each target data directory.

[0071] Step S104, using the nodes included in the data scanning cluster to perform a data scanning task on the target data directory, and obtaining the multi-source heterogeneous data files that have changed in the target data directory.

[0072] Taking meteorological ocean grid data as an example, meteorological ocean grid data is generally stored in multiple directories in the form of files according to type. The data push or update time is not fixed. The number of data files in a certain directory is also different, ranging from dozens to hundreds of thousands. In multi-directory and large data volume scenarios, traditional timed scheduling or folder monitoring methods are very likely to cause job blocking, data loss, machine crashes, etc. Therefore, the embodiment of the present invention can meet the needs of obtaining multi-source heterogeneous meteorological ocean grid data through distributed task scheduling technology combined with a cache database. It can include the following process: configuring job tasks according to data sources, including task names, routing strategies, execution time and other information, implementing job scheduling and execution through a distributed task scheduling system, realizing multi-node scheduling and multi-node task execution, and adapting to multi-directory and large data volume scanning scenarios; and after obtaining the changed meteorological ocean grid data (such as newly added meteorological ocean grid data), performing the next step of data processing.

[0073] For further information, please see Figure 2 ,After acquiring the meteorological and ocean grid data, it can be determined whether the meteorological and ocean grid data has been processed. If so, the task is terminated, otherwise, subsequent data processing tasks are performed.

[0074] Step S106, utilizing the nodes included in the multi-level data processing cluster, based on a preset general configuration information table, performs standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file to obtain a data processing result file corresponding to the multi-source heterogeneous data file.

[0075] Taking the meteorological and ocean grid data as an example, the size of a single file of meteorological and ocean grid data ranges from MB to GB. If the longitude and latitude points are used for statistics, a single file may contain billions of feature values. After unifying the data analysis process, the data processing result files should also be unified as much as possible. The data processing tasks of meteorological and ocean grid data are split into tasks, and loose coupling and cluster processing of tasks are achieved through message queue technology, generating unified DAT data sets, grayscale images, database metadata and other result files.

[0076] Please continue to see Figure 2 ,The data processing tasks are mainly divided into two types of subtasks: Subtask 1 mainly performs data preprocessing, including data copying and decompression, copying the original meteorological and ocean grid data (referred to as source data) before ,operation, to avoid modification or accidental deletion of the source data, and ,decompression for the next task processing; Subtask 2 mainly obtains the type information, element information and ,spatial information of meteorological and ocean grid data based on the constructed ,general configuration information table, and parses the meteorological and ocean grid data through a unified code and generates ,data processing result files.

[0077] On this basis, Kafka message queue technology can be used to achieve asynchrony and decoupling between tasks: based on the division of data processing tasks, two topics, subtask 1 and subtask 2, are defined. When a new meteorological and ocean grid data is generated, a data copy and decompression task will be pushed to the subtask 1 topic. Consumers who subscribe to the subtask 1 topic (that is, the nodes in the first-level data processing cluster) will perform specific job tasks after receiving the task. After completion, a data processing and parsing task will be pushed to the subtask 2 topic. Consumers who subscribe to the subtask 2 topic (that is, the nodes in the second-level data processing cluster) will perform specific job tasks after receiving the task, thus realizing the vertical splitting and horizontal expansion of multi-source heterogeneous meteorological and ocean grid data processing.

[0078] Please continue to see Figure 2 , the data processed by the first-level data processing cluster and the second-level data processing cluster can be cached in Redis.

[0079] Among them, the aforementioned general configuration information table is used to describe the mapping relationship between different data structures and fields corresponding to multiple data dimensions. Continuing to take meteorological and oceanic grid data as an example, meteorological and oceanic grid data has the characteristics of multi-source and heterogeneity, and its internal specific data organization form is also complex and diverse: in the element dimension, the element lists of different types of data are almost different; in the time dimension, there are forecast data and fusion analysis data (determined by whether the data has a forecast time); in the altitude level, there is a distinction between altitude or no altitude, isobaric surface altitude and potential meter altitude; in the latitude and longitude dimension, there is a distinction between regional scope and resolution, the regional scope mainly includes global areas and local areas, and the resolution mainly includes equal longitude and latitude and equal distance. Constructing a general configuration information table based on the above information may include the following processes: reading multiple elements in meteorological and oceanic grid data: generally there may be dozens or hundreds of elements; reading time information: generally including observation time, forecast duration, forecast frequency, etc.; reading altitude layer information: generally refers to the values ​​of multiple altitudes from the ground to high altitude; reading longitude information: generally including starting longitude, ending longitude, resolution and quantity; reading latitude information: generally including starting latitude, ending latitude, resolution and quantity; based on the general configuration information table, all meteorological and oceanic grid data can be parsed through a set of codes, unifying the parsing process of multi-source heterogeneous meteorological and oceanic grid data to obtain the data processing result file.

[0080] The aforementioned data processing result files include unified data set result files, grayscale image result files, and database metadata result files. Specifically:

[0081] (a) Unified dataset result file (also called DAT dataset) refers to a DAT file that stores data in binary format. The file is named with element + time. The binary data represents three-dimensional data constructed by altitude layer, longitude, and latitude. The altitude layer is from low to high, the longitude is from small to large, and the latitude is from small to large, so as to achieve standard unification. When querying data for subsequent services, the source data will be read with the unified DAT result set file to avoid reading data from multi-source heterogeneous data files and ensure the unification of service codes.

[0082] (b) The grayscale image result file is mainly used for front-end rendering and display. It is a result file formed by normalizing the two-dimensional array queried for one element, one time, and one height layer, and then placing it into the grayscale image channel. It greatly improves data compression, front-end and back-end transmission efficiency, and rendering efficiency.

[0083] (c) Database metadata result file refers to the information describing the unified dataset result file and the grayscale image result file, mainly including the file name, size, storage path and creation time, etc., which is used to support subsequent service construction.

[0084] Furthermore, the data processing result file can be used to provide standard unified rendering services, single-point time series diagram services and profile diagram services. The rendering service queries and returns the grayscale image address based on the data type, feature type, observation time, forecast time, and altitude level; the single-point time series diagram service queries and returns JSON values ​​based on the data type, feature type, observation time, forecast time, altitude level, longitude, and latitude; the profile diagram service queries and returns two-dimensional array data based on the data type, feature type, observation time, forecast time, and longitude and latitude point set.

[0085] In summary, the multi-source heterogeneous data processing method provided by the embodiment of the present invention obtains multi-source heterogeneous meteorological and oceanic grid data through distributed task scheduling technology combined with a cache database; constructs a unified general configuration information table based on the data type, element information and spatiotemporal information of the meteorological and oceanic grid data; splits the data processing tasks of the meteorological and oceanic grid data, realizes loose coupling and cluster processing of tasks through message queue technology, and generates unified DAT data sets, grayscale images, database metadata and other result files; uses the result files to provide standard and unified rendering services, single-point time series diagram services and profile diagram services, thereby solving the technical problems of complex processing of multi-source heterogeneous meteorological and oceanic five-dimensional grid data, low efficiency and inconsistent services.

[0086] Therefore, the embodiments of the present invention adopt distributed task scheduling technology, cache database, task splitting and message queue technologies for multi-source heterogeneous meteorological and ocean grid data, and construct efficient data processing methods and standard unified service methods based on data type, element information and spatiotemporal information. It can realize unified acquisition, analysis, storage and visualization services for large-scale meteorological and ocean data including multiple elements such as temperature, air pressure, wind speed, ocean currents, etc., which is particularly suitable for complex multi-dimensional data processing and real-time display needs in the fields of weather forecasting, ocean monitoring and environmental change research.

[0087] For ease of understanding, the present invention provides a specific implementation of a multi-source heterogeneous data processing method, see Figure 3 The technical circuit flow chart of a multi-source heterogeneous data processing method shown in FIG. 1 includes:

[0088] (1) Data acquisition and preprocessing:

[0089] In one example, the process of data acquisition (1.1) is as follows: The embodiment of the present invention efficiently acquires and preprocesses heterogeneous meteorological and oceanographic data from multiple sources through a distributed task scheduling system. The system dynamically scans multiple data directories to ensure real-time monitoring and processing of data from different data sources. The data volume and update frequency of each data directory are not fixed. The system automatically detects newly added or updated data files by comparing file status (such as hash value, timestamp, file size, etc.).

[0090] Specifically, the system uses a dynamic load balancing mechanism to allocate tasks to ensure that each node is load balanced when executing tasks. The system will allocate appropriate data scanning tasks based on the current processing load of each node (such as CPU, memory, and network bandwidth usage). At the same time, the execution order is adjusted according to the real-time changes in node load, so that data directories that are updated frequently can be processed first to prevent data processing delays. For details, please refer to the following steps:

[0091] (1.1.1) Monitor the CPU utilization, memory usage, and network bandwidth of each node in the data scanning cluster to determine the load situation of each node in the data scanning cluster.

[0092] In one example, each node periodically reports its own load status (CPU utilization, memory usage, network bandwidth, etc.), and the scheduler collects this information to update the task allocation strategy in real time. The node load monitoring formula is:

[0093] ;

[0094] in, For the The load situation corresponding to each node, , , They are the weight coefficients of CPU, memory, and bandwidth in load calculation. For the The CPU utilization of each node, For the The memory usage of each node, For the The network bandwidth of each node is adjusted according to system requirements.

[0095] (1.1.2) Determine the first call priority of the data scanning task on each node in the data scanning cluster based on the weight of the data scanning task, the time interval since the last execution of the data scanning task on the target data directory, and the load of each node in the data scanning cluster. The weight of the data scanning task is determined by factors such as the file size and update time of the multi-source heterogeneous data files stored in the data directory.

[0096] In one example, the system dynamically allocates tasks based on the real-time load status of each node and the complexity of the task. The load balancing mechanism determines the first call priority of the task through the following formula:

[0097] ;

[0098] in, Represents a data scanning task In the The first call priority on a node, It is a data scanning task The weight of Is the last time a data scan task was performed on the target data directory The time interval It is The load situation corresponding to each node.

[0099] (1.1.3) Allocate a scanning node to the data scanning task according to the load of each node in the data scanning cluster and the first calling priority of the data scanning task on each node in the data scanning cluster.

[0100] In one example, the system dynamically assigns scanning nodes to data scanning tasks based on the node load and the first call priority of the task. Task scheduling not only considers the current node load, but also predicts the task completion time through historical processing performance. Tasks with high priority will be assigned to nodes with lower current load to maximize the overall throughput of the system.

[0101] (1.1.4) Use the scanning node to perform data scanning tasks on the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory.

[0102] The embodiment of the present invention adopts a mechanism of distributed task scheduling combined with dynamic load balancing through distributed task scheduling technology. Through a distributed architecture, it is possible to efficiently schedule and process large-scale heterogeneous meteorological and ocean data from different data sources. When facing multi-source data with high frequency updates, the traditional timed scheduling method usually allocates tasks statically, and it is difficult to dynamically adjust according to real-time resource load and data changes. It is not only easy to cause task blocking or system crash, but also there is a situation where some nodes are overloaded and other node resources are not fully utilized. The embodiment of the present invention uses a dynamic load balancing mechanism to intelligently allocate tasks according to the actual load status of each node, ensuring that the system maintains efficient and stable operation when processing tasks on a large scale concurrently. Distributed task scheduling technology is not only suitable for the efficient allocation of multi-source data, but also can realize dynamic task scheduling by real-time monitoring of node load, ensuring that the system can flexibly adjust task allocation according to actual needs, so that the system has a high degree of adaptability when dealing with sudden surges in data volume, reducing task accumulation and system bottlenecks.

[0103] In one example, the process of data preprocessing (1.2) is as follows:

[0104] (1.2.1) Perform integrity checks on multi-source heterogeneous data files.

[0105] In one example, before processing a newly added or updated multi-source heterogeneous data file, the system will perform an integrity check on the multi-source heterogeneous data file to ensure that the multi-source heterogeneous data file is not damaged and meets the format requirements. The system will identify and process incomplete multi-source heterogeneous data files or multi-source heterogeneous data files that do not meet the format requirements, mark these multi-source heterogeneous data files and issue an alarm. During the integrity check process, the system will also generate detailed log records to help operation and maintenance personnel quickly locate and resolve potential problems. For damaged multi-source heterogeneous data files, the system has automatic rollback and recovery functions to ensure the stability of the data process when erroneous multi-source heterogeneous data files appear.

[0106] (1.2.2) If the multi-source heterogeneous data file passes the integrity check, the file features of the multi-source heterogeneous data file are extracted. The file features include file size, file format, data distribution, data type, field complexity, etc.

[0107] (1.2.3) Determine the target preprocessing step corresponding to the multi-source heterogeneous data file from the pre-configured multiple preprocessing steps based on the file features through the machine learning model, so as to preprocess the multi-source heterogeneous data file using the target preprocessing step. See (I) to (IV) below for details:

[0108] (I) Based on the historical execution results of multiple pre-configured pre-processing steps, dynamically adjust the weight corresponding to each pre-processing step. The pre-processing step may be data cleaning, format conversion, file decompression, etc., which may be specifically configured based on actual needs, and the embodiments of the present invention are not limited to this.

[0109] In one example, an initial weight can be assigned to each preprocessing step in advance, and the weight can be dynamically updated according to the historical execution results of the preprocessing step. The historical execution results can include indicators such as the execution time, execution effect, and execution failure rate of the preprocessing step, which are maintained in a historical record library, and the weight of the preprocessing step is continuously optimized using the various indicators recorded in the historical record library.

[0110] Specifically, the weight of the preprocessing step can be adjusted according to the following formula:

[0111] ;

[0112] in, Represents the preprocessing step The weight of is the learning rate, which is used to control the magnitude of weight adjustment. is a loss function based on processing time and quality, which represents the overall efficiency of preprocessing. The update of weights will increase the weight of preprocessing steps with short execution time and high efficiency, while reducing the weight of preprocessing steps with long execution time and high failure rate.

[0113] (II) Using a machine learning model, based on file features, historical execution results and weights corresponding to each preprocessing step, the processing time consumed by each preprocessing step to preprocess multi-source heterogeneous data files is predicted. The machine learning model (also referred to as an AI model) may be a linear regression, decision tree or other model. Specifically, it includes:

[0114] Model input: Input variables include file features, historical execution results of preprocessing steps, and current weights. Features are combined into multi-dimensional inputs and input into the machine learning model for processing.

[0115] Model output: The model output is the processing time of each preprocessing step , the processing time of the preprocessing step can be compared and calibrated with the execution time recorded in the historical record library to ensure the accuracy of the prediction. The model output format is as follows: .

[0116] Model optimization: The model is continuously optimized and fine-tuned through feedback mechanisms (such as the deviation between actual execution time and predicted processing time) to improve prediction accuracy.

[0117] (III) Determine a second scheduling priority corresponding to each preprocessing step according to the weight and processing time corresponding to each preprocessing step.

[0118] Based on the estimated processing time and weight , the system selects the optimal preprocessing step. The specific strategy is as follows: Calculate the second scheduling priority corresponding to each preprocessing step according to the following formula :

[0119] .

[0120] (IV) Determine at least one target preprocessing step corresponding to the multi-source heterogeneous data file from each preprocessing step according to the second scheduling priority.

[0121] In one example, the second scheduling priority is preferred The preprocessing step with the largest value ensures that the preprocessing step with short time consumption and high efficiency is executed first.

[0122] Further, according to the second scheduling priority The system dynamically selects one or more preprocessing steps as the target preprocessing steps for multi-source heterogeneous data files; when encountering multiple preprocessing steps, the second scheduling priority When the number of data is the same, the system will select the final preprocessing step based on the historical success rate and task urgency. For example, for multi-source heterogeneous data files with complex formats, the system will prioritize file parsing to determine the file structure; for multi-source heterogeneous data files with large data volumes, the system will enable parallel decompression and preprocessing processes to reduce overall processing time.

[0123] In an application example, take processing a heterogeneous data file as an example:

[0124] File feature extraction includes: file size 500MB; data type is multidimensional time series data; file format is NetCDF.

[0125] Model prediction includes: the system inputs the above features into the AI ​​model to obtain the predicted processing time of each preprocessing step, which includes data cleaning, format conversion, and outlier detection. For example, data cleaning: 5 seconds; format conversion: 12 seconds; outlier detection: 8 seconds.

[0126] Priority calculation, including: Assume that the weights of the preprocessing steps are: Data cleaning = 0.8; format conversion: =0.6; Outlier detection = 0.7.

[0127] The second scheduling priority for the system calculation preprocessing step: data cleaning = 0.8 / 5 = 0.16; format conversion: = 0.6 / 12 = 0.05; Outlier Detection = 0.7 / 8 = 0.0875.

[0128] Prioritization: > > .

[0129] Preprocessing step selection and execution: The system first performs data cleaning, then performs outlier detection, and finally performs format conversion.

[0130] In terms of data processing performance, the present invention introduces an AI intelligent preprocessing module, which intelligently predicts and optimizes the preprocessing steps of data processing through machine learning algorithms. Traditional data processing systems are usually based on fixed rules, cannot flexibly respond to the characteristics and complexity of different data files, and lack the ability to dynamically adjust data characteristics and system resource usage. Through the intelligent preprocessing module, the present invention can automatically adjust the processing path according to the characteristics of the data file, reduce redundant calculations, and significantly improve the efficiency of data processing; and the adaptive optimization mechanism enables the system to automatically adjust the processing flow as the data changes, realize continuous optimization of the preprocessing steps, continuously improve the prediction accuracy, and ensure the flexibility and efficiency of the system. Especially when processing large-scale data sets and complex file structures, intelligent preprocessing significantly improves the overall processing performance of the system.

[0131] Furthermore, in the preprocessing stage, the system also introduces a cache mechanism to cache the file information (such as hash value, file size, etc.) of processed multi-source heterogeneous data files to reduce the overhead of repeated scanning. For common update directories, the system can quickly extract the metadata of data files, greatly improving the efficiency of data acquisition.

[0132] (2) Data analysis and general configuration information table:

[0133] The universal configuration information table proposed in the embodiment of the present invention is designed for the format differences of multi-source heterogeneous meteorological and oceanographic data. Data files of different formats usually contain similar physical elements and spatiotemporal information, but their data structures, file formats and field naming methods are different. In order to uniformly process data from different sources and formats, the embodiment of the present invention introduces a universal configuration information table, which realizes standardized parsing of cross-format data by mapping data fields of different formats.

[0134] Specifically, the configuration information table contains the mapping relationship between data formats and fields from different data sources. Through this table, the system can parse data files of different formats into a standardized five-dimensional data structure (element, time, longitude, latitude, altitude) to achieve unified data processing.

[0135] In one implementation, the general configuration information table uses key-value pairs to represent the mapping relationship between different data formats and five-dimensional elements. The core mapping relationship is represented by the following formula:

[0136] ;

[0137] in, The number of the field or data element, corresponding to different physical quantities, such as temperature, humidity, wind speed and other meteorological and oceanographic elements; Indicates the specific value or attribute of the field, such as time point, observation height or geographical location; Represents specific dimensions in multidimensional space, usually including time dimension, altitude dimension, longitude and latitude dimension, etc.; Represents data format Fields in The corresponding mapping relationship in the five-dimensional structure; is a mapping of the time dimension, is a mapping of spatial dimensions (longitude and latitude), is a mapping of the height dimension.

[0138] Through the universal configuration information table, the system can easily cope with the differences in different data formats, realize standardized data processing across formats, and greatly simplify the parsing process of multi-source heterogeneous data.

[0139] Furthermore, the general configuration information table provided in the embodiment of the present invention is highly extensible. With the introduction of new data formats, the system can support new data formats by simply adding corresponding data field mapping rules in the configuration information table, without the need for large-scale modifications to the parsing logic, thereby reducing development and maintenance costs. The use of the general configuration information table enables the system to achieve seamless compatibility and expansion when dealing with multi-source data, and adapt to more formats of data sources that may appear in the future.

[0140] (3) Task splitting and message queue:

[0141] In one example, in order to improve the efficiency of data processing, the system adopts an asynchronous processing mode combining task splitting and message queues. The data processing task is subdivided into multiple independent subtasks, each of which can be processed in parallel on different nodes. The introduction of the Kafka message queue ensures that tasks can be loosely coupled and asynchronously scheduled, improves processing efficiency, and avoids blocking between tasks. Exemplarily, the data processing task is split into two subtasks: a data replication and decompression task and a data processing and parsing task. The data processing cluster that performs the data replication and decompression task is recorded as the first-level data processing cluster, and the data processing cluster that performs the data processing and parsing task is recorded as the second-level data processing cluster.

[0142] Specifically, the process of task splitting and message queue is as follows:

[0143] (3.1) Using the Kafka message queue, push the data replication and decompression tasks for multi-source heterogeneous data files to the first-level data processing cluster, and use the nodes contained in the first-level data processing cluster to perform data replication and decompression tasks for multi-source heterogeneous data files to obtain the file decompression results;

[0144] (3.2) Continue to use the Kafka message queue to push data processing and parsing tasks for the file decompression results to the second-level data processing cluster, and use the nodes contained in the second-level data processing cluster to perform data processing and parsing tasks on the file decompression results. The data processing and parsing tasks are used to perform standardized conversions on the fields corresponding to each data dimension in the file decompression results based on the general configuration information table to obtain the data processing result file.

[0145] In the specific implementation, after each subtask is executed, the execution result of the subtask is seamlessly passed to the next task queue through the Kafka message queue, ensuring the independence between tasks and achieving the continuity and consistency of data processing. The message queue system supports large-scale concurrent task scheduling. After a subtask is completed, there is no need to wait for the result of the previous task, and the next task can be executed immediately. In this way, the system can process multiple different tasks at the same time and ensure the efficiency and continuity of the processing process.

[0146] The embodiment of the present invention adopts message queue and task splitting technology to ensure that the system can efficiently process tasks in parallel and avoid execution delays caused by task dependence. Through the asynchronous processing mechanism of Kafka message queue, the task is automatically split into multiple subtasks and distributed to different nodes for parallel execution, and the processing results of the task are seamlessly connected through the message queue. The embodiment of the present invention adopts message queue technology and task splitting mechanism with the following advantages: asynchronous processing and scheduling of tasks are realized through Kafka message queue, ensuring that large-scale data processing tasks can be executed in parallel, greatly improving the concurrent processing capability of the system, and being able to efficiently process large-scale data, especially in high concurrent request scenarios, the system shows better stability; the processing process between each task is loosely coupled, with high scalability and stability; this task scheduling and splitting technology not only greatly improves the throughput of the system, but also improves the continuity and stability of data processing, ensuring that the system can still maintain efficient operation in the face of multiple data sources and high concurrent scenarios.

[0147] For ease of understanding, the present invention further explains (3.1) and (3.2) above:

[0148] First, the multi-source heterogeneous data files need to be split in advance, so that the corresponding data copy and decompression tasks, data processing and parsing tasks can be created for the multiple files obtained by splitting. Specifically, task splitting and parallel execution: The system splits the multi-source heterogeneous data files by time period or physical elements. For example, for a multi-source heterogeneous data file containing multiple time periods, the system will split it into several time period tasks, and each time period corresponds to an independent subtask. Each subtask can be executed in parallel on different processing nodes, thereby speeding up the overall data processing. The granularity of task splitting can be dynamically adjusted according to the file size, update frequency, and complexity of physical elements.

[0149] The embodiment of the present invention also provides a task splitting logic: a large-scale data processing task is split into multiple independent subtasks, and each subtask is separated according to different data time periods or physical elements. The total time of task splitting is:

[0150] ;

[0151] in, Represents a subtask The execution time, is the total number of task splits. After the task is split, each subtask can be executed in parallel, thus shortening the execution time of the overall task.

[0152] Then, the asynchronous processing of the message queue is implemented. Taking the asynchronous processing of data copying and decompression tasks as an example, the third scheduling priority of each node in the first-level data processing cluster for multi-source heterogeneous data files can be determined based on the current resource occupancy rate and current processing capacity corresponding to each node in the first-level data processing cluster; processing nodes are allocated to multi-source heterogeneous data files according to the third scheduling priority; and processing nodes are used to perform data copying and decompression tasks for multi-source heterogeneous data files. It should be noted that the asynchronous processing of data processing and parsing tasks is consistent with the asynchronous processing of data copying and decompression tasks, and the embodiments of the present invention will not be repeated.

[0153] In a specific implementation, each subtask is scheduled through the Kafka message queue, and the execution order of the tasks does not depend on the completion of the previous task. After the subtask is completed, the processing result will be automatically passed to the next task queue to achieve asynchronous processing and avoid mutual dependence between tasks. The task scheduling algorithm in the message queue is:

[0154] ;

[0155] in, Represents a subtask Nodes in a data processing cluster The second scheduling priority on Is a node The current processing power of Is a node Current resource usage.

[0156] The asynchronous processing mechanism avoids the linear dependency of subtasks, improves the execution efficiency of tasks, and ensures the continuity and efficiency of processing.

[0157] Finally, the corresponding subtasks are executed according to the assigned nodes. The data copying and decompression tasks are to copy and decompress the split data files; the data processing and parsing tasks are to uniformly parse the multi-source heterogeneous data based on the general configuration information table constructed in (2) above.

[0158] The embodiment of the present invention also provides a specific implementation method for uniformly parsing multi-source heterogeneous data based on the general configuration information table constructed in the above (2), including: based on the mapping relationship between the data structure described by the preset general configuration information table and the fields corresponding to the multiple data dimensions, the fields corresponding to each data dimension in the file decompression result are standardized and converted to obtain a data processing result file.

[0159] In one implementation, first, according to the format type of each file, the system calls different parsing libraries (such as NetCDF4, PyGRIB, etc.) to parse the data file. During the parsing process, the system will extract physical elements (such as temperature, air pressure, wind speed, etc.), as well as time and space information (such as observation time, forecast time, longitude, latitude, altitude). Second, query the general configuration information table based on the metadata of the data file (file type, format) to obtain the corresponding field mapping relationship. Third, the system uses the mapping relationship to perform standardized conversion on each field in the file and map it to a standardized five-dimensional structure, thereby realizing the conversion of data in different formats into a unified structure, thereby ensuring that data from different sources can be uniformly converted using the same processing flow.

[0160] The data mapping model defines the mapping relationship between the fields in the data format and their corresponding multidimensional structure through the configuration information table. , where the mapping of the time dimension , Mapping of spatial dimensions (latitude and longitude) and the mapping of the height dimension Together they constitute the five-dimensional mapping relationship of the fields. During the standardization process, first, the time dimension is unified into a standard time format (such as UTC) according to the mapping rules, the longitude and latitude are converted into a consistent coordinate system (such as WGS84), and the height value is converted into a unified unit (such as meters). Then, all fields are standardized one by one according to the mapping rules, mapping them from a variety of formats to a consistent five-dimensional structure to ensure the relevance and consistency between data dimensions. Finally, the standardized data can be stored in an efficient format for subsequent analysis and service calls. This method effectively solves the complexity of parsing and processing multi-source heterogeneous data.

[0161] The universal configuration information table solves the technical problem of non-uniform formats of multi-source heterogeneous data. With the diversification of data sources, traditional data processing systems often need to design independent parsing logic for each format, which increases the complexity of the system and the development and maintenance costs. The present invention unifies the parsing process of multiple data formats by constructing a universal configuration information table, and realizes the standardized processing of cross-format data. This not only greatly simplifies the complexity of system development, but also improves the compatibility and scalability of the system, making future data source expansion easier.

[0162] Furthermore, the embodiment of the present invention can also realize complex file structure processing: the system supports parsing of nested file structures or compressed files. First, the compressed file is decompressed, and then the nested file is parsed layer by layer. For multi-layer nested data structures, the system can gradually expand the data content through a recursive parsing algorithm to ensure that each field and data information can be correctly read and mapped.

[0163] Furthermore, the execution status of each task is recorded in the log in real time, and the system tracks the execution time, execution results, and abnormal situations of the task. If there is a problem during the execution of the task, the system can quickly backtrack and remedy it. In addition, the parallel execution of tasks can be monitored through log analysis tools to help administrators optimize system performance.

[0164] (4) Generate a unified dataset:

[0165] After the data analysis is completed, the system generates standardized DAT format files and grayscale image files. DAT files are used to store three-dimensional meteorological and oceanographic data, and grayscale image files are used to visualize the spatial distribution of each physical element at a specific time and altitude. The generated DAT files and grayscale image files can be used for subsequent data query, analysis and rendering.

[0166] (4.1) DAT file generation and storage optimization: DAT files are stored through a layered compression and indexing mechanism to ensure efficient use of data storage space. DAT files are stored in three dimensions: altitude, longitude, and latitude, and indexed by physical elements, spatial resolution, and time dimensions to facilitate quick query by users. Each DAT file is accompanied by a metadata file that records the file format, generation time, element description, etc., to facilitate subsequent data management and analysis.

[0167] (4.2) Grayscale image generation and image compression: The system generates a grayscale image file based on the results of data analysis. The grayscale image is normalized to represent the two-dimensional distribution of a specific physical element at a certain time and height. The generated grayscale image file is compressed using an efficient image compression algorithm (such as JPEG2000 or WebP) to reduce the file transfer size and improve the data transmission speed between the front and back ends. The compressed grayscale image file can be quickly transferred to the front end for real-time rendering and display.

[0168] (5) Standardized rendering and query services:

[0169] In one example, if a service request for a data processing result file is received, target data information is extracted from the data processing result file based on the query conditions carried in the service request, and the target data information is rendered and displayed; wherein the service request includes one or more of a rendering service request, a single-point timing diagram service request, and a profile diagram service request.

[0170] The present invention provides standardized rendering and query services, and users can obtain corresponding data results based on the query conditions (such as latitude and longitude, physical elements, time, etc.) entered. The system ensures that efficient query and display services can be provided even in complex multidimensional data sets through fast query and real-time rendering. Specifically:

[0171] (5.1) Real-time query and rendering: Users input query conditions through the front end, and the system will quickly extract the required data information based on the DAT file and grayscale image file, and perform real-time rendering. In order to improve the query speed, the system uses a cache mechanism to cache common query conditions and reduce the burden of repeated queries. When the user makes a new query, the system will quickly find the target data based on the index of the DAT file and generate the corresponding visualization image.

[0172] (5.2) Time series and profile services: The system supports query services for time series and profiles. Time series graphs show the trend of element changes at specific longitude and latitude points in different time periods, and profile graphs show the distribution of elements at different altitudes at specific time points. The system optimizes the query algorithm to quickly locate the target data from the DAT file, ensuring that the query results can be quickly presented to the user, and uses modern front-end technologies such as WebGL for image rendering to achieve a smooth user interaction experience.

[0173] (5.3) Query optimization and data management: The system adopts a distributed data query architecture, which can automatically adjust the query strategy according to the complexity of the query request and the amount of data, ensuring a high response speed even in high-concurrency query situations. The system ensures efficient execution of queries through indexing and caching mechanisms, and can dynamically adjust the display format of query results according to user needs, such as time series curves, two-dimensional heat maps, or three-dimensional profiles.

[0174] The standardized data service and rendering system of the present invention achieves the ability to quickly respond to user query requests by generating a unified data set and grayscale image files. Traditional data rendering and query systems often have a long response time due to excessive data volume or complex format, which affects the user experience. The present invention ensures that the system can return query results in a short time and meet the real-time needs of users through optimized data storage format and cache mechanism. In addition, the scalability provided by the system enables it to adapt to larger-scale data processing and query tasks in the future, and has the potential for long-term development in practical applications.

[0175] For further information, see Figure 4 The functional radar chart of a multi-source heterogeneous data processing method shown is used to display the functional coverage and implementation level of each module in the system, including multiple modules such as "data preprocessing module", "data acquisition module", and "data analysis module", which presents the functional distribution and comprehensiveness of the system in an intuitive way, helping to evaluate the integrity of system development and the relative importance of modules.

[0176] In summary, the purpose of the present invention is to provide a multi-source heterogeneous data processing method to cope with the challenges encountered in the existing technology when processing large-scale, multi-source heterogeneous meteorological and ocean grid data. The amount of data in the meteorological and ocean fields is huge and the data sources are extensive, often in different formats and structures, which puts extremely high demands on the unified parsing and efficient processing of data. Existing solutions usually have many limitations in data parsing, processing efficiency, and service response, and cannot cope with the growing amount of data and real-time requirements. The embodiment of the present invention successfully solves the problems of low efficiency, non-uniform format, slow response speed and other problems in the existing technology of multi-source heterogeneous data processing. In the large-scale meteorological and ocean data processing scenario, traditional technologies are often unable to cope with the complexity and diversity of massive data, especially in applications with high real-time requirements. The embodiment of the present invention realizes an efficient and standardized data processing process through the organic combination of multiple technologies, and has a high degree of flexibility and scalability, which can be widely used in meteorology, ocean monitoring and other multi-dimensional data processing fields.

[0177] Specifically, the embodiments of the present invention can solve the following key problems:

[0178] Regarding the aforementioned problem 1, the goal of the embodiment of the present invention is to efficiently parse and convert data from different data sources and in different formats within the same framework by introducing a universal configuration information table. The universal configuration information table provides a standardized mapping mechanism that maps fields in different data formats to a unified five-dimensional structure (element, time, longitude, latitude, altitude). This design not only enables the system to parse files from different data sources, but also ensures that all data can be uniformly stored and processed after parsing, avoiding the complexity of multi-format parsing.

[0179] Through the embodiments of the present invention, users can easily process heterogeneous data from different data sources. The system automatically adapts and parses the data, eliminating the work bottleneck caused by inconsistent formats and greatly improving the efficiency of heterogeneous data parsing.

[0180] For the aforementioned problem 2, the goal of the embodiment of the present invention is to achieve rapid acquisition, allocation and processing of large-scale data by introducing distributed task scheduling and cache database. The distributed task scheduling system dynamically allocates tasks according to the update frequency and file size of each data directory to ensure that different data processing tasks can be executed in parallel and avoid single-point task blocking. At the same time, the application of cache database technology enables the system to temporarily store intermediate processing results and quickly call them in subsequent tasks, reducing the overhead of repeated processing.

[0181] This architecture enables the system to efficiently cope with the challenges of massive data. Regardless of the diversity of data sources or the update frequency, it can achieve rapid data storage and processing through reasonable task scheduling and distributed processing architecture, greatly improving the overall processing efficiency.

[0182] Regarding the aforementioned question three, the goal of the embodiment of the present invention is to provide standardized data rendering services and query services by generating a unified data set and grayscale image files, ensuring that the system can quickly respond to user query requests when processing five-dimensional meteorological and oceanographic data. The unified data set is stored in the efficient dat format, and combined with the standardized processing of grayscale images, the system can quickly generate two-dimensional images for front-end display, significantly improving the speed of data query and rendering. In addition, with the help of a cache database and an intelligent scheduling mechanism, the system can cache common query requests and increase data access speed.

[0183] Through the embodiments of the present invention, users can obtain the required information from complex five-dimensional meteorological and oceanographic data in a very short time and quickly visualize it, meeting the needs for real-time and high-frequency queries. This capability is particularly important for application scenarios that require rapid response, such as weather forecasting and ocean monitoring.

[0184] In addition, although the embodiments of the present invention are mainly aimed at the processing of meteorological and oceanographic data, the multi-source heterogeneous data processing framework proposed therein is highly versatile and applicable to other multidimensional data processing fields. For example, in the fields of environmental monitoring, geological exploration, remote sensing data analysis, etc., the data also comes from a wide range of sources and has a complex format. The technology proposed in the embodiments of the present invention can also be applied to the analysis and processing of complex multi-source data in these fields. Therefore, the embodiments of the present invention are not only limited to the processing of meteorological and oceanographic data, but also provide an efficient and standardized framework for other multidimensional data processing, which has a wide range of application prospects.

[0185] Based on the above embodiments, an embodiment of the present invention provides a multi-source heterogeneous data processing device, which is applied to a system, wherein a data scanning cluster and a multi-level data processing cluster are configured in the system, see Figure 5 The structure diagram of a multi-source heterogeneous data processing device shown in FIG. 1 mainly includes the following parts:

[0186] The scanning task creation module 502 is used to monitor multiple data directories and create a data scanning task for the changed target data directory when it is monitored that the multi-source heterogeneous data files stored in the data directory have changed. The target data directory corresponds to the data scanning task one by one.

[0187] The scanning task execution module 504 is used to execute the data scanning task for the target data directory by using the nodes included in the data scanning cluster to obtain the multi-source heterogeneous data files that have changed in the target data directory;

[0188] The processing task execution module 506 is used to utilize the nodes contained in the multi-level data processing cluster to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file based on a preset general configuration information table to obtain a data processing result file corresponding to the multi-source heterogeneous data file.

[0189] The multi-source heterogeneous data processing device provided by the embodiment of the present invention introduces a universal configuration information table to perform standardized conversion on multi-source heterogeneous data from different data sources and with non-uniform formats, thereby realizing unified processing of multi-source heterogeneous data, helping to improve the processing efficiency of multi-source heterogeneous data and reducing the complexity of multi-format data processing. In addition, the use of data scanning clusters and multi-level data processing clusters can efficiently cope with the challenges of processing massive amounts of multi-source heterogeneous data, greatly improving the processing efficiency of multi-source heterogeneous data, thereby effectively alleviating problems such as task congestion, data loss, and service delays in the process of multi-source heterogeneous data processing.

[0190] In one implementation, the scanning task execution module 504 is specifically used to:

[0191] Monitor the CPU utilization, memory usage, and network bandwidth of each node in the data scanning cluster to determine the load of each node in the data scanning cluster;

[0192] Determine the first call priority of the data scanning task on each node in the data scanning cluster according to the weight of the data scanning task, the time interval of the last execution of the data scanning task for the target data directory, and the load of each node in the data scanning cluster;

[0193] Allocate a scanning node for the data scanning task according to the load corresponding to each node in the data scanning cluster and the first call priority of the data scanning task on each node in the data scanning cluster;

[0194] The scanning node is used to perform data scanning tasks on the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory.

[0195] In one embodiment, a pre-processing module is further included, which is used to:

[0196] Perform integrity checks on multi-source heterogeneous data files;

[0197] If the multi-source heterogeneous data file passes the integrity check, then the file features of the multi-source heterogeneous data file are extracted;

[0198] Through a machine learning model, a target preprocessing step corresponding to a multi-source heterogeneous data file is determined from a plurality of pre-configured preprocessing steps based on file features, so that the multi-source heterogeneous data file is preprocessed using the target preprocessing step.

[0199] In one embodiment, the preprocessing module is specifically used for:

[0200] Based on the historical execution results of multiple pre-configured pre-processing steps, dynamically adjust the weight corresponding to each pre-processing step;

[0201] Through the machine learning model, based on the file features, the historical execution results and weights corresponding to each preprocessing step, the processing time consumed by each preprocessing step to preprocess multi-source heterogeneous data files is predicted;

[0202] Determine the second scheduling priority corresponding to each preprocessing step according to the weight and processing time corresponding to each preprocessing step;

[0203] According to the second scheduling priority, at least one target preprocessing step corresponding to the multi-source heterogeneous data file is determined from each preprocessing step.

[0204] In one embodiment, the processing task execution module 506 is specifically used to: utilize the nodes included in the multi-level data processing cluster, based on the preset general configuration information table, perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file, and obtain the data processing result file corresponding to the multi-source heterogeneous data file, including:

[0205] Use the Kafka message queue to push data replication and decompression tasks for multi-source heterogeneous data files to the first-level data processing cluster, and use the nodes contained in the first-level data processing cluster to perform data replication and decompression tasks for multi-source heterogeneous data files to obtain file decompression results;

[0206] Continue to use the Kafka message queue to push data processing and parsing tasks to the second-level data processing cluster for the file decompression results, and use the nodes contained in the second-level data processing cluster to perform data processing and parsing tasks on the file decompression results. The data processing and parsing tasks are used to perform standardized conversions on the fields corresponding to each data dimension in the file decompression results based on the preset general configuration information table to obtain the data processing result file.

[0207] In one embodiment, the processing task execution module 506 is specifically used to: use the nodes included in the first-level data processing cluster to perform data replication and decompression tasks for multi-source heterogeneous data files, including:

[0208] Determine the third scheduling priority of each node in the first-level data processing cluster for the multi-source heterogeneous data file based on the current resource occupancy rate and current processing capacity corresponding to each node in the first-level data processing cluster;

[0209] Allocating processing nodes to multi-source heterogeneous data files according to a third scheduling priority;

[0210] Use processing nodes to perform data replication and decompression tasks for multi-source heterogeneous data files.

[0211] In one implementation, the processing task execution module 506 is specifically used to: perform standardized conversion on the fields corresponding to each data dimension in the file decompression result based on a preset general configuration information table, to obtain a data processing result file, including:

[0212] Based on the mapping relationship between the data structure described in the preset general configuration information table and the fields corresponding to multiple data dimensions, the fields corresponding to each data dimension in the file decompression result are standardized and converted to obtain a data processing result file.

[0213] In one embodiment, a service module is further included, which is used to:

[0214] If a service request for a data processing result file is received, target data information is extracted from the data processing result file based on the query conditions carried in the service request, and the target data information is rendered and displayed;

[0215] The service request includes one or more of a rendering service request, a single-point timing diagram service request, and a profile diagram service request.

[0216] The device provided in the embodiment of the present invention has the same implementation principle and technical effects as those of the aforementioned method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0217] An embodiment of the present invention provides a multi-source heterogeneous data processing system. Specifically, the system includes a processor and a storage device. The storage device stores a computer program, and when the computer program is executed by the processor, it executes the method described in any one of the above-mentioned embodiments.

[0218] Figure 6 A structural diagram of a multi-source heterogeneous data processing system provided for an embodiment of the present invention, the system 100 includes: a processor 60, a memory 61, a bus 62 and a communication interface 63, wherein the processor 60, the communication interface 63 and the memory 61 are connected via the bus 62; the processor 60 is used to execute an executable module stored in the memory 61, such as a computer program.

[0219] The memory 61 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 63 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0220] The bus 62 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0221] Among them, the memory 61 is used to store programs, and the processor 60 executes the program after receiving the execution instruction. The method executed by the device for flow process definition disclosed in any embodiment of the above-mentioned embodiment of the present invention can be applied to the processor 60 or implemented by the processor 60.

[0222] The processor 60 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 60. The above processor 60 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in the embodiment of the present invention can be directly embodied as a hardware decoding processor to execute, or can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 61, and the processor 60 reads the information in the memory 61 and completes the steps of the above method in combination with its hardware.

[0223] The computer program product of the readable storage medium provided in the embodiment of the present invention includes a computer-readable storage medium storing program code, and the instructions included in the program code can be used to execute the methods described in the previous method embodiments. The specific implementation can be referred to the previous method embodiments, which will not be repeated here.

[0224] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, system, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0225] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A method for processing multi-source heterogeneous data, characterized in that: The method is applied to a system, wherein a data scanning cluster and a multi-level data processing cluster are configured in the system, and the method comprises: Monitor multiple data directories, and when changes are detected in multi-source heterogeneous data files stored in the data directories, create a data scanning task for the changed target data directories, wherein the target data directories correspond to the data scanning tasks one by one; Utilizing the nodes included in the data scanning cluster, executing the data scanning task for the target data directory, and obtaining the multi-source heterogeneous data files that have changed in the target data directory; Using the nodes included in the multi-level data processing cluster, based on a preset general configuration information table, a field corresponding to each data dimension in the multi-source heterogeneous data file is standardized and converted to obtain a data processing result file corresponding to the multi-source heterogeneous data file; Utilizing the nodes included in the data scanning cluster, executing the data scanning task for the target data directory, and obtaining the multi-source heterogeneous data files that have changed in the target data directory, including: monitoring the CPU utilization, memory usage, and network bandwidth of each node in the data scanning cluster to determine the load situation corresponding to each node in the data scanning cluster; determining the first call priority of the data scanning task on each node in the data scanning cluster according to the weight of the data scanning task, the time interval of the last execution of the data scanning task for the target data directory, and the load situation of each node in the data scanning cluster; allocating scanning nodes to the data scanning task according to the load situation corresponding to each node in the data scanning cluster and the first call priority of the data scanning task on each node in the data scanning cluster; utilizing the scanning nodes to execute the data scanning task for the target data directory, and obtaining the multi-source heterogeneous data files that have changed in the target data directory; Before using the nodes included in the multi-level data processing cluster to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file based on a preset general configuration information table to obtain a data processing result file corresponding to the multi-source heterogeneous data file, the method further includes: performing an integrity check on the multi-source heterogeneous data file; if the multi-source heterogeneous data file passes the integrity check, extracting the file features of the multi-source heterogeneous data file; and determining, through a machine learning model, a target preprocessing step corresponding to the multi-source heterogeneous data file from a plurality of pre-configured preprocessing steps based on the file features, so as to preprocess the multi-source heterogeneous data file using the target preprocessing step; Through a machine learning model, based on the file features, a target preprocessing step corresponding to the multi-source heterogeneous data file is determined from a plurality of preprocessing steps that are preconfigured, including: based on the historical execution results of the plurality of preprocessing steps that are preconfigured, a weight corresponding to each of the preprocessing steps is dynamically adjusted; through a machine learning model, based on the file features, the historical execution results corresponding to each of the preprocessing steps and the weight, a processing time consumed for preprocessing the multi-source heterogeneous data file using each of the preprocessing steps is predicted; according to the weight and the processing time corresponding to each of the preprocessing steps, a second scheduling priority corresponding to each of the preprocessing steps is determined; according to the second scheduling priority, at least one target preprocessing step corresponding to the multi-source heterogeneous data file is determined from each of the preprocessing steps.

2. The multi-source heterogeneous data processing method according to claim 1, characterized in that: Using the nodes included in the multi-level data processing cluster, based on a preset general configuration information table, a field corresponding to each data dimension in the multi-source heterogeneous data file is standardized and converted to obtain a data processing result file corresponding to the multi-source heterogeneous data file, including: Using the Kafka message queue, pushing the data copy and decompression tasks for the multi-source heterogeneous data files to the first-level data processing cluster, and using the nodes included in the first-level data processing cluster, executing the data copy and decompression tasks for the multi-source heterogeneous data files to obtain the file decompression results; Continue to use the Kafka message queue to push data processing and parsing tasks to the second-level data processing cluster for the file decompression result, and use the nodes contained in the second-level data processing cluster to execute the data processing and parsing tasks for the file decompression result. The data processing and parsing tasks are used to perform standardized conversions on the fields corresponding to each data dimension in the file decompression result based on a preset general configuration information table to obtain a data processing result file.

3. The multi-source heterogeneous data processing method according to claim 2, characterized in that: Utilizing the nodes included in the first-level data processing cluster to perform the data replication and decompression tasks for the multi-source heterogeneous data files includes: Determine a third scheduling priority of the multi-source heterogeneous data file on each node in the first-level data processing cluster based on a current resource occupancy rate and a current processing capacity corresponding to each node in the first-level data processing cluster; Allocating processing nodes to the multi-source heterogeneous data files according to the third scheduling priority; The processing nodes are utilized to execute the data replication and decompression tasks for the multi-source heterogeneous data files.

4. The multi-source heterogeneous data processing method according to claim 2, characterized in that: Based on the preset general configuration information table, the fields corresponding to each data dimension in the file decompression result are standardized and converted to obtain a data processing result file, including: Based on the mapping relationship between the data structure described in the preset general configuration information table and the fields corresponding to the multiple data dimensions, the fields corresponding to each of the data dimensions in the file decompression result are standardized and converted to obtain a data processing result file.

5. The multi-source heterogeneous data processing method according to claim 1, characterized in that: The method further comprises: If a service request for the data processing result file is received, extracting target data information from the data processing result file based on the query condition carried in the service request, and rendering and displaying the target data information; The service request includes one or more of a rendering service request, a single-point timing diagram service request and a profile diagram service request.

6. A multi-source heterogeneous data processing device, characterized in that: The device is applied to a system, wherein a data scanning cluster and a multi-level data processing cluster are configured in the system, and the device comprises: A scanning task creation module is used to monitor multiple data directories and, when it is monitored that the multi-source heterogeneous data files stored in the data directories have changed, create a data scanning task for the changed target data directory, wherein the target data directory corresponds to the data scanning task one by one; A scanning task execution module, used to execute the data scanning task for the target data directory using the nodes included in the data scanning cluster, and obtain the multi-source heterogeneous data files that have changed in the target data directory; A processing task execution module, used to utilize the nodes included in the multi-level data processing cluster to perform standardized conversion on the fields corresponding to each data dimension in the multi-source heterogeneous data file based on a preset general configuration information table, so as to obtain a data processing result file corresponding to the multi-source heterogeneous data file; The scanning task execution module is specifically used to: monitor the CPU utilization, memory usage, and network bandwidth of each node in the data scanning cluster to determine the load situation corresponding to each node in the data scanning cluster; determine the first calling priority of the data scanning task on each node in the data scanning cluster according to the weight of the data scanning task, the time interval of the last execution of the data scanning task for the target data directory, and the load situation of each node in the data scanning cluster; assign a scanning node to the data scanning task according to the load situation corresponding to each node in the data scanning cluster and the first calling priority of the data scanning task on each node in the data scanning cluster; use the scanning node to execute the data scanning task for the target data directory to obtain the multi-source heterogeneous data files that have changed in the target data directory; The system further includes a preprocessing module, which is used to: perform an integrity check on the multi-source heterogeneous data file; if the multi-source heterogeneous data file passes the integrity check, extract the file features of the multi-source heterogeneous data file; and determine a target preprocessing step corresponding to the multi-source heterogeneous data file from a plurality of pre-configured preprocessing steps based on the file features through a machine learning model, so as to preprocess the multi-source heterogeneous data file using the target preprocessing step; The preprocessing module is specifically used to: dynamically adjust the weight corresponding to each preprocessing step based on the historical execution results of multiple preprocessing steps pre-configured; predict the processing time consumed by preprocessing the multi-source heterogeneous data file using each preprocessing step based on the file features, the historical execution results corresponding to each preprocessing step and the weight through a machine learning model; determine the second scheduling priority corresponding to each preprocessing step according to the weight and the processing time corresponding to each preprocessing step; and determine at least one target preprocessing step corresponding to the multi-source heterogeneous data file from each preprocessing step according to the second scheduling priority.

7. A multi-source heterogeneous data processing system, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer executable instructions that can be executed by the processor, and the processor executes the computer executable instructions to implement the method according to any one of claims 1 to 5.

Images