Method and device for automatically downloading and processing ocean data

Abstract

The invention relates to the technical field of data acquisition, and particularly provides a method and a device for automatically downloading and processing ocean data, a data capturing module captures a data file in a NetCDF format from a designated ocean data center, and captures SAR type data from Alaka SATF; the data processing module is used for decoding, cleaning and converting the captured data file in the NetCDF format; and the data storage module stores the processed data in a database or a file system for long-term storage and retrieval. Compared with the prior art, the method has the advantages that the defects in the prior art can be effectively overcome, and efficient, accurate and flexible data acquisition and processing are realized.

Description

Technical Field

[0001] The present invention relates to the technical field of data acquisition, and specifically provides a method and device for automatically downloading and processing ocean data. Background Art

[0002] With the increasing demands for climate change and ocean environment monitoring, the real-time acquisition and processing of ocean data have become particularly important. Ocean data is usually stored in the NetCDF format, which is a widely used format for storing scientific data and is suitable for the storage and exchange of multi-dimensional data. SAR data, i.e., synthetic aperture radar data, is a type of data that generates surface images by transmitting and receiving microwave signals through active remote sensing technology. Currently, most of these data acquisition methods rely on manual downloading and processing, which is inefficient and error-prone and cannot meet the needs of large-scale and real-time data processing.

[0003] In response to this problem, although there are already some automated data acquisition tools, they often lack flexibility and generality and are difficult to adapt to different data sources and processing requirements.

[0004] In addition, existing solutions have limitations in terms of the complexity and scalability of data processing and cannot effectively handle the increasing data volume and complex processing tasks. At the same time, the existing data download technology combines less with ocean data download, making it more difficult to download ocean-related data.

[0005] Therefore, how to solve the deficiencies in the prior art and improve the efficiency of data acquisition and processing is an urgent problem for those skilled in the art to solve. Summary of the Invention

[0006] The present invention aims at the above-mentioned deficiencies of the prior art and provides a method for automatically downloading and processing ocean data with strong practicability.

[0007] The further technical task of the present invention is to provide a device for automatically downloading and processing ocean data with reasonable design, safety and applicability.

[0008] The technical solution adopted by the present invention to solve its technical problems is:

[0009] A method for automatically downloading and processing ocean data, wherein a data scraping module scrapes data files in the NetCDF format from a designated ocean data center and scrapes SAR type data from the Alaska Satellite Facility;

[0010] A data processing module decodes, cleans, and converts the scraped data files in the NetCDF format;

[0011] The data storage module stores the processed data in a database or a file system for long-term preservation and retrieval.

[0012] Furthermore, the Spring Boot framework is used to implement scheduled task scheduling. The automatic download of data is controlled by the task interval time passed from the front end and the specific method to be executed. For some special data websites, Python scripts are called through Java scheduled tasks for downloading.

[0013] Furthermore, by utilizing the scheduled task function of Spring Boot, according to the task interval time set by the front end, the data scraping task is triggered at regular intervals. In the front-end configuration, through the front-end interface, the user sets the parameters of the scraping task, and the configuration information is transmitted to the back-end system via the API.

[0014] Furthermore, for data integrity detection, before downloading the data, its integrity is detected by reading the data; if the data is incomplete, the system will continue to download; if the data is complete, the download step will be skipped.

[0015] Furthermore, for some special data websites, Java calls Python scripts through scheduled tasks for data downloading. The Python script is responsible for downloading data in a specific format or with complex logic and returns the download result to the Java system.

[0016] Furthermore, for error handling and logging, for error handling, there is an error handling mechanism to promptly detect and handle problems that occur during the scraping and processing processes;

[0017] In logging, the operation logs of the system are detailedly recorded, including the links of data scraping, processing, and storage, for tracking and auditing.

[0018] Furthermore, the downloaded NetCDF format data files are decoded to remove invalid data and noise, and the decoded data is converted into a unified format for subsequent processing and storage.

[0019] An apparatus for automatically downloading and processing ocean data includes: at least one memory and at least one processor;

[0020] The at least one memory is used to store machine-readable programs;

[0021] The at least one processor is used to call the machine-readable program to execute a method for automatically downloading and processing ocean data.

[0022] Compared with the prior art, a method and an apparatus for automatically downloading and processing ocean data of the present invention have the following outstanding beneficial effects:

[0023] Improve data download efficiency: Through the scheduled task function of Spring Boot, automatic data download is realized, greatly reducing manual intervention and improving the efficiency of data download. After testing, compared with manual download, the data download time is reduced by more than 90% on average.

[0024] Flexible configuration: Users can set the parameters of the scraping task through the front-end interface, such as scraping frequency, target website, data screening conditions, etc., to meet the needs of different application scenarios. The system can automatically adjust the task according to the configuration to ensure the timeliness and accuracy of data acquisition.

[0025] Graphical interface: The system provides a graphical user interface, allowing users to conveniently set the parameters and configuration information of the scraping task, greatly reducing the usage threshold and improving the user experience.

[0026] Wide applicability: The system is applicable to various scenarios such as marine scientific research, weather forecasting, environmental protection, etc., providing reliable data support. Brief Description of the Drawings

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0028] Attached Figure 1 is a schematic flow chart of a method for automatic download and processing of marine data. Detailed Embodiments

[0029] In order to enable those skilled in the art to better understand the solutions of the present invention, the following will further describe the present invention in detail in conjunction with specific embodiments. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts belong to the scope of protection of the present invention.

[0030] The following gives a best embodiment:

[0031] As Figure 1 shown, in a method for automatic download and processing of marine data in this embodiment, the data scraping module is responsible for scraping data files in NetCDF format from specified marine data centers (hycom, ERA5, Copernicus Marine Service) and scraping SAR type data from the Alaska Satellite Facility.

[0032] Use the Spring Boot framework to implement scheduled task scheduling, and control the automatic download of data through the task interval time passed by the front end and the specific methods to be executed.

[0033] For some special data websites (ERA5, Copernicus Marine Service, Alaska Satellite Facility), Python scripts are called through Java scheduled tasks for downloading.

[0034] Scheduled task scheduling, utilize the scheduled task function of Spring Boot, and trigger data scraping tasks at regular intervals according to the task interval time set by the front end.

[0035] Front-end configuration, through the front-end interface, users can set the parameters of the scraping task, including data source address, scraping frequency, data filtering conditions, etc. The configuration information is passed to the back-end system through the API.

[0036] Data integrity detection, before downloading data, detect its integrity by reading the data. If the data is incomplete, the system will continue to download; if the data is complete, the download step will be skipped.

[0037] For some specific data sources (ERA5, Copernicus Marine Service, Alaska Satellite Facility), Java calls Python scripts through scheduled tasks for data download. The Python script is responsible for downloading data in a specific format or with complex logic, and returns the download result to the Java system.

[0038] Error handling and logging: Error handling, the system has a perfect error handling mechanism, which can detect and handle problems that occur during the scraping and processing process in a timely manner to ensure the stable operation of the system.

[0039] Logging: Detailed operation logs of the system are recorded, including all links of data scraping, processing, and storage, for tracking and auditing.

[0040] The data processing module decodes, cleans, and converts the captured NetCDF format data files to ensure the availability and standardization of the data.

[0041] Decode the downloaded NetCDF format data files, remove invalid data and noise, and convert the decoded data into a unified format for subsequent processing and storage.

[0042] The data storage module stores the processed data in a database or file system for long-term preservation and quick retrieval.

[0043] Based on the above method, a device for automatic download and processing of ocean data in this embodiment includes: at least one memory and at least one processor;

[0044] The at least one memory is used for storing machine-readable programs;

[0045] The at least one processor is used for calling the machine-readable program to execute a method for automatic download and processing of ocean data.

[0046] The above specific embodiments are only specific cases of the present invention. The patent protection scope of the present invention includes but is not limited to the above specific embodiments. Any technical solution that conforms to the technical solutions described in the above specific embodiments of the present invention and any appropriate changes or substitutions made by those of ordinary skill in the art shall fall within the patent protection scope of the present invention.

[0047] Although the embodiments of the present invention have been shown and described, for those of ordinary skill in the art, it can be understood that various changes, modifications, substitutions, and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for automatic downloading and processing of ocean data, characterized in that, The data scraping module scrapes data files in NetCDF format from a designated marine data center and SAR type data from the Alaska Satellite Facility; The data processing module decodes, cleans, and transforms the scraped NetCDF format data files; The data storage module stores the processed data in a database or file system for long-term preservation and retrieval.

2. The method for automatically downloading and processing marine data according to claim 1, wherein The Spring Boot framework is used to implement scheduled task scheduling. The automatic download of data is controlled by the task interval time passed by the front end and the specific method to be executed. For some special data websites, Python scripts are called through Java scheduled tasks for downloading.

3. The method for automatically downloading and processing ocean data according to claim 2, characterized in that, Utilizing the scheduled task function of Spring Boot, according to the task interval time set by the front end, the data scraping task is triggered periodically. In the front-end configuration, through the front-end interface, the user sets the parameters of the scraping task, and the configuration information is passed to the back-end system through the API.

4. A method for automatically downloading and processing ocean data according to claim 3, characterized in that, Data integrity detection: Before downloading the data, its integrity is detected by reading the data; if the data is incomplete, the system will continue to download; if the data is complete, the download step will be skipped.

5. A method for automatically downloading and processing ocean data according to claim 4, characterized in that For some special data websites, Java calls Python scripts through scheduled tasks for data downloading. The Python script is responsible for downloading data in a specific format or with complex logic and returns the download result to the Java system.

6. The method for automatically downloading and processing ocean data according to claim 5, wherein Regarding error handling and logging, for error handling, there is an error handling mechanism to promptly detect and handle problems that occur during the scraping and processing processes; In logging, the operation logs of the system are detailedly recorded, including the links of data scraping, processing, and storage, for tracking and auditing.

7. A method for automatically downloading and processing ocean data according to claim 6, characterized in that, Decode the downloaded NetCDF format data files, remove invalid data and noise, and convert the decoded data into a unified format for subsequent processing and storage.

8. An apparatus for automatically downloading and processing ocean data, characterized in that Including: At least one memory and at least one processor; The at least one memory is used to store machine-readable programs; The at least one processor is used to call the machine-readable program and execute the method described in any one of claims 1 to 7.

Images