Multi-source marine environment data intelligent sharing method

Abstract

The invention discloses a multi-source marine environment data intelligent sharing method. The method comprises the following steps: acquiring multi-source marine environment data; product adaptation is carried out on the multi-source marine environment data to obtain first multi-source marine environment data, and the product adaptation is used for indicating the multi-source marine environment data meeting user requirements; performing data packaging on the first multi-source marine environment data to obtain second multi-source marine environment data; performing distributed storage on the second multi-source marine environment data to obtain stored multi-source marine environment data; and carrying out intelligent data sharing on the stored multi-source marine environment data through intelligent subscription distribution. The data sharing efficiency and the distribution speed can be improved.

Description

Technical Field

[0001] The present application relates to the technical field of intelligent data sharing, and in particular to a method for intelligent sharing of multi-source marine environment data. Background Art

[0002] The rapid growth of marine environmental data and technological progress have brought about an urgent need for efficient data sharing and management. Marine environmental data not only include historical data, but also include marine geological and surveying historical data, basic information products, thematic products, early warning products and other types of data. These data are of great value to many fields such as marine scientific research, resource development, environmental monitoring and disaster warning.

[0003] However, the diversity and massiveness of ocean data pose challenges to data storage, processing and sharing. Traditional technologies mainly rely on centralized storage and manual distribution systems. These systems usually use a single data center to store and manage massive data, and transmit data through protocols such as FTP and HTTP. Centralized storage systems usually use large database management systems (such as Oracle, MySQL) to manage data, and cooperate with file system storage (such as NFS, HDFS) to achieve centralized access to data. In addition, data distribution methods usually push data from central nodes to user terminals or edge nodes through manual or scheduled tasks, which lacks flexibility and real-time performance. Therefore, traditional data sharing technologies are difficult to meet the current needs for efficient management and rapid distribution of ocean data. Summary of the invention

[0004] This application aims to propose a multi-source marine environment data intelligent sharing method that can improve data sharing efficiency and distribution speed.

[0005] In a first aspect, an embodiment of the present application provides a method for intelligently sharing multi-source marine environment data, the method comprising:

[0006] Obtain marine environmental data from multiple sources;

[0007] Performing product adaptation on the multi-source ocean environment data to obtain first multi-source ocean environment data, wherein the product adaptation is used to indicate multi-source ocean environment data that meets user needs;

[0008] Packaging the first multi-source ocean environment data to obtain second multi-source ocean environment data;

[0009] Distributed storage is performed on the second multi-source ocean environment data to obtain stored multi-source ocean environment data;

[0010] The stored multi-source ocean environment data is intelligently shared through intelligent subscription distribution.

[0011] In some embodiments, the intelligent sharing of the stored multi-source ocean environment data through intelligent subscription distribution includes:

[0012] Obtaining a user request including multiple conditions, and obtaining metadata corresponding to each data record and condition in the stored multi-source marine environment data;

[0013] Based on the metadata, calculating the matching degree between each data record and each condition in the user request;

[0014] Determining, according to the matching degree, a marine environment subscription dataset that meets multiple conditions in the user request;

[0015] The marine environment subscription dataset is intelligently shared through a content distribution strategy.

[0016] In some implementations, the degree of match between each data record and each condition in the user request is calculated based on the metadata in the following manner:

[0017]

[0018] Among them, M j represents the matching degree, Q represents the total number of conditions in the user request, and w i represents the importance weight of each condition, q i represents the i-th condition, m ij represents metadata, and f(·) represents the matching function.

[0019] In some embodiments, the intelligently sharing the marine environment subscription dataset through a content distribution strategy includes:

[0020] Get the product topics subscribed by the user;

[0021] Calculating the correlation between the product theme and the marine environment subscription data set to obtain a correlation calculation result;

[0022] Determining a target push data set according to the correlation calculation result;

[0023] The target data set is pushed for intelligent data sharing.

[0024] In some implementations, the intelligently sharing the target pushed data set includes:

[0025] The target pushed data set is intelligently shared in real-time push mode; or, the target pushed data set is intelligently shared in batch distribution mode.

[0026] In some embodiments, the correlation between the product topic and the marine environment subscription dataset is calculated as follows:

[0027]

[0028] Among them, R new represents the correlation, n represents the total amount of data in the marine environment subscription dataset, α k Represents the corresponding weight, s k (·) represents the correlation function, D new represents the marine environment subscription dataset, and S represents the product theme.

[0029] In some embodiments, the distributed storage of the second multi-source ocean environment data includes:

[0030] The second multi-source marine environment data is distributedly stored through MinIO.

[0031] In some embodiments, after intelligently sharing the stored multi-source ocean environment data through intelligent subscription distribution, the method further includes:

[0032] If new data is obtained or the data is updated, the data update is triggered through the event-driven mechanism;

[0033] The multi-source ocean environment data is synchronously updated through incremental synchronization.

[0034] In some embodiments, the multi-source ocean environment data is synchronously updated through incremental synchronization in the following manner:

[0035] ΔB=B(t+Δt)-B(t)

[0036]

[0037] Where ΔB represents the incremental data, B(t+Δt) represents the amount of data at time t+Δt, B(t) represents the amount of data at the current time t, and V sync Indicates the total amount of synchronized data blocks, N indicates the number of data blocks to be synchronized, ΔB j Indicates the jth data block that needs to be synchronized.

[0038] In some embodiments, after synchronously updating the multi-source ocean environment data through incremental synchronization, the method further includes:

[0039] The consistency of the data to be updated synchronously is checked through the hash function.

[0040] Compared with the prior art, this application has the following beneficial effects:

[0041] This method obtains the first multi-source marine environment data by product adaptation of the multi-source marine environment data; data packaging of the first multi-source marine environment data to obtain the second multi-source marine environment data; distributed storage of the second multi-source marine environment data to obtain the stored multi-source marine environment data; and intelligent data sharing of the stored multi-source marine environment data through intelligent subscription distribution. In this way, by performing distributed storage of the multi-source marine environment data, efficient storage and management of massive marine environment data can be achieved, and then intelligent data sharing of the multi-source marine environment data through intelligent subscription distribution can significantly improve the real-time and flexibility of data acquisition. Thus, data sharing efficiency and distribution speed can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the description of the embodiments in conjunction with the following drawings, in which:

[0043] Figure 1 It is a flowchart of an embodiment of a multi-source ocean environment data intelligent sharing method provided by the present application;

[0044] Figure 2 It is a structural schematic diagram of the best embodiment of the multi-source ocean environment data intelligent sharing method provided by the present application;

[0045] Figure 3 It is a flow chart of the best embodiment of the multi-source ocean environment data intelligent sharing method provided by this application. DETAILED DESCRIPTION

[0046] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and cannot be understood as limiting the present application.

[0047] In the description of this application, if there is a description of first, second, etc., it is only for the purpose of distinguishing technical features, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features or implicitly indicating the order of the indicated technical features.

[0048] In the description of the present application, it should be understood that the descriptions involving orientation, such as the orientation or positional relationship indicated as up, down, etc., are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0049] In the description of this application, it should be noted that, unless otherwise clearly defined, terms such as setting, installing, connecting, etc. should be understood in a broad sense, and technicians in the relevant technical field can reasonably determine the specific meaning of the above terms in this application based on the specific content of the technical solution.

[0050] The rapid growth of marine environmental data and technological progress have brought about an urgent need for efficient data sharing and management. Marine environmental data not only include historical data, but also include marine geological and surveying historical data, basic information products, thematic products, early warning products and other types of data. These data are of great value to many fields such as marine scientific research, resource development, environmental monitoring and disaster warning.

[0051] However, the diversity and massiveness of ocean data pose challenges to data storage, processing and sharing. Traditional technologies mainly rely on centralized storage and manual distribution systems. These systems usually use a single data center to store and manage massive data, and transmit data through protocols such as FTP and HTTP. Centralized storage systems usually use large database management systems (such as Oracle, MySQL) to manage data, and cooperate with file system storage (such as NFS, HDFS) to achieve centralized access to data. In addition, data distribution methods usually push data from central nodes to user terminals or edge nodes through manual or scheduled tasks, which lacks flexibility and real-time performance. Therefore, traditional data sharing technologies are difficult to meet the current needs for efficient management and rapid distribution of ocean data.

[0052] In order to solve the problem that the above-mentioned traditional data sharing technology is difficult to meet the current needs of efficient management and rapid distribution of ocean data, this application proposes a multi-source ocean environment data intelligent sharing method.

[0053] Reference Figure 1 The embodiment of the present application provides a multi-source marine environment data intelligent sharing method, the method comprising the following steps:

[0054] Step S100, acquiring multi-source ocean environment data;

[0055] Step S200: Perform product adaptation on the multi-source ocean environment data to obtain first multi-source ocean environment data, where the product adaptation is used to indicate the multi-source ocean environment data that meets user needs;

[0056] Step S300, packaging the first multi-source ocean environment data to obtain second multi-source ocean environment data;

[0057] Step S400: Distributed storage is performed on the second multi-source ocean environment data to obtain stored multi-source ocean environment data;

[0058] Step S500: Intelligently share the stored multi-source ocean environment data through intelligent subscription distribution.

[0059] In this embodiment, by performing product adaptation on multi-source marine environment data, first multi-source marine environment data is obtained; the first multi-source marine environment data is packaged to obtain second multi-source marine environment data; the second multi-source marine environment data is distributedly stored to obtain stored multi-source marine environment data; and the stored multi-source marine environment data is intelligently shared through intelligent subscription distribution. In this way, by performing distributed storage on multi-source marine environment data, efficient storage and management of massive marine environment data can be achieved, and then intelligent data sharing of multi-source marine environment data is performed through intelligent subscription distribution, which can significantly improve the real-time and flexibility of data acquisition. Thereby, data sharing efficiency and distribution speed can be improved.

[0060] The above-mentioned product adaptation of multi-source marine environmental data may be format adaptation, element adaptation, and region adaptation of multi-source marine environmental data.

[0061] The first multi-source ocean environment data may be packaged by using compression tools, such as ZIP and TAR.GZ, to reduce data volume and improve transmission efficiency.

[0062] In some embodiments, intelligent data sharing of stored multi-source ocean environment data is performed through intelligent subscription distribution, including:

[0063] Obtain a user request containing multiple conditions, and obtain metadata corresponding to each data record and condition in the stored multi-source marine environment data;

[0064] Based on the metadata, calculate the matching degree between each data record and each condition in the user request;

[0065] According to the matching degree, determine the marine environment subscription dataset that meets multiple conditions in the user request;

[0066] Intelligent data sharing of marine environment subscription datasets through content distribution strategies.

[0067] In this embodiment, by calculating the matching degree between each data record and each condition in the user request, and then determining the marine environment subscription data set that meets multiple conditions in the user request according to the matching degree, and then intelligently sharing the marine environment subscription data set through the content distribution strategy. In this way, determining the marine environment subscription data set according to the conditions in the user request can better meet the user's needs, and the user can subscribe to data products of a specific type or theme.

[0068] In some implementations, the matching degree between each data record and each condition in the user request is calculated based on the metadata in the following manner:

[0069]

[0070] Among them, M j represents the matching degree, Q represents the total number of conditions in the user request, and w i represents the importance weight of each condition, q i represents the i-th condition, m ij represents metadata, and f(·) represents the matching function.

[0071] In some embodiments, intelligent data sharing of the marine environment subscription dataset is performed through a content distribution strategy, including:

[0072] Get the product topics subscribed by the user;

[0073] Calculate the correlation between the product theme and the marine environment subscription data set to obtain the correlation calculation result;

[0074] According to the correlation calculation results, the target push data set is determined;

[0075] Push the target dataset for intelligent data sharing.

[0076] In this embodiment, after obtaining the marine environment subscription data set through the conditions given by the user, it is also necessary to calculate the correlation between the product theme and the marine environment subscription data set, so as to further obtain data that meets the user's needs, and then determine the target push data set (that is, the data that needs to be distributed and shared) based on the correlation calculation results, so as to quickly obtain customized data products to meet personalized needs.

[0077] In some implementations, intelligent data sharing of the target push data set includes:

[0078] The target push data set is pushed in real time for intelligent data sharing; or, the target push data set is pushed in batch distribution for intelligent data sharing.

[0079] In this embodiment, the target push data set is intelligently shared, and the optimal distribution path can be selected according to the user's geographical location, network conditions and data requirements. The system automatically manages the push and distribution of data based on subscription information, supports real-time data push and regular batch distribution, and significantly improves the real-time and flexibility of data acquisition.

[0080] In some embodiments, the correlation between the product topic and the marine environment subscription dataset is calculated as follows:

[0081]

[0082] Among them, R new represents the correlation, n represents the total amount of data in the marine environment subscription dataset, α k Represents the corresponding weight, s k (·) represents the correlation function, D new represents the marine environment subscription dataset, and S represents the product theme.

[0083] In some embodiments, distributing and storing the second multi-source ocean environment data includes:

[0084] Distributed storage of the second multi-source marine environmental data is carried out through MinIO.

[0085] In this embodiment, the MinIO distributed storage system is used to achieve efficient storage and management of massive marine environmental data (i.e., the second multi-source marine environmental data). Through the distributed architecture, the system can be horizontally expanded to support parallel storage and access of large-scale data, solving the scalability problem of traditional centralized storage.

[0086] In some embodiments, after intelligently sharing the stored multi-source ocean environment data through intelligent subscription distribution, the method further includes:

[0087] If new data is obtained or the data is updated, the data update is triggered through the event-driven mechanism;

[0088] Multi-source ocean environment data are updated synchronously through incremental synchronization.

[0089] In this embodiment, it is ensured that when new data is generated or existing data is updated, the synchronization process can be quickly responded and started. Through event-driven mechanisms (such as webhooks or message queue events), data synchronization tasks can be triggered in real time to ensure that the data in the system is synchronized with the source data. Incremental synchronization is one of the core technologies of this process. By comparing the version or timestamp of the data, the system can identify the data that has been added or updated since the last synchronization.

[0090] In some embodiments, the multi-source ocean environment data is synchronously updated through incremental synchronization in the following manner:

[0091] ΔB=B(t+Δt)-B(t)

[0092]

[0093] Where ΔB represents the incremental data, B(t+Δt) represents the amount of data at time t+Δt, B(t) represents the amount of data at the current time t, and V syncIndicates the total amount of synchronized data blocks, N indicates the number of data blocks to be synchronized, ΔB j Indicates the jth data block that needs to be synchronized.

[0094] In some embodiments, after synchronously updating the multi-source ocean environment data through incremental synchronization, the method further includes:

[0095] The consistency of the data to be updated synchronously is checked through the hash function.

[0096] In this embodiment, after data synchronization is completed, consistency checks are performed on the synchronized data regularly to ensure that data between the cloud and edge nodes remain consistent.

[0097] To facilitate understanding by those skilled in the art, a set of best embodiments is provided below:

[0098] The rapid growth of marine environmental data and technological progress have brought about an urgent need for efficient data sharing and management. Marine environmental data not only include historical data, but also include marine geological and surveying historical data, basic information products, thematic products, early warning products and other types of data. These data are of great value to many fields such as marine scientific research, resource development, environmental monitoring and disaster warning.

[0099] However, the diversity and massiveness of ocean data pose challenges to data storage, processing and sharing. Traditional data sharing technologies are difficult to meet the current needs for efficient management and rapid distribution of ocean data. Traditional technologies mainly rely on centralized storage and manual distribution systems. These systems usually use a single data center to store and manage massive data, and transmit data through protocols such as FTP and HTTP. Centralized storage systems usually use large database management systems (such as Oracle, MySQL) to manage data, and cooperate with file system storage (such as NFS, HDFS) to achieve centralized access to data. In addition, data distribution methods usually push data from the central node to the user end or edge node through manual or scheduled tasks, which lacks flexibility and real-time performance.

[0100] Therefore, it is necessary to develop multi-source massive marine environmental data intelligent sharing technology to achieve efficient data integration, secure transmission and convenient access. The core of the technical background lies in how to build an intelligent sharing platform that can adapt to the characteristics of marine data. This platform needs to be able to process and distribute multi-source data, support secure data exchange, and meet the needs of different users for data access and services. With the development of marine observation technology, the real-time and accuracy requirements of data are getting higher and higher, which requires the sharing platform to not only be able to respond quickly to user needs, but also to ensure the synchronous update and consistency of data. In addition, the marine data sharing platform also needs to be highly flexible and customizable to meet the needs of different users and different application scenarios. This includes providing a variety of service modes, such as data download, online data service, subscription push, customized processing and emergency services. At the same time, the platform also needs to support a variety of query methods, including by feature type, date, regional range and combination conditions, to improve the efficiency of user retrieval of data. Therefore, the technical background of multi-source massive marine environmental data intelligent sharing technology is multifaceted, involving multiple levels such as data storage, processing, sharing, security and application. These background factors have jointly promoted the development and innovation of marine data sharing technology to meet the growing demand for data resources in the marine field.

[0101] The current research on multi-source massive marine environmental data sharing technology faces many key issues and challenges. These issues are particularly prominent in the context of big data, which seriously affects the management, sharing and application efficiency of data. Combining the application of MinIO distributed storage technology, intelligent subscription distribution technology and data synchronization update technology, we can deeply explore the problems existing in the current research from the following aspects:

[0102] (1) The data sources are diverse and the formats are heterogeneous.

[0103] Marine environmental data comes from multiple sources, including satellite observations, buoy monitoring, unmanned ship measurements, and weather radars, which often use different storage formats and standards. For example, satellite data may exist in NetCDF or GeoTIFF format, while buoy data may be stored in time series form, resulting in serious data heterogeneity problems. Current research faces the challenge of how to standardize and uniformly manage these multi-source heterogeneous data. Existing solutions are inefficient in processing massive data and lack good support for complex data formats.

[0104] (2) Bottlenecks in massive data storage and management.

[0105] With the development of marine environmental monitoring technology, the amount of data has increased exponentially, and data storage and management have become core issues. The traditional centralized storage architecture is difficult to meet the current storage needs of marine environmental big data, has poor scalability, and often has delays and performance bottlenecks when accessing data. Especially when facing large-scale concurrent reading and writing of data, the processing capacity of the existing system is limited, which can easily cause system overload or slow response speed, and cannot effectively support the sharing and application of real-time data.

[0106] (3) Real-time data acquisition and distribution are inefficient.

[0107] Marine environmental monitoring requires the ability to obtain and analyze the latest ocean status data in real time, such as ocean temperature, salinity, wind speed, etc. However, existing data distribution systems often use scheduled updates or batch push methods, which cannot efficiently support the subscription and distribution of real-time data. In particular, in scenarios where multiple users request data at the same time, the system is prone to congestion, and the delay in data push is large, making it difficult to meet application scenarios that require real-time decision support.

[0108] (4) Data synchronization and consistency maintenance are difficult.

[0109] Sharing marine environmental data not only involves a large amount of distributed data stored in different locations, but also requires ensuring the synchronization and consistency of data between different nodes. Traditional data synchronization methods are inefficient, especially in terms of incremental data updates and data conflict handling. The lack of effective mechanisms makes it difficult to ensure the timeliness and accuracy of data. In the process of multi-source data synchronization, data consistency and integrity are often difficult to maintain effectively, especially in the case of network instability or node failure, the problems of data loss and asynchrony are more prominent.

[0110] (5) Technological gaps in edge computing and cloud-edge collaboration.

[0111] With the increase in the number of ocean data collection points and their wide distribution, massive amounts of data need to be pre-processed and stored locally to reduce the burden on the main data center. However, existing research on the application of edge computing and cloud-edge collaboration is relatively limited, and how to effectively use edge nodes for data processing and synchronization remains an urgent problem to be solved. The traditional centralized processing architecture is less efficient in this scenario and cannot fully utilize the advantages of edge computing to improve the timeliness and sharing efficiency of data processing.

[0112] (6) Lack of effective sharing and collaboration mechanisms.

[0113] In the study of multi-source massive marine environmental data sharing, different users and institutions often have different needs for data, which leads to poor flexibility and customizability of data sharing. The existing system lacks an intelligent subscription and distribution mechanism, making it difficult for users to flexibly subscribe to specific data content or topics according to their needs, limiting the efficient use of data. At the same time, cross-institutional collaboration and data sharing face compatibility issues such as data format and interface standards, which affect data flow and cooperation.

[0114] The technical solution of this embodiment can effectively deal with the problems existing in current research by combining MinIO distributed storage, intelligent subscription distribution technology and data synchronization update technology. The application of these technologies not only solves the heterogeneity and storage bottleneck problems of multi-source data, but also significantly improves the real-time and transmission efficiency of data sharing through efficient subscription distribution and edge computing support. At the same time, MinIO's data encryption and access control functions enhance the security of the data sharing process and provide comprehensive technical guarantees for the collaborative sharing of multi-source massive marine environmental data. The specific technical solutions of this embodiment include:

[0115] The multi-source massive marine environment data intelligent sharing technology (i.e., multi-source marine environment data intelligent sharing method) constructs a data sharing exchange directory and a sharing exchange scheduling strategy, provides sharing services after adapting and processing the marine environment application thematic products, and supports distributed data synchronization update services. It supports the use of data sharing exchange directories to formulate exchange scheduling strategies to achieve product identification, product configuration processing, and shared packaging and sorting. For multi-node and various computing and storage environments, it can realize product sharing services such as directory data browsing, data retrieval engine services, demand service processing, subscription distribution management, and content distribution services, and supports the exchange of marine environment application thematic products. In addition, it supports distributed data synchronization update services, which can realize functions such as data synchronization mechanism construction, synchronization monitoring, and synchronization update. The shared exchange information includes historical data of marine environment, historical data of marine geology and surveying and mapping, basic information product data, thematic product data, and early warning product data.

[0116] The multi-source massive marine environment data intelligent sharing technology mainly includes marine environment directory service, marine environment data sharing service and marine environment data synchronization update service, such as Figure 2 shown.

[0117] The processing flow of multi-source massive marine environment data intelligent sharing technology, such as Figure 3 The detailed process of multi-source massive ocean environment data intelligent sharing technology is as follows:

[0118] (1) Acquisition of multi-source massive marine environmental data (i.e., multi-source marine environmental data).

[0119] The sources of multi-source massive marine environment data and fishery operation data are marine environment databases in the prior art known to those skilled in the art, such as the Nation Oceanic and Atmospheric Administration (NOAA) environmental database, the Columbia environmental database, the Copernicus Marine Environment Monitoring Service (CMEMS) and the Western and Central Pacific Fisheries Commission (WCPFC). The types of marine environment data include chlorophyll concentration, sea surface height and sea surface temperature. The fishery operation data are South Pacific longline fishing data and fishery data provided by relevant company groups in various provinces and cities. The downloaded data formats include .csv, .xls and .nc. The data comes from high-resolution satellite remote sensing, which ensures the accuracy of data acquisition and provides a prerequisite for the quality of later experimental data.

[0120] There are many methods and ways to obtain multi-source ocean data, involving satellite remote sensing, buoy monitoring, ship observation, unmanned platforms, and ocean model simulation. Satellite remote sensing technology is one of the main means of obtaining large-scale, long-term continuous ocean data. Through multi-spectral and hyperspectral imaging equipment carried on meteorological and ocean observation satellites, various parameters such as ocean surface temperature, sea level, sea ice distribution, and ocean pigment concentration can be obtained. These satellites include the Fengyun series, ocean satellites, and Copernicus satellites, which can provide global ocean monitoring data.

[0121] The buoy monitoring network is another important source of data. Fixed or drifting buoys deployed in the ocean can collect real-time physical, chemical and biological data from the surface and deep layers of the ocean, such as temperature-salinity depth (CTD), dissolved oxygen and nutrient concentrations. Through international buoy networks, such as the ARGO program, these buoys can obtain continuous monitoring data of the global marine environment in a long-term and automated manner. In addition, ship observations along the coast and in the ocean are also an important way to obtain ocean data, especially in areas with dense shipping. Ships can obtain detailed local marine meteorology, waves, currents and other information through onboard observation equipment.

[0122] The application of unmanned platforms is also increasing, especially the use of drones, unmanned ships (USVs) and autonomous underwater vehicles (AUVs). These platforms are flexible and efficient, and can go deep into sea areas that are difficult for humans to reach or operate under extreme climatic conditions to obtain high-resolution regional marine environmental data. On the other hand, with the advancement of computer technology, ocean model simulation has become an important supplement to obtaining ocean data.

[0123] By combining meteorological data, historical ocean observation data and numerical models, future trends in ocean environmental changes, such as the impact of ocean circulation, tides and climate change on the ocean, are simulated.

[0124] International ocean data exchange and cooperation is also an important way. By participating in global or regional ocean data sharing platforms and programs, such as the Global Ocean Observing System (GOOS) and the ocean data sharing program of the Intergovernmental Oceanographic Commission (IOC) under UNESCO, abundant international ocean monitoring data can be obtained. These data provide basic support for weather forecasts, environmental assessments, and marine resource development.

[0125] Due to the large amount of data and the long years involved, it is time-consuming and meaningless to download them manually one by one. Therefore, this embodiment writes a simple data acquisition crawler code to automatically download the required data and store it in the local database. The data acquisition method can obtain various environmental data information from the official website, including the data acquisition source, data query format (grid or table), the time year and time interval of the data. The time unit is in the form of day, week or month.

[0126] (2) Product adaptation to the marine environment.

[0127] According to specific research needs, adaptation and structured data generation and packaging are carried out from the acquired multi-source ocean data and products, which is mainly divided into two stages: product adaptation processing and data packaging.

[0128] In the product adaptation processing stage, format adaptation, feature adaptation and regional adaptation are required. Since the acquired data comes from different data sources, such as satellite observations, buoy monitoring and ship observations, the formats are also different, such as HDF, NetCDF and GRIB. In order to meet the needs of users, data products need to be converted into user-specified formats, such as CSV, JSON or NetCDF. This process is performed by using data conversion tools such as Pandas and NumPy for format conversion and data preprocessing. Feature adaptation is performed according to the specific needs of users, and key marine environmental elements such as temperature, salinity and current velocity are selected from the data products and adapted. Feature adaptation not only includes screening, but may also involve unit conversion (such as conversion between Celsius and Fahrenheit), data aggregation (such as calculation of average values ​​in time or space), and statistical calculations (such as extreme values, coefficient of variation, etc.). Through these operations, it is ensured that users can obtain the key elements they care about and that the data meets their usage standards. Regional adaptation is also required. Users may only be interested in a specific geographic area, so the data needs to be cropped or resampled to the specified area. This can be done by using geospatial processing tools, such as GDAL, to clip global or large-scale data in data products to the area required by the user. For data involving different spatial resolutions, resampling may also be required to ensure that data from different sources can be effectively fused.

[0129] After completing the data adaptation process, the first multi-source marine environment data is obtained, and then the data packaging and sorting stage is entered. Data packaging is to facilitate data storage and transmission. The data products after data adaptation processing are packaged into a single file or a set of files to obtain the second multi-source marine environment data. Specifically, compression tools such as ZIP, TAR.GZ, etc. can be used to reduce the data volume and improve transmission efficiency. Especially when transmitting large amounts or large-scale data, compression packaging can significantly reduce the occupancy of network bandwidth. In order to ensure that the data has good traceability during transmission and use, it is necessary to generate corresponding metadata files for the packaged data. Metadata records the source of the data, processing steps, format conversion process, detailed information on adaptation operations, etc., to ensure that users can clearly understand the generation and processing process of the data after receiving it, and can correctly use and interpret the data.

[0130] (3)MinIO distributed storage.

[0131] MinIO is a high-performance distributed object storage system that is compatible with the Amazon S3 API and is particularly suitable for processing massive unstructured data such as marine environmental data. In the multi-source massive marine environmental data processing scenario, MinIO provides a reliable, efficient and flexible distributed storage architecture, data management method and security guarantee to meet the storage and access needs of massive data.

[0132] MinIO's distributed storage architecture design is suitable for the distributed storage and high concurrent access requirements of marine data. The MinIO cluster consists of multiple independent storage nodes, each node runs a MinIO instance, and all nodes are connected through the network to form a complete cluster architecture. Each node can process data read and write operations at the same time, so that the system can achieve horizontal expansion by adding nodes, so that it can still maintain high availability and high performance in the face of the growing marine environmental data. MinIO uses erasure coding technology to shard data and store it redundantly on multiple nodes. In this way, even if some nodes fail, the system can still recover data through the remaining data blocks and redundant blocks to ensure data integrity and availability. For example, an ocean observation data object can be divided into multiple data blocks and redundant blocks and stored on different nodes. Even if some nodes fail, the data can still be reconstructed through redundant data on other nodes. This design not only ensures high data availability, but also greatly improves the disaster recovery capability and stability of the system.

[0133] In terms of data storage and access, MinIO provides an API compatible with Amazon S3, so users can manage and access massive amounts of marine environmental data through standard S3 operations. By being compatible with S3 operations such as PUT, GET, and DELETE, users can easily integrate MinIO into existing S3 clients and tools to conveniently store and read marine data. MinIO uses an object storage model, which enables each marine data object to be stored in the form of an object. The data object contains data content and related metadata, such as data identifiers, data types, timestamps, and geographic locations. This storage method is particularly suitable for multi-source marine data, because each data object can flexibly store different data information to meet the management needs of diverse and complex data structures.

[0134] MinIO has excellent data management and expansion capabilities, which makes it particularly outstanding when processing ocean data. Through the consistent hashing algorithm, MinIO automatically distributes data evenly to each node in the cluster, thereby achieving load balancing and avoiding the performance bottleneck of a single node. Assume that a hash function H is used to map nodes and data to the hash space. For the node set N = {n1,n2,...,n k} and data set D = {d1, d2, ..., d m}, the specific steps of the consistent hashing algorithm are as follows:

[0135] The first step is to construct a hash ring. The node identifier (such as the node's IP address) is converted into a hash value through a hash function, and the hash value is placed at a certain position in the hash ring. Similarly, the identifier of the data object (such as the key of the data) is also mapped to the hash ring through the same hash function. The node hash value conversion formula is:

[0136] H(n i )=hash(n i )

[0137] Where H(n i ) is node n i The hash value of .

[0138] The data object hash value conversion formula is:

[0139] H(d j )=hash(d j )

[0140] Where H(d j ) is a data object, d j The hash value of .

[0141] The second step is the data mapping rule. Each data object is assigned to the first node that is greater than or equal to its hash value by searching clockwise. This means that the data object is stored on the node closest to it.

[0142] The third step is to redistribute data after the node changes. When a new node joins, only the data in front of the new node needs to be redistributed to the node, and other data will not be affected. Similarly, when a node leaves, only the part of the data that the node is responsible for needs to be redistributed to the next node. j , find the first node n in the clockwise direction i , so that:

[0143] H(n i )≥H(d j )

[0144] If no H(d j ) nodes, they are assigned to the first node on the ring. When new nodes join the cluster or old nodes go offline, the system automatically adjusts the data distribution to maintain the overall load balance and performance optimization of the cluster. This automatic load balancing capability is particularly critical for storing data in dynamically changing marine environments. By simply adding nodes, MinIO can easily expand storage capacity and throughput, which is especially suitable for deployment requirements in marine data centers. MinIO supports deployment across multiple data centers, which provides strong guarantees for geographic redundancy and disaster recovery, ensuring secure storage and timely access to marine data worldwide.

[0145] In terms of data security, MinIO also provides a comprehensive solution to cope with the sensitivity and high security requirements of marine data. MinIO supports TLS / SSL encryption protocols to ensure that data is not tampered with or leaked during transmission. When storing static data, MinIO provides server-side encryption functions, supports strong encryption algorithms such as AES-256, and implements secure key management through external key management services (such as HashiCorp Vault). This means that even if the storage device itself is physically damaged or stolen, the data remains encrypted, ensuring the security of the data. In addition, MinIO integrates a role-based access control mechanism (RBAC), and users can control access rights to data through policy files, thereby effectively managing the permission relationship between users and systems and ensuring data compliance and security.

[0146] MinIO has significant advantages in the storage and management of multi-source massive marine environmental data. Its distributed architecture ensures the high availability, scalability and disaster recovery capabilities of the system. The S3-compatible API and object storage model provide flexible data management methods, while the powerful load balancing and data security mechanisms ensure the stability of the system and the security of the data. These features make MinIO an ideal choice for processing and storing massive marine data, especially in complex and dynamic marine environmental data application scenarios, providing efficient and reliable support.

[0147] (4) Intelligent subscription distribution technology and data synchronization updates.

[0148] After MinIO distributed storage of multi-dimensional massive ocean data, it is necessary to call, distribute and synchronize data according to the user's instructions and needs. This embodiment integrates intelligent subscription distribution technology and data synchronization update technology into the entire multi-source massive ocean environment data intelligent sharing technology system, facing multi-source massive ocean environment data, optimizing the user's data demand experience, and conveniently and quickly sharing massive ocean data intelligently.

[0149] In the distributed object storage system MinIO, data products are extracted through an Amazon S3-compatible API, and the system reads the ocean environment data objects requested by users from different nodes. At the same time, to ensure the integrity and consistency of the data, MinIO combines consistent hashing and erasure coding technology to perform data consistency checks. Once a data block is found to be lost or damaged, the system will automatically start the data repair mechanism to ensure the reliability and integrity of the data. This is crucial for the accurate analysis of marine environmental data and can ensure that scientists and users obtain accurate marine observation data.

[0150] Assume that the user request contains Q conditions, each condition can be expressed as qi (such as time range, geographical area, data type, etc.). Each data record in the database D j Has metadata m corresponding to the condition ij . The matching function f(q i ,m ij ) is used to determine data record D j Whether the user condition q is met i .

[0151] Data Record D j The matching degree with the user request M j It can be expressed as:

[0152]

[0153] Among them, w i is the importance weight of each condition, usually satisfied The final returned list C is all the names that satisfy M j >T data record set, T is the set threshold: C = {D j |M j >T}.

[0154] In data retrieval, the matching function f(q i ,m ij ) is used to determine data record D j Whether the user condition q is met i The specific matching function may vary based on different data types and retrieval requirements, and this embodiment does not impose any specific restrictions. Two examples are given as follows:

[0155] (1) Interval matching function - suitable for matching time range, value range, etc.:

[0156]

[0157] For example: if the user queries the time range "2024-01-01 to 2024-01-23" and the data record time is 2024-01-08, the matching function returns 1, otherwise it returns 0. ij is within the range q i Indicates that if the metadata m ij Under the condition q i Otherwise means otherwise, that is, metadata m ij Not in condition q i The situation inside.

[0158] (2) Boolean matching function - suitable for structured data:

[0159]

[0160] For example: if the user queries "Data Type = Temperature" and the type field of the data record is "Temperature", the matching function returns 1, otherwise it returns 0. i is contained in m ij Indicates the condition q i Contains metadata m ij .

[0161] In data distribution management, the subscription mechanism enables users to flexibly subscribe to specific ocean data products or topics, such as "real-time ocean temperature" or "ocean salinity changes." By recording the user's subscription information and setting the time interval or trigger condition for automatic push according to the user's needs. When new data is generated, the data is automatically pushed to the subscriber through the event trigger mechanism. In this process, the message queue system (such as Apache Kafka) is used to manage data distribution to ensure that users can receive the latest data in a timely manner. Data push is carried out through persistent connection technologies such as WebSocket or HTTP / 2, ensuring real-time and high efficiency.

[0162] The content distribution strategy further optimizes the efficiency of data transmission. The optimal distribution path is selected based on the user's geographical location, network conditions and data requirements. For massive marine environmental data, the data can be cached to edge nodes through CDN (content distribution network) accelerators, thereby shortening the data transmission distance and improving user access speed. This not only improves responsiveness, but also provides better services for users who need to update data in real time. For real-time data, real-time push can be achieved through persistent connections, and for large-scale historical data, batch data transmission is provided to achieve more efficient transmission through HTTP download or SFTP.

[0163] The user subscribes to the marine environment data product S. The new data product generated by the data publisher is D new , D new It can be through data record D j The matching degree with the user request M j The obtained marine environment data set (i.e., the target marine environment data set), where D new The relevance to the topic S subscribed by the user can be expressed as P new :

[0164]

[0165] Among them, s k represents the correlation function, α k represents the corresponding weight. new >T subscribe (Tsubscribe is the distribution threshold), the data will be pushed to the user.

[0166] Correlation coefficient k It is used to calculate the new data product D new The degree of matching between the user's subscribed topic S. The specific correlation calculation method may vary based on different data types and retrieval requirements, and this embodiment does not make specific restrictions. For example, the following method may be used:

[0167] (1) Boolean correlation: used to determine whether a data product meets the subscription topic.

[0168]

[0169] Calculation process: Check data product D new Whether the conditions for subscribing to topic S are met. For example, if S is "data type = temperature" and D new If the data type of is also "temperature", the correlation score is 1, otherwise it is 0.

[0170] (2) TF-IDF: used to calculate the correlation between data products and subscription topics, suitable for text data matching.

[0171] TF: term in data product D new The frequency of occurrence in .

[0172]

[0173] IDF: Inverse document frequency of a term in all data records.

[0174]

[0175] TF-IDF: Calculated in data product D new The relevance of the subscription topic S.

[0176] s k (D new ,S)=TF(q i ,D new )×IDF(q i )

[0177] Calculation process: First, the subscription topic and data product are textualized, and then the TF-IDF weight is calculated to represent the term q i Importance in data products and subscription topics. TF-IDF scores of multiple terms are weighted to give an overall relevance.

[0178] (3) Cosine similarity: used in vector space models, it evaluates relevance by calculating the cosine value of the angle between the query condition and the data record vector.

[0179]

[0180] Among them: · represents the dot product of the vector, ‖·‖ represents the norm (i.e., length) of the vector.

[0181] Calculation process: Data product D new and subscription topic S are converted into vector representations, and the cosine similarity between them is calculated. The value ranges from -1 to 1. The closer the value is to 1, the higher the correlation is.

[0182] Comprehensive relevance score: In subscription management, the relevance score P new It is calculated by weighted average.

[0183] Amount of data pushed in real time V realtime The user's network bandwidth B limit should be met:

[0184] V realtime ≤B·t

[0185] Here, t represents the time interval.

[0186] The amount of data distributed in batches V batch Can be divided into m blocks, each block is of size v i , the total amount is:

[0187]

[0188] Data security is also critical in marine environmental data management. MinIO ensures the security of data transmission through TLS / SSL protocol, and uses symmetric encryption algorithms (such as AES-256) to encrypt static data. Keys are securely transmitted through asymmetric encryption algorithms (such as RSA) to ensure protection from external attacks during data transmission and storage. In addition, this embodiment uses a role-based access control (RBAC) model to limit user access rights to data through access policies. Each data request undergoes strict credential verification to ensure that only authorized users can access specific data.

[0189] Cloud-edge data exchange technology greatly improves the processing efficiency of multi-source ocean data. In areas where ocean data demand is concentrated, the system using the method of this embodiment (i.e., a multi-source ocean environment data intelligent sharing system, a multi-source ocean environment data intelligent sharing system corresponding to a multi-source ocean environment data intelligent sharing method setting, which is not described in detail in this embodiment) can deploy edge computing nodes to share the load of the cloud main data center. These edge nodes can not only process and store part of the data, but also provide localized data access services, thereby reducing the pressure on the main data center. The flexibility of MinIO's distributed architecture allows data to be distributed on different edge nodes according to demand, ensuring that users can quickly access the required ocean environment data. At the same time, incremental synchronization technology is used to synchronize the changed data of the edge node back to the cloud main database to ensure the consistency of data between the cloud and the edge node. MinIO realizes real-time synchronization of edge node and cloud data by automatically handling data conflicts and consistency checks.

[0190] During the data synchronization and update process, the system monitors each data source in real time to ensure that when new data is generated or existing data is updated, it can respond quickly and start the synchronization process. Through event-driven mechanisms (such as webhooks or message queue events), data synchronization tasks can be triggered in real time to ensure that the data in the system is synchronized with the source data. Incremental synchronization is one of the core technologies of this process. By comparing the version or timestamp of the data, the system can identify the new or updated data since the last synchronization. Using MinIO's object version control function, the system can obtain incremental data and update it to the storage node, while using erasure coding technology to ensure high availability and fault tolerance of data during the synchronization process.

[0191] Assume that the data version before time t is B(t), and the data version at time t+Δt is B(t+Δt). The incremental data ΔB is:

[0192] ΔB=B(t+Δt)-B(t)

[0193] The total amount of synchronized data blocks V sync for:

[0194] Wherein, N represents the number of data blocks that need to be synchronized.

[0195] After data synchronization is completed, this embodiment will also periodically perform consistency checks on the synchronized data to ensure that the data between the cloud and edge nodes is consistent. Use the hash function H(x) to check data consistency. Assume that data block B j The hash value of B j ):

[0196] H(B j)=H(source(B j ))=H(destination(B j ))

[0197] Among them, source and destination represent the source node and destination node of the data respectively. The source node refers to the original storage location of the data or the initial source of the data. In the data consistency check, the source node is responsible for generating the initial hash value of the data block. The destination node refers to the storage target location of the data, which may be the location where the data is synchronized or copied. In the data consistency check, the destination node is responsible for storing the data blocks received from the source node and calculating the hash value of the data block to verify the consistency.

[0198] If data inconsistency is found, this embodiment will automatically start the data repair process. In addition, this embodiment will record detailed logs for each data synchronization, including synchronization time, data source, amount of synchronized data, and results. These logs can not only be used for subsequent audits, but also provide strong support for troubleshooting. At the same time, this embodiment sends a synchronization status report to the administrator via email or other message channels, informing the administrator of the success or failure of the synchronization, and attaching detailed log information.

[0199] Subscription distribution technology and data synchronization update play a key role in the management of multi-source massive marine environmental data. Through data extraction, consistency verification, content distribution and secure transmission, efficient data acquisition and distribution are ensured. At the same time, the combination of cloud-edge data exchange and real-time synchronization update technology further enhances the flexibility and scalability of intelligent sharing of multi-source marine environmental data, meeting the needs of massive marine data management and distribution.

[0200] This embodiment achieves efficient sharing and synchronous updating of multi-source massive marine environmental data by adopting MinIO distributed storage and subscription distribution technology. Compared with traditional centralized storage systems, this embodiment can better cope with the storage needs of massive data, while improving the reliability and speed of data access through a distributed architecture. The data adaptation and subscription distribution mechanism enables users to quickly obtain customized data products to meet personalized needs. In addition, the combination of edge computing and cloud synchronization technology significantly improves the efficiency and security of data transmission, ensures the consistency and integrity of data between different nodes, and provides a more intelligent and flexible marine data sharing solution. The beneficial effects of this embodiment specifically include:

[0201] (1) Distributed storage architecture based on MinIO: This embodiment uses the MinIO distributed storage system to achieve efficient storage and management of massive marine environmental data. Through the distributed architecture, the multi-source marine environmental data intelligent sharing method can be horizontally expanded to support parallel storage and access of large-scale data, solving the scalability problem of traditional centralized storage.

[0202] (2) Intelligent data adaptation mechanism: The multi-source marine environment data intelligent sharing method uses intelligent retrieval algorithms to automatically filter and adapt data according to user needs and provide customized marine environment data products. It includes functions such as data format conversion, feature screening, and geographic area clipping, allowing users to quickly obtain data that meets specific requirements.

[0203] (3) Data subscription and distribution management: This embodiment introduces a subscription-based distribution mechanism, where users can subscribe to data products of a specific type or theme. The multi-source marine environment data intelligent sharing method automatically manages data push and distribution based on subscription information, supports real-time data push and regular batch distribution, and significantly improves the real-time and flexibility of data acquisition.

[0204] (4) Automated and intelligent data management: The multi-source marine environment data intelligent sharing method integrates a variety of intelligent data management functions, such as automated data collation, packaging, and metadata generation, which reduces manual intervention and improves the efficiency and accuracy of data management. At the same time, the multi-source marine environment data intelligent sharing method supports personalized data recommendations based on user historical behavior to optimize user experience.

[0205] In summary, compared with the existing technology, the technical solution of this embodiment realizes the efficient storage, intelligent retrieval and personalized distribution of massive marine environmental data by adopting MinIO distributed storage and subscription distribution mechanism, significantly improving the speed and real-time performance of data access. At the same time, through cloud-edge collaboration and secure encrypted transmission, it ensures the efficient synchronization and security of data between different nodes, solves the shortcomings of traditional centralized storage with poor scalability, slow data transmission and lack of intelligent management, and provides users with a more flexible, reliable and secure data sharing solution.

[0206] The embodiments of the present application are described in detail above in conjunction with the accompanying drawings, but the present application is not limited to the above embodiments. Various changes can be made within the knowledge scope of ordinary technicians in the relevant technical field without departing from the purpose of the present application.

Claims

1. A multi-source marine environment data intelligent sharing method, characterized in that: The method comprises: Obtain marine environmental data from multiple sources; Performing product adaptation on the multi-source ocean environment data to obtain first multi-source ocean environment data, wherein the product adaptation is used to indicate multi-source ocean environment data that meets user needs; Packaging the first multi-source ocean environment data to obtain second multi-source ocean environment data; Distributed storage is performed on the second multi-source ocean environment data to obtain stored multi-source ocean environment data; The stored multi-source ocean environment data is intelligently shared through intelligent subscription distribution.

2. The multi-source marine environment data intelligent sharing method according to claim 1 is characterized in that: The intelligent sharing of the stored multi-source ocean environment data through intelligent subscription distribution includes: Obtaining a user request including multiple conditions, and obtaining metadata corresponding to each data record and condition in the stored multi-source marine environment data; Based on the metadata, calculating the matching degree between each data record and each condition in the user request; Determining, according to the matching degree, a marine environment subscription dataset that meets multiple conditions in the user request; The marine environment subscription dataset is intelligently shared through a content distribution strategy.

3. The multi-source marine environment data intelligent sharing method according to claim 2 is characterized in that: The matching degree between each data record and each condition in the user request is calculated based on the metadata in the following manner: Among them, M j represents the matching degree, Q represents the total number of conditions in the user request, and w i represents the importance weight of each condition, q i represents the i-th condition, m ik represents metadata, and f(·) represents the matching function.

4. The multi-source ocean environment data intelligent sharing method according to claim 2 is characterized in that: The intelligent sharing of the marine environment subscription data set by using a content distribution strategy includes: Get the product topics subscribed by the user; Calculating the correlation between the product theme and the marine environment subscription data set to obtain a correlation calculation result; Determining a target push data set according to the correlation calculation result; The target data set is pushed for intelligent data sharing.

5. The multi-source ocean environment data intelligent sharing method according to claim 4 is characterized in that: The intelligent data sharing of the target pushed data set includes: The target pushed data set is intelligently shared in real-time push mode; or, the target pushed data set is intelligently shared in batch distribution mode.

6. The multi-source ocean environment data intelligent sharing method according to claim 4 is characterized in that: The correlation between the product topic and the marine environment subscription dataset is calculated as follows: Among them, R new represents the correlation, n represents the total amount of data in the marine environment subscription dataset, α k Represents the corresponding weight, s k (·) represents the correlation function, D new represents the marine environment subscription dataset, and S represents the product theme.

7. The multi-source ocean environment data intelligent sharing method according to claim 1 is characterized in that: The distributed storage of the second multi-source ocean environment data includes: The second multi-source marine environment data is distributedly stored through MinIO.

8. The multi-source ocean environment data intelligent sharing method according to claim 1 is characterized in that: After intelligently sharing the stored multi-source ocean environment data through intelligent subscription distribution, the method further includes: If new data is obtained or the data is updated, the data update is triggered through the event-driven mechanism; The multi-source ocean environment data is synchronously updated through incremental synchronization.

9. The multi-source ocean environment data intelligent sharing method according to claim 8, characterized in that: The multi-source ocean environment data is synchronously updated through incremental synchronization in the following manner: ΔB=B(t+Δt)-B(t) Where ΔB represents the incremental data, B(t+Δt) represents the amount of data at time t+Δt, B(t) represents the amount of data at the current time t, and V sync Indicates the total amount of synchronized data blocks, N indicates the number of data blocks to be synchronized, ΔB j Indicates the jth data block that needs to be synchronized.

10. The multi-source ocean environment data intelligent sharing method according to claim 8, characterized in that: After synchronously updating the multi-source ocean environment data through incremental synchronization, the method further includes: The consistency of the data to be updated synchronously is checked through the hash function.

Images