Mass sea condition data aggregation processing method and device based on streaming framework

By introducing a massive sea condition data aggregation processing method based on a streaming framework in marine data processing, the problem of insufficient flexibility in mass sea condition data processing in the existing technology is solved, efficient and flexible data processing and quality assurance are achieved, and reliable data support is provided for marine research.

CN119719195AInactive Publication Date: 2025-03-28NINGBO YUYAO TECH CO LTD
9 Cites 0 Cited by

Patent Information

Application Number
CN202411670831.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-11-21
Publication Date
2025-03-28
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

When facing massive sea conditions data, the existing technology has poor flexibility and cannot process data processes efficiently, resulting in low data processing efficiency, difficult to guarantee data quality, and timely providing accurate and effective data support for marine-related research, decision-making, etc.

Method used

A large amount of sea condition data aggregation processing method based on a streaming framework is designed. By building a data transfer warehouse, using data perceptron for file detection and layered storage, and standardized data processing, the flexible and efficient processing of sea condition data is realized.

Benefits of technology

It improves data processing efficiency and data quality, ensures the orderliness and accessibility of data, and provides accurate and effective data support for ocean-related research, decision-making, etc.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

The invention provides a mass sea condition data aggregation processing method and device based on a streaming framework, and relates to the technical field of ocean data processing. The method comprises the following steps: acquiring sea condition data to be processed from an external data source according to a set rule, constructing a data transfer warehouse based on a streaming framework principle, naming the data according to a set format, storing the data in a temporary area of the data transfer warehouse, detecting newly added data and state information by utilizing a sensor, and storing the newly added data and the state information according to a hierarchical and state corresponding relation. And finally, processing according to a layering process to obtain standard sea condition data. According to the method, a meta-information rule highly compatible with massive sea condition data is designed, a complex multi-step flow is defined through a declarative method for abstracting and standardizing the data, a repeatable data pipeline with a retry strategy is arranged and constructed, sea condition data downloading and flow modification of a newly added sea condition data processing mode are simplified, and the data processing efficiency is improved. The flexible processing requirement of the sea condition data is effectively met, and the data processing efficiency and quality are improved.
Need to check novelty before this filing date? Find Prior Art