Big data-oriented data stream type caching method and system

A data streaming and caching system technology, applied in the field of big data processing, can solve the problems of lack of simple and effective caching solutions, and achieve the effect of improving operational efficiency and reliability, and improving utilization

Pending Publication Date: 2021-12-17
上海健交科技服务有限责任公司
View PDF6 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

This method can only solve the demand for point-like data access and cache, and lacks a simple and effective cache solution for streaming data formed based on time series, which is more common in big data

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Big data-oriented data stream type caching method and system
  • Big data-oriented data stream type caching method and system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0022] Below in conjunction with specific embodiment, further illustrate the present invention. It should be understood that these examples are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the teachings of the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms also fall within the scope defined by the appended claims of the present application.

[0023] The first embodiment of the present invention relates to a big data-oriented data stream caching method, such as figure 1 shown, including the following steps:

[0024] In step (1), when the data stream arrives at the system, the time stamp marking of each item of record information in the data stream is completed in real time to form a time series index.

[0025] Step (2), through the analysis of the flow velocity of ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to a big data-oriented data stream type caching method and system. The method comprises the following steps: completing timestamp marking of each item of record information in a data stream arriving at a system to form a time sequence index; selecting the minimum time limited by a pre-configured basic data volume, and performing slicing based on the time sequence index to complete automatic fragmentation processing of a data stream time section to obtain sub-data streams of a plurality of fragments; performing determinant conversion on the data information in the sub-data stream of each fragment according to a pre-configured cacheable data column, completing real-time compression of the same data column, and forming a cache data block; and allocating idle data stream cache resources in a pre-configured data stream cache resource pool according to the size of the cache data block, and writing the cache data block into the data stream cache resources. According to the invention, a high-performance cache mechanism for data access can be provided under the condition of ensuring that the original stream sequence is not changed.

Description

technical field [0001] The invention relates to the technical field of big data processing, in particular to a big data-oriented data streaming caching method and system. Background technique [0002] The existing big data caching method is usually based on the KV form of key-value pairs, which mainly solves the need to quickly read or update the value when the key key is known. This method can only solve the demand for point-like data access and cache, and lacks a simple and effective cache solution for streaming data formed based on time series, which is more common in big data. Contents of the invention [0003] The technical problem to be solved by the present invention is to provide a big data-oriented data stream cache method and system, which can provide a high-performance cache mechanism for data access while ensuring that the original stream sequence remains unchanged. [0004] The technical solution adopted by the present invention to solve the technical problem...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/23G06F16/2455G06F16/2458
CPCG06F16/221G06F16/23G06F16/24552G06F16/24568G06F16/2474
Inventor 不公告发明人
Owner 上海健交科技服务有限责任公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap