Data storage and reading method and device

A data storage and data technology, applied in the field of big data, can solve the problems of JVM business processing efficiency reduction and other problems

Pending Publication Date: 2020-10-02
HANGZHOU HIKVISION DIGITAL TECH
View PDF6 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] However, since the JVM will suspend the processing of the current business during the GC process, when Spark frequently initiates GC, it

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage and reading method and device
  • Data storage and reading method and device
  • Data storage and reading method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0091] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Apparently, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts all belong to the protection scope of the present invention.

[0092] In related technologies, when data is processed based on the Spark computing engine, the data is usually stored in the heap memory of the JVM. Among them, in order to avoid storing a large amount of data in the heap memory for a long time, the Spark computing engine can frequently initiate GC to clean up the data stored in the heap memory to save the storage space of the heap memory. However, since the JVM will suspend the processing of...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the invention relates to the technical field of big data, and provides a data storage and reading method and device. The data storage method comprises the steps of obtaining to-be-stored data; determining a corresponding relationship between each storage unit corresponding to a preset storage format and each sub-data in the to-be-stored data; and storing the to-be-stored data into an out-of-heap memory of the Spark calculation engine by calling a preset data interface according to the corresponding relationship. The data reading method comprises the steps of determining a filtering condition of data required by a to-be-processed service when the to-be-processed service is received; by calling a preset data interface, reading target data meeting a filtering condition froman out-of-heap memory of the Spark calculation engine; and storing the target data into an in-heap memory of a Spark calculation engine, so as to read the target data from the in-heap memory when theto-be-processed service is processed. Compared with the prior art, by applying the scheme provided by the embodiment of the invention, the service processing efficiency of the JVM can be improved.

Description

technical field [0001] The present invention relates to the technical field of big data, in particular to a data storage and reading method and device. Background technique [0002] JVM (Java Virtual Machine, Java Virtual Machine) is a new network computing platform, which is realized by simulating various computer functions on a physical computer. Usually, the JVM uses the Spark computing engine to manage the data involved in the various businesses it processes. And, when the JVM is applied in a data processing system, the Spark computing engine used by the JVM can be included in the data processing system at the same time. [0003] Among them, the Spark computing engine is a fast and general computing engine that supports distributed data sets and can be used for large-scale data processing. It can complete text processing, machine learning, SQL (Structured Query Language, Structured Query Language) queries, etc. Various data processing tasks. [0004] In related techno...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F3/06
CPCG06F3/061G06F3/0608G06F3/0664
Inventor 王雪磊刘中军
Owner HANGZHOU HIKVISION DIGITAL TECH
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap