Method, device and system for data processing

A data processing and data technology, applied in the information field, can solve problems such as occupying system memory, achieve the effect of shrinking system memory and improving concurrent throughput

Active Publication Date: 2019-02-12
BEIJING QIHOO TECH CO LTD
View PDF3 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] In view of the above problems, the present invention provides a data processing method, device and system, which can solve the problem that a large amount of system memory is occupied by retrieval requests

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Method, device and system for data processing
  • Method, device and system for data processing
  • Method, device and system for data processing

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0028] Exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be embodied in various forms and should not be limited by the embodiments set forth herein. Rather, these embodiments are provided for more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0029] In order to save memory resources of the Cassandra storage system so that the Cassandra system can simultaneously support more retrieval requests, the embodiment of the present invention provides a data processing method. Such as figure 1 As shown, the method includes:

[0030] 101. Group the file sets involved in the retrieval request to obtain multiple file subsets.

[0031] Any node in the Cassandra storage system can b...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method, device and system, relates to the information technical field and solves the problem that a retrieval request occupies massive system memory. The method of the invention comprises following steps: dividing a file set related to the retrieval request into groups so as to obtain multiple file subsets; allocating a cache to a first file subset so as to read the data in the first file subset; after finishing reading the data in the first file subset, releasing the cache of the first file subset, allocating the cache to a next file subset so as to read the data in the next file subset; after reading the data of all the file subsets, combining the data of all the file subsets to obtain the user data returned to a client. The method, the device and the system of the invention are mainly applied in a data retrieval process based on a distributed storage system.

Description

technical field [0001] The present invention relates to the field of information technology, in particular to a data processing method, device and system. Background technique [0002] Cassandra is a storage system without a central node, which distributes data equally to different nodes according to the Hash algorithm. Compared with the traditional centralized storage system, the distributed storage system can improve the problem of limited system performance caused by centralized data storage, improve the efficiency of data storage, data query and data processing, and is more suitable for large-scale data storage scenarios. [0003] MemTable is a space of a certain size allocated in the distributed node memory to store data written by users. When a user writes data to a node, the written data will be directly appended to the MemTable in the node memory. When the MemTable data is full, the node will dump all the data in the MemTable to the disk to form a Sorted String Tab...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/182G06F16/172G06F16/2455
Inventor 王锋
Owner BEIJING QIHOO TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap