Data processing method and device

A data processing and data table technology, applied in the computer field, can solve problems such as easy omission, lack of versatility, and low efficiency, so as to improve the efficiency of merging and avoid low efficiency of system operation

Pending Publication Date: 2021-12-10
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD +1
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] For larger clusters, when there are a large number of empty partitions or partitions with too small capacity, the method of actively searching and remerging is too inefficient, easy to miss, and lacks versatility

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and device
  • Data processing method and device
  • Data processing method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0069] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0070] figure 1 is a schematic diagram of the main flow of a data processing method according to the first embodiment of the present invention, such as figure 1 shown, including:

[0071] Step S101, when it is monitored that the trigger condition of the merging task is met, the configuration file is obtained;

[0072] Step S102, according to the configuration file, dete...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data processing method and device, and relates to the technical field of computers. A specific embodiment of the method comprises the steps of obtaining a configuration file when it is monitored that a trigger condition of a merging task is met; determining a target data table in the cluster according to the configuration file, and selecting a to-be-processed data partition from the target data table according to a threshold condition in the configuration file; calling a merging interface to obtain a merging method, and merging the to-be-processed data partitions based on a partition target capacity threshold in the configuration file to generate a merged data partition; and calling a load balancing interface, obtaining an allocation strategy according to the merged data partitions, and further allocating the merged data partitions to each device. According to the embodiment, a universal method is provided for screening out the data partitions needing to be merged, the interface is called to complete merging and partition distribution after merging, the merging efficiency of the data partitions is improved, and then the situation that the system running efficiency is low due to the fact that the system gauge pressure is too large is avoided.

Description

technical field [0001] The present invention relates to the field of computer technology, in particular to a data processing method and device. Background technique [0002] With the continuous expansion of the cluster scale and the number of users, the way of business usage is becoming more and more complex, and the application of the HBase framework is becoming more and more common; the number of data partitions in the data table is also increasing; due to improper use by users or normal Businesses often generate a large number of empty partitions or partitions with too small capacity, resulting in excessive pressure on the system table, which in turn leads to low system operation efficiency. In the prior art, developers often actively search for these empty partitions or partitions with too small capacity, and then issue merge instructions to merge them. [0003] In the course of realizing the present invention, the inventor finds that there are at least the following pr...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50
CPCG06F9/5077G06F9/5083
Inventor 郑思城吴怡燃
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap