Data writing optimization method and terminal

A technology for data writing and optimization methods, applied in the field of distributed databases, can solve problems such as request delays, increased time-consuming write requests, etc., and achieve the effect of achieving response speed and improving processing efficiency

Active Publication Date: 2021-08-24
FUJIAN TIANQUAN EDUCATION TECH LTD
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

As long as there is a slow response when a node writes data, it will lead to an increase in the time spent on this write request, which will cause a delay in the request

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data writing optimization method and terminal
  • Data writing optimization method and terminal

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0061] Please refer to figure 1 , Embodiment 1 of the present invention is:

[0062] An optimization method for data writing, comprising the steps of:

[0063] S1. When a node in the node set receives multiple write data packets corresponding to one piece of write data, monitor the response time of each write data packet;

[0064] S2. Mark the first written data packet whose response time exceeds a preset first threshold, and mark the node whose number of the first written data packet exceeds a preset second threshold as a first node;

[0065] S3. Write the first write data corresponding to the first node into the second node in the node set, specifically:

[0066] Obtaining unwritten data in the first written data corresponding to the first node, writing the unwritten data into the second node, and writing the written data in the first written data migrating incoming data to the second node;

[0067] Or, stop writing the first write data into the first node, and completel...

Embodiment 2

[0071] A method for optimizing data writing, which differs from Embodiment 1 in that:

[0072] The S3 is specifically:

[0073] judging the written ratio of the first written data in the first node, if the written ratio exceeds a preset third threshold, continue to write the unwritten data in the first written data Incoming data is written to the first node;

[0074] Otherwise, write the first write data corresponding to the first node into the second node in the node set, and increase the number of slow reactions corresponding to the first node by 1;

[0075] If the number of slow responses of the first node reaches the preset fourth threshold, mark the first node as a slow response node, send a slow response node warning and set the status of the slow response node as unwritable ;

[0076] In an optional implementation manner, the number of slow responses in the node is cleared every preset time interval.

Embodiment 3

[0078] Apply one of the above data writing optimization methods to actual scenarios:

[0079] Suppose there is a distributed software (such as MongoDB, Elasticsearch or Hadoop), in which there is a metadata management center and 6 data nodes, respectively marked as A, B, C, D, E and F, if there is a For data writing, 3 data nodes will be selected for data writing according to the preset rules (different distributed software preset rules are different, such as writing to hdfs files in the Hadoop system). When the 3 data nodes After the data is written successfully, the data writing returns a successful status;

[0080] In this embodiment, there is a write data, and three data nodes A, B, and C need to be written, then the following steps need to be performed at this time:

[0081] S1. Monitor the response time of each written data packet corresponding to the written data;

[0082] S2. If the response time of the written data packet corresponding to node A exceeds the threshol...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention discloses a data writing optimization method. The method comprises the following steps: when nodes in a node set receive a plurality of writing data packets corresponding to a piece of writing data, monitoring response time of each writing data packet; marking a first write-in data packet of which the response time exceeds a preset first threshold value, and marking a node of which the number of the first write-in data packets exceeds a preset second threshold value as a first node; and writing the first write-in data corresponding to the first node into a second node in the node set. When data writing is carried out, the response time of each write-in data packet written into each node is monitored for each node, if the state of the node at the moment is not good, the node is replaced for data writing instead of waiting for the response result of the node, and the processing efficiency of the write-in request of the client side is improved.

Description

technical field [0001] The invention relates to the field of distributed databases, in particular to a method for optimizing data writing and a terminal. Background technique [0002] Many basic server-side software now use distributed deployment to meet high concurrency and high throughput functions, that is, there is a metadata management center and multiple data nodes, and the metadata information is managed by the metadata management center. Data nodes are dedicated to data storage, and the metadata management center decides which data nodes to write data to. In such an architecture, in order to improve data security, a piece of data will be written to multiple different nodes at the same time in general scenarios to improve security. In order to improve write performance, in general scenarios, a piece of data will be divided into multiple smaller data blocks, which are written to multiple different nodes at the same time to improve write efficiency. [0003] Regardles...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/27G06F3/06
CPCG06F16/27G06F3/0611G06F3/067Y02D10/00
Inventor 刘德建林伟陈宏
Owner FUJIAN TIANQUAN EDUCATION TECH LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap