Unlock instant, AI-driven research and patent intelligence for your innovation.

Data processing method, client and distributed computing system

A distributed computing and data processing technology, applied in the Internet field, can solve the problems of prone to errors and many resources, and achieve the effect of improving accuracy, reducing redis servers, and saving bandwidth

Active Publication Date: 2021-04-16
CHINANETCENT TECH
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] The technical problem to be solved by the present invention is to overcome the defect that the distributed computing system consumes too many system resources and is prone to errors during operation in the prior art, and provides a method that reduces the consumption of cpu, memory and other resources, and can compress data to save A data processing method, a client and a distributed computing system that ensure the accuracy of the data when transferring data while ensuring the accuracy of the data when executing commands in batches

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method, client and distributed computing system
  • Data processing method, client and distributed computing system
  • Data processing method, client and distributed computing system

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0057] see figure 1 , the present embodiment provides a distributed computing system 1, the distributed computing system includes 5 clients 11, 1 server 12, the server is a redis server, and the redis server includes several redis instances .

[0058] One of the five clients is a first client, and the first client includes an acquisition module 111, a processing module 112, a sending module 113, a setting module 114, a first judging module 115 and a second judging module Module 116.

[0059] The obtaining module is used to obtain data to be processed.

[0060] The first judging module is used for judging whether the data type of the key exists locally, if so, calling the processing module, otherwise calling the creating module.

[0061] The creation module is used to locally create the data type of the key value, and then call the processing module.

[0062] The processing module is used to process the data to be processed according to a preset rule to obtain a result valu...

Embodiment 2

[0085] see image 3 , this embodiment is basically the same as Embodiment 1, the only difference is:

[0086] Step 200 is executed after step 104 .

[0087] Step 200 , judging whether the time from the current moment to the last time to send data to be processed is greater than 5 seconds, if so, execute step 201 , otherwise return to step 100 .

[0088] Step 201. Send the data to be processed to obtain the result value to the redis server.

[0089] The data processing method, client, and distributed computing system of this embodiment can achieve the following effects: reduce the number of redis servers, because in the process of stream computing, the redis commands that will be generated locally will be simplified and compressed according to the data type, and will The commands that need to be processed centrally by the redis server are scattered on various stream computing machines, which can reduce the CPU, memory and other resources consumed by executing the redis comman...

Embodiment 3

[0091] see Figure 4 , this embodiment is basically the same as Embodiment 1, the only difference is:

[0092] Step 105 is replaced by step 105'.

[0093] Step 105', judging whether the total number of data to be processed processed according to the preset rules is greater than 10000, if yes, execute step 106', if not, execute step 107'.

[0094] Step 106', sending 10,000 pieces of data to be processed after performing calculation locally to the server, and then ending the process.

[0095] Step 107', judging whether the time between the current time and the last time to send the data to be processed is greater than 5 seconds, if so, execute step 108', otherwise return to step 100.

[0096] Step 108', sending the data to be processed to obtain the result value to the redis server.

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method, a client and a distributed computing system. The distributed system includes several clients and a server. The data processing method includes: the client obtains the data to be processed; The data to be processed is to obtain the result value, wherein the preset rule is to calculate locally the data with the same data type and / or the same key value in the data to be processed; The result value of the data is sent to the server to be processed. This invention can reduce the computing load of the distributed computing system, reduce the consumption of cpu, memory and other resources, and can compress the data to save the bandwidth when transmitting the data, and at the same time, it can also ensure batch execution. Accuracy of data at time of order.

Description

technical field [0001] The embodiments of the present invention relate to the technical field of the Internet, and in particular to a data processing method, a client and a distributed computing system. Background technique [0002] The rapid development of the Internet industry has not only brought about explosive growth in data scale, but also made big data more and more streamlined. The traditional batch processing mode is difficult to meet the real-time computing requirements of streaming big data processing. Therefore, More efficient distributed computing systems are being used more and more widely. [0003] During the business processing of streaming computing, the system uses redis for data calculation and temporarily stores the data in redis (a high-performance key-value database, namely key-value database). According to business requirements, during the calculation process, a large amount of data will inevitably generate a large number of redis commands. In order ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): H04L29/08
CPCH04L67/10
Inventor 刘成彦李瑜婷刘华明
Owner CHINANETCENT TECH