Unlock instant, AI-driven research and patent intelligence for your innovation.

Batch processing data fragmentation method and system, computer equipment and storage medium

A data sharding and batch processing technology, applied in computing, electrical digital data processing, digital data information retrieval, etc., can solve problems such as unfavorable batch task execution efficiency and stand-alone deployment, and reduce development access costs and development difficulties. , the effect of improving execution efficiency

Pending Publication Date: 2021-06-18
开店宝科技集团有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] As the application of distributed systems is becoming more and more widespread, and the amount of data processed is also increasing, in the traditional mode, batch processing tasks are often deployed on a single machine, so there will be bottlenecks, which is not conducive to improving batch processing. task execution efficiency

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Batch processing data fragmentation method and system, computer equipment and storage medium
  • Batch processing data fragmentation method and system, computer equipment and storage medium
  • Batch processing data fragmentation method and system, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0058] An embodiment of the present invention, such as figure 1 and image 3 As shown, the present invention provides a batch data fragmentation method, comprising steps:

[0059] S1. When the client starts, register the service with Zookeeper.

[0060] Preferably, when the client starts, the registration service is to Zookeeper, specifically including:

[0061] Initialize the Zookeeper connection; establish a connection with Zookeeper through the open source framework (Curator framework); monitor the real-time status of the custom persistent node in Zookeeper, so as to receive the broadcast sent by Zookeeper to the newly registered message in time.

[0062] Specifically, the open source framework encapsulates a whole set of underlying APIs connected to Zookeeper, which can greatly simplify the connection process and ensure the stability of the link, and internally encapsulates the heartbeat detection mechanism.

[0063] S2. Receive the broadcast sent by Zookeeper for the n...

Embodiment 2

[0079] An embodiment of the present invention, such as figure 2 and image 3 As shown, the present invention also provides another batch data fragmentation method, comprising steps:

[0080] S10. Receive the registration service sent by the SDK when the client starts.

[0081] S11. Update the nodes in the node list according to the registration service.

[0082] S12. After the node is created, broadcast a newly registered message.

[0083] S13. Accept the node list query request sent by the SDK after receiving the broadcast, so that the SDK calculates and obtains the fragmentation coefficient corresponding to the client according to the current ranking position of the node corresponding to the client in the node list and the total number of registrations.

[0084] Zookeeper can update the nodes in the node list by receiving the registration service sent by the SDK when the client starts, and broadcast the newly registered message after the node is created. After receiving ...

Embodiment 3

[0091] An embodiment of the present invention, such as Figure 7 As shown, the present invention also provides a batch processing data sharding system, including a registration module 1 , a receiving module 2 , a query module 3 , a calculation module 4 and a sending module 5 .

[0092] The registration module 1 is used to register the service to Zookeeper when the client starts; the receiving module 2 is used to receive the broadcast sent by Zookeeper to the newly registered message; the query module 3 is used to query the updated node list of Zookeeper and obtain the corresponding The current ranking position of the node in the node list, and the total number of registrations; the calculation module 4 is used to calculate and obtain the fragmentation coefficient corresponding to the client according to the current ranking position and the total registration number; the sending module 5 is used to send the fragmentation coefficient to The client enables the client to query the...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a batch processing data fragmentation method and system, computer equipment and a storage medium, and the method comprises the steps: registering a service to a Zookeeper when a client is started; receiving a broadcast sent by the Zookeeper to a newly registered message; inquiring the node list after the Zookeeper is updated; obtaining a current sorting position of a node corresponding to the client in the node list and a total registration number; and according to the current sorting position and the total registration number, performing calculation to obtain a fragmentation coefficient corresponding to the client. According to the scheme, dynamic and real-time capacity expansion of the batch processing data fragments can be realized through the Zookeeper, so that the execution efficiency of the batch processing task can be improved, and the development access cost and the development difficulty can be reduced.

Description

technical field [0001] The invention relates to the technical field of data sharding, in particular to a batch processing data sharding method, system, computer equipment and storage medium. Background technique [0002] Zookeeper is a distributed, open source distributed application coordination service, an open source implementation of Google's Chubby, and an important component of Hadoop and Hbase. It is a software that provides consistent services for distributed applications. Its functions include: configuration maintenance, domain name service, distributed synchronization, group service, etc. The goal of Zookeeper is to encapsulate complex and error-prone key services, and provide users with simple and easy-to-use interfaces and systems with high performance and stable functions. [0003] As the application of distributed systems is becoming more and more widespread, and the amount of data processed is also increasing, in the traditional mode, batch processing tasks a...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48G06F9/50G06F9/54G06F16/2455
CPCG06F9/4881G06F9/542G06F9/5027G06F16/24554
Inventor 刘胜平崔俊荣韩美荣
Owner 开店宝科技集团有限公司