Unlock instant, AI-driven research and patent intelligence for your innovation.

Data skew determination method and device, electronic equipment and readable storage medium

A data and internal storage technology, applied in the input/output process of data processing, electrical digital data processing, complex mathematical operations, etc., can solve problems such as the inability to achieve quantitative analysis of data tilt

Active Publication Date: 2020-08-14
BEIJING QIYI CENTURY SCI & TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the existing method can only indicate that there is data skew between partitions, and cannot achieve quantitative analysis of data skew

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data skew determination method and device, electronic equipment and readable storage medium
  • Data skew determination method and device, electronic equipment and readable storage medium
  • Data skew determination method and device, electronic equipment and readable storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] The technical solutions in the embodiments of the present invention will be described below with reference to the drawings in the embodiments of the present invention.

[0050] The distributed message system is widely used, and it is the basic software for sending and receiving messages in a distributed system. Distributed message system can use efficient and reliable message delivery mechanism for platform-independent data exchange, and integrate distributed systems based on data communication. Interprocess communication can be extended in a distributed environment by providing a message passing and message queuing model. Applications or components using the distributed message system can perform reliable asynchronous communication, thereby reducing the coupling between systems and improving the scalability and availability of the system. In this scenario, the distributed message system takes advantage of the distributed characteristics, and its high throughput and hi...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention provides a data skew determination method and device, electronic equipment and a readable storage medium, and relates to the technical field of computer application. The method can comprise the steps of: acquiring the message writing rate of each partition in a distributed message system, and enabling the message writing rate to represent the amount of messages stored in the partition in unit time; calculating the dispersion degree between the message writing rates of the partitions through a preset calculation mode, wherein the preset calculation mode comprisesa mode capable of calculating the dispersion degree; and the dispersion degree is used for reflecting the data skew degree between the partitions. According to the data skew determination method anddevice, the electronic equipment and the readable storage medium provided by the embodiment of the invention, quantitative analysis of data skew in the distributed message system can be realized.

Description

technical field [0001] The invention relates to the field of computer application technology, in particular to a method, device, electronic equipment and readable storage medium for determining data skew in a distributed message system. Background technique [0002] In a distributed message system such as Kafka (Kafka), in order to achieve high throughput and high availability, message services can be provided through partitions. Specifically, each partition stores messages separately, and the messages of multiple partitions constitute the total amount of messages . [0003] In an actual production environment, it is possible that the amount of messages stored in different partitions is different, which causes the data in each partition to be unbalanced. However, the data in each partition is unbalanced. For the server in the distributed message system, the distribution characteristics of Kafka cannot be fully utilized. The high-load partition may become a hot spot in the d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F3/06G06F17/18
CPCG06F3/0604G06F3/0644G06F3/0653G06F3/067G06F17/18
Inventor 冯浩
Owner BEIJING QIYI CENTURY SCI & TECH CO LTD