Message queue-based data processing method and device, computer equipment and medium

A message queue and data processing technology, applied in the field of data processing based on message queue, can solve the problems of data synchronization link delay, processing progress delay, data backlog, etc., to avoid redundancy, avoid thread allocation, and realize data backlog effect of the problem

Pending Publication Date: 2021-04-06
平安消费金融有限公司
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, the inventor found that such a solution has obvious problems, because business data usually changes with time, and when the data suddenly increases, there will be delays and backlogs in the data synchronization link of a large number of tables
In addition, in the process of data processing based on message queues in other scenarios, there will also be problems such as delays in processing progress and data backlogs

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Message queue-based data processing method and device, computer equipment and medium
  • Message queue-based data processing method and device, computer equipment and medium
  • Message queue-based data processing method and device, computer equipment and medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0028] Embodiment 1 of the present application provides a data processing method based on message queues. Through this method, during the data processing process, it can be detected in time that the data backlog of message topics in the message queue is serious, and the current thread cannot consume quickly. state, and can dynamically adjust the appropriate amount of thread allocation, so that the data backlog problem can be solved within the inspection cycle and redundant thread allocation can be avoided, manual operations can be reduced, and the efficiency of program operation can be improved. specifically, figure 1 The flow chart of the message queue-based data processing method provided by Embodiment 1 of this application, such as figure 1 As shown, the message queue-based data processing method includes the following steps S101 to S104:

[0029] Step S101: Determine the consumption rate threshold of the thread corresponding to the message subject in the message queue.

...

Embodiment 2

[0084] Corresponding to Embodiment 1 above, Embodiment 2 of the present application provides a data processing device based on message queues. For corresponding technical features and corresponding technical effects, refer to Embodiment 1 above, which will not be repeated here. figure 2 A block diagram of a message queue-based data processing device provided for Embodiment 2 of the present application, such as figure 2 As shown, the device includes a determination module 201 , a first calculation module 202 , a second calculation module 203 and an adjustment module 204 .

[0085] Wherein, the determination module 201 is used to determine the consumption rate threshold of the thread corresponding to the message topic in the message queue; the first calculation module 202 is used to calculate the current data backlog and data consumption rate of the message topic at each preset inspection cycle; the second The second calculation module 203 is configured to: if the data backlog...

Embodiment 3

[0095] Embodiment 3 also provides a computer device, such as a smart phone, a tablet computer, a notebook computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server, Or a server cluster composed of multiple servers), etc. Such as image 3 As shown, the computer device 01 in this embodiment at least includes but is not limited to: a memory 012 and a processor 011 that can communicate with each other through a system bus, such as image 3 shown. It should be pointed out that, image 3 Only the computer device 01 is shown with components memory 012 and processor 011, but it should be understood that implementing all of the illustrated components is not a requirement and that more or fewer components may instead be implemented.

[0096] In this embodiment, the memory 012 (that is, a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (for example, SD or DX memo...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention relates to the technical field of data processing, and provides a data processing method and device based on a message queue, computer equipment and a medium. The method comprises the steps of determining a consumption rate threshold of a thread corresponding to a message theme in a message queue; calculating the current data backlog and data consumption rate of the message theme every other preset check period; if the data backlog quantity reaches a preset backlog quantity threshold value and the data consumption rate reaches a threshold value condition set by utilizing a consumption rate threshold value, calculating an increased thread quantity according to the data backlog quantity, the consumption rate threshold value and an inspection period; and adjusting the thread allocation quantity of the message theme according to the increased thread quantity. Through the data processing method and device, link delay and data backlog caused by data volume increase can be reduced in the data processing process based on the message queue.

Description

technical field [0001] The present application relates to the technical field of data processing, and in particular to a message queue-based data processing method, device, computer equipment and media. Background technique [0002] With the deep intersection of various businesses and the strengthening of dependencies between upstream and downstream business systems, the requirements for data synchronization between upstream and downstream are also increasing. The upstream and downstream data synchronization solutions in the industry generally use message queues to realize business decoupling and data flooding. During the data synchronization process, the amount of data is generally estimated based on experience and business conditions, and a fixed number of threads is used to consume one or several topics. However, the inventor found that such a solution has obvious problems, because business data usually changes with time, and when the data suddenly increases, there will b...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F9/50G06F9/54
CPCG06F9/5027G06F9/546
Inventor 王晓初郑鹰飞
Owner 平安消费金融有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap