Supercharge Your Innovation With Domain-Expert AI Agents!

Data processing method and architecture

A data processing and database technology, applied in the field of data processing, can solve the problems of data loss, different execution time or execution efficiency, and inability to guarantee the order of each data consumption, so as to avoid data loss and improve accuracy.

Pending Publication Date: 2022-04-29
AGRICULTURAL BANK OF CHINA
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, since the consumer randomly reads each piece of data from the message middleware, and the consumption execution time or execution efficiency of each piece of data is different, the order of consumption of each piece of data cannot be guaranteed, resulting in the occurrence of Circumstances such as data loss, which reduces the accuracy and reliability of data processing

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing method and architecture
  • Data processing method and architecture
  • Data processing method and architecture

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0026] figure 2 It is a flow chart of a data processing method provided according to Embodiment 1 of the present invention. This embodiment is applicable to the case where a large amount of data is processed in an orderly manner. The method can be executed by a data processing architecture, and the data processing architecture can adopt implemented in the form of hardware and / or software. Such as figure 2 As shown, the method includes:

[0027] S110. Store the data to be processed through the distributed publish-subscribe message system. The data to be processed is divided into message data of added type, message data of deleted type and message data of modified type according to preset data types.

[0028]In this embodiment, the distributed publish-subscribe message system can refer to a high-throughput message middleware for caching and transmitting data, which can be understood as a pipeline for transmitting real-time data and cached data, which can be processed asynchr...

Embodiment 2

[0045] image 3It is a flow chart of a data processing method according to Embodiment 2 of the present invention, and this Embodiment 2 is refined on the basis of the foregoing embodiments. In this embodiment, the process of obtaining the data to be processed, sorting and caching the data to be processed, and processing after storing the first consumption data or the second consumption data in the target database is described in detail. It should be noted that for technical details not exhaustively described in this embodiment, reference may be made to any of the foregoing embodiments. Such as image 3 As shown, the method includes:

[0046] S210. Convert the database operation log message acquired from the source database according to a preset format to obtain data to be processed.

[0047] In this embodiment, the preset format may refer to a data format corresponding to cached data in the distributed publish-subscribe message system. Convert the data format of the databa...

Embodiment 3

[0088] Figure 5 It is a schematic structural diagram of a data processing architecture provided according to Embodiment 3 of the present invention. Such as Figure 5 As shown, the architecture includes: a distributed publish-subscribe message system 310, an application server system 320 and a key-value storage system 330;

[0089] The data processing architecture is used to execute any data processing method described in the embodiments of the present invention.

[0090] Optionally, in the architecture,

[0091] The distributed publish-subscribe message system 310 is used to store data to be processed, and the data to be processed is divided into message data of increasing type, message data of deleting type and message data of changing type according to preset data types;

[0092] The application server system 320 is configured to read the data to be processed in the distributed publish-subscribe message system 310, and determine the data type of the data to be processed;...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing method and architecture. Storing the to-be-processed data through the distributed publishing and subscribing message system; reading to-be-processed data in the distributed publishing and subscribing message system through the application server system, and judging the data type of the to-be-processed data; if the to-be-processed data is the increased type message data, performing consumption processing on the to-be-processed data through at least one consumer in the application server system; and if the to-be-processed data is deleted type message data or modified type message data, sorting and caching the to-be-processed data through the key value storage system, and reading the to-be-processed data from the key value storage system based on a sorting and caching result through at least one consumer in the application server system and performing consumption processing. According to the invention, the type deletion or type change message data is sorted through the key value storage system, so that the application server system can consume the to-be-processed data in order, and the accuracy and reliability of data processing are improved.

Description

technical field [0001] Embodiments of the present invention relate to the technical field of data processing, and in particular, to a data processing method and architecture. Background technique [0002] With the continuous development of data processing technology, for the processing of large amounts of data, a message middleware is usually added between the producer and the consumer in the prior art to temporarily cache the large amount of data produced by the data producer, in case Consumers cannot consume large amounts of data for a while. Among them, the producer and the consumer are interrelated, the producer is used to produce data, and the consumer is used to consume the data produced by the producer. However, since the consumer randomly reads each piece of data from the message middleware, and the consumption execution time or execution efficiency of each piece of data is different, the order of consumption of each piece of data cannot be guaranteed, resulting in ...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/215G06F16/22G06F16/2455
CPCG06F16/215G06F16/22G06F16/24552
Inventor 林子昂张羽飞
Owner AGRICULTURAL BANK OF CHINA
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com