Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Quasi-real-time data processing method and device, server and storage medium

A data processing, quasi-real-time technology, applied in the field of data processing, can solve the problems of inability to query data, not support data quasi-real-time update and deletion, unable to synchronize data in quasi-real-time, etc. The effect of storage efficiency

Pending Publication Date: 2021-03-02
PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
View PDF0 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0002] In the current big data platform, the HIVE data warehouse is usually used as the offline data warehouse of the big data platform. In order to ensure the performance of the big data platform, the current big data platform does not support quasi-real-time update and deletion of data. The latest data cannot be synchronized to the HIVE data warehouse in quasi-real time, resulting in the inability to query quasi-real-time data from the HIVE data warehouse

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Quasi-real-time data processing method and device, server and storage medium
  • Quasi-real-time data processing method and device, server and storage medium
  • Quasi-real-time data processing method and device, server and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0058] In order to make the object, technical solution and advantages of the present invention clearer, the present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0059] Such as figure 1 Shown is a flowchart of a preferred embodiment of the quasi-real-time data processing method of the present invention. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

[0060] The quasi-real-time data processing method is applied to one or more servers, and the server is a device that can automatically perform numerical calculation and / or information processing according to preset or stored computer-readable instructions, and its hardware includes But not limited to microprocessors, application specific integrated circuits (Application Specific Integrated Circuit, ASIC), programmable gate arrays (Field-Programmable Gate Array, FPGA), digital proces...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention relates to data processing, and provides a quasi-real-time data processing method and device, a server and a storage medium. The method can determine and synchronize database operation logs with message-oriented middleware, can consume and analyze the database operation logs to obtain quasi-real-time data and update time, can store the quasi-real-time data into a KUDU database, can obtain a pre-created HIVE database and can create an HIVE external table, and an HIVE internal table in the HIVE database stores data before the update time, determines a storage path of the quasi-real-time data in the KUDU database, and stores the storage path in the HIVE external table, and when the query termination time is the update time the HIVE external table and the HIVE internal table arecombined to obtain and query the target data table and obtain response data. According to the invention, the query performance of quasi-real-time data is improved. In addition, the invention also relates to a blockchain technology, and the response data can be stored in the blockchain.

Description

technical field [0001] The invention relates to the technical field of data processing, in particular to a quasi-real-time data processing method, device, server and storage medium. Background technique [0002] In the current big data platform, the HIVE data warehouse is usually used as the offline data warehouse of the big data platform. In order to ensure the performance of the big data platform, the current big data platform does not support quasi-real-time update and deletion of data. The latest data cannot be synchronized to the HIVE data warehouse in quasi-real time, resulting in the inability to query quasi-real-time data from the HIVE data warehouse. Contents of the invention [0003] In view of the above, it is necessary to provide a quasi-real-time data processing method, device, server and storage medium, which can still query quasi-real-time data and improve query performance while ensuring the performance of the big data platform corresponding to the HIVE dat...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/2453G06F16/23G06F16/28
CPCG06F16/2453G06F16/283G06F16/2308
Inventor 赵乐
Owner PINGAN PUHUI ENTERPRISE MANAGEMENT CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com