Data storage processing method and device and server

A processing method and database technology, applied in the Internet field, can solve the problems of large consumption of data resources and low processing efficiency, and achieve the effects of reducing consumption, improving processing efficiency, and solving low processing efficiency.

Pending Publication Date: 2020-12-18
CHINA CONSTRUCTION BANK
View PDF0 Cites 4 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0005] This manual provides a data storage processing method, device and server to make full use of the performance advantages of the spark distributed framework, and use custom identification information instead of the default ID automatically generated by the system to perform multi-threaded parallel processing and reduce data processing. Resource consumption, improve the processing efficiency of data storage, can efficiently and accurately complete data storage in batches, and solve the problems of low processing efficiency and large consumption of data resources in existing methods question

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data storage processing method and device and server
  • Data storage processing method and device and server
  • Data storage processing method and device and server

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0049] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described The embodiments are only some of the embodiments in this specification, not all of them. Based on the embodiments in this specification, all other embodiments obtained by persons of ordinary skill in the art without creative efforts shall fall within the protection scope of this specification.

[0050]Considering that based on the existing data warehousing processing method, the processing efficiency is often low when storing the data in the database based on ElasticSearch. Moreover, based on the existing methods, in the process of storing data based on ElasticSearch, the default ID of ES is often generated and used for specific data processing. The a...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a data storage processing method and device and a server. Based on the data storage processing method, after to-be-stored target data is obtained, the method incldes adding corresponding custom identification information to the target data through a spark-based preset operator, wherein the length of the custom identification information is smaller than a preset length threshold; and calling a preset data interface based on spark, and storing the target data carrying the custom identification information into a database based on ElasticSearch in batches through multi-thread parallel processing. Therefore, the performance advantages of the spark distributed framework can be fully utilized, the user-defined identification information is used for replacing disordered default IDs generated by the system, multi-thread parallel processing is carried out, consumption of data resources is reduced, the data storage processing efficiency is improved, and data storage can becompleted efficiently and accurately in batches.

Description

technical field [0001] This specification belongs to the technical field of the Internet, and in particular relates to a data storage processing method, device and server. Background technique [0002] In many data processing scenarios, ElasticSearch (ES, a data analysis search engine) is often used to query and use relevant data. [0003] However, based on the existing data warehousing processing method, when the data to be warehousing is stored in the ES database, the storage and processing efficiency is often low, and the consumption of data resources is large. Especially in scenarios where the amount of data to be stored in the database is large and the complexity of the data type is high (for example, transaction data processing scenarios of financial institutions such as banks), the above problems are more obvious. [0004] For the above problems, no effective solution has been proposed yet. Contents of the invention [0005] This manual provides a data storage pro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Applications(China)
IPC IPC(8): G06F16/22G06F16/245G06F16/2458G06F16/27G06Q40/02
CPCG06Q40/02G06F16/2282G06F16/245G06F16/2471G06F16/27
Inventor 李威覃鹏刘增文吴秦明张小宁
Owner CHINA CONSTRUCTION BANK
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap