Data storage processing method and device and server

A processing method and database technology, applied in the Internet field, can solve the problems of large consumption of data resources and low processing efficiency, and achieve the effects of reducing consumption, improving processing efficiency, and solving low processing efficiency.
CN112100182APending Publication Date: 2020-12-18CHINA CONSTRUCTION BANK
0 Cites 4 Cited by

Patent Information

Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
CHINA CONSTRUCTION BANK
Publication Date
2020-12-18

Smart Images

  • Figure 1
    Figure 1
  • Figure 2
    Figure 2
  • Figure 3
    Figure 3
Patent Text Reader

Abstract

The invention provides a data storage processing method and device and a server. Based on the data storage processing method, after to-be-stored target data is obtained, the method incldes adding corresponding custom identification information to the target data through a spark-based preset operator, wherein the length of the custom identification information is smaller than a preset length threshold; and calling a preset data interface based on spark, and storing the target data carrying the custom identification information into a database based on ElasticSearch in batches through multi-thread parallel processing. Therefore, the performance advantages of the spark distributed framework can be fully utilized, the user-defined identification information is used for replacing disordered default IDs generated by the system, multi-thread parallel processing is carried out, consumption of data resources is reduced, the data storage processing efficiency is improved, and data storage can becompleted efficiently and accurately in batches.
Need to check novelty before this filing date? Find Prior Art

Description

technical field

[0001] This specification belongs to the technical field of the Internet, and in particular relates to a data storage processing method, device and server. Background technique

[0002] In many data processing scenarios, ElasticSearch (ES, a data analysis search engine) is often used to query and use relevant data.

[0003] However, based on the existing data warehousing processing method, when the data to be warehousing is stored in the ES database, the storage and processing efficiency is often low, and the consumption of data resources is large. Especially in scenarios where the amount of data to be stored in the database is large and the complexity of the data type is high (for example, transaction data processing scenarios of financial institutions such as banks), the above problems are more obvious.

[0004] For the above problems, no effective solution has been proposed yet. Contents of the invention

[0005] This manual provides a data storage pro...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More