Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Task storage method and device based on external data files

A technology of external data and task storage, applied in the field of communication, can solve the problems of slow query speed of task data and inability to find task data at one time, and achieve the effect of improving the speed

Active Publication Date: 2019-12-10
BEIJING GRIDSUM TECH CO LTD
View PDF4 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The embodiment of the present invention provides a task storage method and device based on external data files, to at least solve the problem that task data in the prior art is stored in a distributed database, and all task data cannot be found at one time, resulting in slow task data query speed technical issues

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Task storage method and device based on external data files
  • Task storage method and device based on external data files
  • Task storage method and device based on external data files

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0029] According to an embodiment of the present invention, a method embodiment of a task storage method based on an external data file is provided. It should be noted that the steps shown in the flow chart of the accompanying drawings can be implemented in a computer system such as a set of computer-executable instructions and, although a logical order is shown in the flowcharts, in some cases the steps shown or described may be performed in an order different from that shown or described herein.

[0030] ETL, short for Extract-Transform-Load, is used to describe the process of extracting, transforming, and loading data from the source to the destination. Parquet is a column storage format for analytical services. In distributed real-time query engines, such as Impala, Hive and other query engines, it is often used as an external storage file. The Impala query engine is used as an example to illustrate. In this embodiment, to achieve The task storage method based on the exter...

Embodiment 2

[0070] According to this embodiment, a task storage device based on an external data file is also provided, and the task storage device based on an external data file is mainly used to execute the task storage method based on an external data file provided in the above-mentioned content of the embodiment of the present invention, The task storage device based on the external data file provided by this embodiment will be specifically introduced below.

[0071] figure 2 is a schematic structural diagram of a task storage device based on an external data file according to Embodiment 2 of the present invention, as shown in figure 2 As shown, the device includes:

[0072] The first obtaining module 21 is configured to obtain a task set to be stored, wherein the task set includes: a plurality of task data, and type information and partition information corresponding to each task data.

[0073] The first reading module 23 is used to read the corresponding storage location of each...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a task storage method and a task storage device based on an external data file. The method comprises the steps of acquiring a task set to be stored, wherein the task set includes a plurality of task data, and type information and partitioning information corresponding to each piece of task data; reading a storage device corresponding to each piece of task data from the preconfigured external data file according to the type information and the partitioning information corresponding to each piece of task data, wherein the external data file is used for storing each piece of type information, each piece of partitioning information and a corresponding relation of each storage position; and respectively storing each piece of task data in the task set to the corresponding storage position of each piece of task data. The task storage method and the task storage device solve the technical problem in the prior art of slow task data query speed caused by the situation that the task data are stored in a distributed database and all task data cannot be found once.

Description

technical field [0001] The present invention relates to the communication field, in particular, to a task storage method and device based on external data files. Background technique [0002] In the prior art, when performing data analysis for multiple users, one user may correspond to one profile (user configuration file), or a bunch of profiles (the unique field of the corresponding website for data analysis). Users not only hope to obtain high analysis ability when viewing the data of a single website, but also hope to put all the profile data together to view the relevant data of the entire station group. This creates a contradiction. Putting all the data in the same In a database, when querying a single site, due to data interference from other site groups, the query speed is affected. [0003] Aiming at the technical problem that task data is stored in a distributed database in the prior art, and all task data cannot be found at one time, resulting in slow query speed...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/14G06F16/182
CPCG06F16/148G06F16/182
Inventor 洪超
Owner BEIJING GRIDSUM TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com