A method and apparatus for generating and calling big data

A big data and data technology, applied in the field of big data generation and invocation methods and devices, can solve problems such as difficult to achieve test goals, difficult to traverse encrypted data, data type restrictions, etc., to achieve analysis and processing, and simplify the system. The effect of building, reducing execution time

Inactive Publication Date: 2018-12-11
深圳市牛鼎丰科技有限公司
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

Simply using a few pieces of data for verification is already difficult to achieve the goal of testing
Inevitably causes problems to slip into production
[0005] Existing schemes, one is to use the corresponding performance testing tool function to generate limited da

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A method and apparatus for generating and calling big data
  • A method and apparatus for generating and calling big data
  • A method and apparatus for generating and calling big data

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0052] Such as figure 1 with figure 2 As shown, a method for generating and calling big data according to an embodiment of the present invention is provided, including the following steps:

[0053] S101 acquires data;

[0054] Specifically, use etl and / or hive tools to obtain data.

[0055] etl, which is the abbreviation of Extract-Transform-Load in English, is used to describe the process of extracting, transforming, and loading data from the source to the destination. The term etl is more commonly used in data warehouses, but its objects are not limited to data warehouses. ETL is an important part of building a data warehouse. Users extract the required data from the data source, after data cleaning, and finally load the data into the data warehouse according to the pre-defined data warehouse model.

[0056] The quality problems of etl are specifically manifested in several characteristics such as correctness, completeness, consistency, completeness, validity, timelines...

example

[0112] CREATE TABLE STUDENT INFO

[0113] (

[0114] NAME STRING,

[0115] FAVORITE ARRAY ,

[0116] COURSE MAP,

[0117] ADDRESS STRUCT

[0118] )

[0119] Query syntax: SELECT S.NAME,S.FAVORITE[0],S.COURSE["ENGLISH"],S.ADDRESS.CITY FROM STUDENTINFO S;

[0120] hive partition table

[0121] Create partition table: create table employee(name string,age int,sex string) partitioned by(city string)row format delimited fields terminated by'\t';

[0122] Partition table loading data: load data local inpath' / usr / local / lee / employee'intotable employee partition(city='hubei').

[0123] S102 encrypting data to obtain encrypted data;

[0124] S103 importing the encrypted data into the database;

[0125] The database is a local database.

[0126] S104 responds to the keyword input by the user to view the request of the target data associated with the keyword;

[0127] Specifically, the sql script is used to find out the address and message of the target...

Embodiment 2

[0160] Such as image 3 As shown, a device for generating and invoking big data according to another embodiment of the present invention is provided, including:

[0161] An acquisition module A201, configured to acquire data;

[0162] Specifically, the acquiring module A201 uses etl and / or hive tools to acquire data.

[0163] An encryption module A202, configured to encrypt data to obtain encrypted data;

[0164] The storage module A203 is used to import encrypted data into the database;

[0165] The database is a local database.

[0166] Response module A204, for responding to the keyword input by the user to view the request of the data associated with the keyword;

[0167] The processing module A208 specifies the type of the target data.

[0168] The first configuration module A205 is used to configure the address and message of the target data;

[0169] Specifically, the first configuration module A205 configures the address of the target data in...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The invention provides a method for generating and calling big data, which comprises the following steps: acquiring data; encrypting the data to obtain encrypted data; importing the encrypted data into a database; responding to a request of a keyword input by a user to view target data associated with the keyword; configuring an address and a message of the target data; configuring a correspondingdecryption method; finding an address and a message of target data associated with the keyword according to the keyword; calling and decrypting the target data; displaying the target data. The invention simplifies the system construction and facilitates the analysis and processing of the big data by connecting the data into the database, requesting the address and the request message of the configuration data and encrypting and decrypting by the user, and eliminating the need to repeat the construction of the related functions.

Description

technical field [0001] The invention relates to the field of data storage and processing, in particular to a method and device for generating and calling big data. Background technique [0002] For a long time, functional testing has always used a limited number of data for functional verification, but there has always been a problem with production data. Even if there is performance testing, due to the complexity of the data and the complexity of the system, the system’s ability to perform various tests is ignored. Compatibility processing of data. [0003] Reasons for such problems: 1. The generation of business data is complex; 2. It is difficult to request batch data and analyze the results; 3. Non-production sensitive data is not included in the test process [0004] The significance of researching this project: As the system becomes more and more complex, and the type complexity of data increases. It is difficult to achieve the goal of testing simply by using a few p...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
IPC IPC(8): G06F17/30G06F21/60G06F21/62
CPCG06F21/602G06F21/6227
Inventor 刘勇蔡春娥
Owner 深圳市牛鼎丰科技有限公司
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap