Page data processing method, device, computer equipment and storage medium

A technology of page data and processing method, applied in the field of big data, can solve the problem of low page data processing efficiency, and achieve the effect of improving configuration efficiency, fast response, and improving processing efficiency

Active Publication Date: 2020-12-15
PINGAN INT SMART CITY TECH CO LTD
View PDF9 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiments of the present application is to propose a page data processing method, device, computer equipment, and storage medium to solve the problem of low efficiency of page data processing by traditional ETL tools

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Page data processing method, device, computer equipment and storage medium
  • Page data processing method, device, computer equipment and storage medium
  • Page data processing method, device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the application; the terms used herein in the description of the application are only to describe specific embodiments The purpose is not to limit the present application; the terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the description of the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

[0052] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrenc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The embodiment of the present application belongs to the field of big data, is applied in the field of smart cities, and relates to a page data processing method, including: when receiving a selection instruction sent by the terminal, selecting a crawler operator from the data processing operators deployed on the ETL platform Obtain crawler configuration information according to the configuration command triggered in the crawler configuration page of the terminal; configure the crawler operator according to the crawler configuration information; run the configured crawler operator through the crawler application, and instruct the crawler application The crawled page data is stored in Redis; the page data in the Redis is added to the ETL data flow of the ETL platform; ETL processing is performed on the ETL data flow to obtain inventory data. The present application also provides a page data processing device, computer equipment and a storage medium. In addition, this application also relates to block chain technology, and inventory data can be stored in the block chain. The application improves the processing efficiency of page data.

Description

technical field [0001] The present application relates to the field of big data, and in particular to a page data processing method, device, computer equipment and storage medium. Background technique [0002] With the development of big data technology, the application of ETL is becoming more and more extensive. ETL (Extract-Transform-Load) is the process of extracting, transforming, and loading data from the source to the destination. The data sources of ETL are usually various business systems, and the destination is usually a data warehouse, but it is not limited to a data warehouse. The purpose of ETL is to integrate all kinds of scattered, messy, and non-uniform data to provide analysis basis for decision-making. ETL has important applications in business intelligence. [0003] However, traditional ETL tools can only obtain data from databases or specified files, and cannot directly process a large amount of data that is not stored in databases or files, such as page...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F16/951H04L29/08G06F16/25
CPCG06F16/951H04L67/02G06F16/254H04L67/568
Inventor 贾波涛
Owner PINGAN INT SMART CITY TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap