Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Page data processing method and device, computer equipment and storage medium

A data processing operator and page data technology, applied in the field of big data, can solve the problem of low page data processing efficiency

Active Publication Date: 2020-10-20
PINGAN INT SMART CITY TECH CO LTD
View PDF9 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0004] The purpose of the embodiments of the present application is to propose a page data processing method, device, computer equipment, and storage medium to solve the problem of low efficiency of page data processing by traditional ETL tools

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Page data processing method and device, computer equipment and storage medium
  • Page data processing method and device, computer equipment and storage medium
  • Page data processing method and device, computer equipment and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0051] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the technical field of the application; the terms used herein in the description of the application are only to describe specific embodiments The purpose is not to limit the present application; the terms "comprising" and "having" and any variations thereof in the specification and claims of the present application and the description of the above drawings are intended to cover non-exclusive inclusion. The terms "first", "second" and the like in the description and claims of the present application or the above drawings are used to distinguish different objects, rather than to describe a specific order.

[0052] Reference herein to an "embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The occurrenc...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The embodiment of the invention belongs to the field of big data, is applied to the field of smart cities, and relates to a page data processing method. The method comprises the following steps of: when a selection instruction sent by a terminal is received, selecting a crawler operator from data processing operators deployed by an ETL platform; obtaining crawler configuration information according to a configuration instruction triggered in a crawler configuration page of the terminal; configuring the crawler operator according to the crawler configuration information; running the configuredcrawler operator through a crawler application, and indicating the crawler application to store crawled page data in Redis; adding the page data in the Redis to an ETL data stream of the ETL platform;and performing ETL processing on the ETL data stream to obtain inventory data. The invention further provides a page data processing device, computer equipment and a storage medium. In addition, theinvention further relates to a block chain technology, and the inventory data can be stored in a block chain. According to the invention, the processing efficiency of the page data is improved.

Description

technical field [0001] The present application relates to the field of big data, and in particular to a page data processing method, device, computer equipment and storage medium. Background technique [0002] With the development of big data technology, the application of ETL is becoming more and more extensive. ETL (Extract-Transform-Load) is the process of extracting, transforming, and loading data from the source to the destination. The data sources of ETL are usually various business systems, and the destination is usually a data warehouse, but it is not limited to a data warehouse. The purpose of ETL is to integrate all kinds of scattered, messy, and non-uniform data to provide analysis basis for decision-making. ETL has important applications in business intelligence. [0003] However, traditional ETL tools can only obtain data from databases or specified files, and cannot directly process a large amount of data that is not stored in databases or files, such as page...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/951H04L29/08G06F16/25
CPCG06F16/951H04L67/02G06F16/254H04L67/568
Inventor 贾波涛
Owner PINGAN INT SMART CITY TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com