Unlock instant, AI-driven research and patent intelligence for your innovation.

Distributed mass dynamic task engine and method for processing data with same

A dynamic task and large-volume technology, applied in other database retrieval, network data retrieval, network data indexing, etc., can solve problems such as adjustment and change, the performance of a server cannot meet the requirements of the system, and the system cannot meet the requirements of task execution, etc.

Active Publication Date: 2015-03-04
德比软件(上海)有限公司
View PDF5 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, with the increase in the number and types of tasks, the system faces several challenges
[0003] 1. In a small server system, data capture and data processing are generally implemented by one server, but as the amount of data increases, the performance of one server can no longer meet the requirements of the system
Since this single-server system is difficult to meet the task execution requirements, it is necessary to call a large number of servers for calculation execution
[0004] 2. The execution content of the task may vary widely according to different business requirements, and may be adjusted and changed at any time
[0005] 3. The results of tasks require centralized processing and storage

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Distributed mass dynamic task engine and method for processing data with same
  • Distributed mass dynamic task engine and method for processing data with same
  • Distributed mass dynamic task engine and method for processing data with same

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0024] Embodiment one is the example that the distributed large batch dynamic task engine of the present invention is used for the data management of hotel system; figure 2 , the hotel system needs to capture the real-time prices of the hotel throughout the year, and it needs to send about 10,000,000 messages to the hotel system, and store the captured prices in the corresponding data warehouse (ie figure 2 Data Warehouses in A-D). To achieve this goal, we use three groups of servers. The first group of servers is responsible for managing these crawling tasks. Considering that different types of tasks have different parameters and different frequencies, they are managed by different servers (ie figure 2 Task Manager A-D in . The second group of servers is a DTE gateway server (DTE Gateway) and a DTE proxy server (DTE Agent). The third group of servers is responsible for storing the data warehouse of crawled data (i.e. figure 2 Data Warehouses in A-D). When the task rea...

Embodiment 2

[0026] Embodiment 2 presents the process of using the distributed large-batch dynamic task engine of the present invention to regularly synchronize data in different data warehouses.

[0027] see image 3, when the data in different data warehouses need to be synchronized regularly, it is necessary to check whether data warehouses A1, A2, A3 and A4 have data updates every five minutes, and if so, these changed data need to be synchronized to data warehouse B. The data of data warehouse B comes from multiple data warehouses (namely data warehouses A1-A4), and only contains part of the data, not all of them.

[0028] In order to realize this function, the task manager is responsible for fragmenting the data in the data warehouse B, obtaining the data source from the data source location server according to the keyword of the data, grouping the synchronization script according to the number of data keywords of the data source , a set of keywords plus the address of the source da...

Embodiment 3

[0031] Embodiment 3 presents the process of using the distributed large-batch dynamic task engine of the present invention to grab data from different websites.

[0032] see Figure 4 , when it is necessary to grab data from different websites and store it in a unified data warehouse after analysis, the task manager submits the task to the DTE Gateway server (DTE Gateway) and the DTE proxy server (DTE Agent). The DTE proxy server is responsible for fetching data from different target websites (namely websites 1-4), and finally delivers the results to the data warehouse for updating. Due to differences in crawling tasks on different websites, we store the scripts of task execution directly on the task manager. When there is a new website to crawl, you only need to add a new task script and configuration to the task manager.

[0033] Through the description of the first to third embodiments above, the distributed large-batch dynamic task engine proposed by the present inventio...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a distributed mass dynamic task engine and a method for processing data with the distributed mass dynamic task engine. The distributed mass DTE comprises a DTE Gateway server and a DTE Agent server, wherein the DTE Gateway server is used for managing the DTE Agent server, and the DTE Agent server is used for executing tasks. When only a certain kind of tasks need to be executed, a task client side uses the distributed mass DTE; when a certain group of tasks need to be executed and managed, a task manager uses the distributed mass DTE. According to the distributed mass dynamic task engine and the method for processing data with the distributed mass dynamic task engine, not only can a plurality of tasks be executed in parallel, but also when content of the tasks changes, the mask executing server does not need to be updated, and the tasks can be executed just by adjusting task scripts.

Description

technical field [0001] The invention relates to the technical field of distributed tasks, in particular to a distributed large-batch dynamic task engine and a data processing method using the distributed large-batch dynamic task engine. Background technique [0002] In the WEB system, it is often necessary to implement the part of timing task execution. In conventional JavaEE systems, Quartz is commonly used for simple task scheduling and setting the execution frequency of tasks. When execution is reached, the task is executed locally, and the result of the task is processed for storage, etc. However, with the increase of the number and types of tasks, the system faces several challenges. [0003] 1. In a small server system, data capture and data processing are generally implemented by one server, but as the amount of data increases, the performance of one server can no longer meet the requirements of the system. Since such a single-server system is difficult to meet the...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Patents(China)
IPC IPC(8): G06F17/30
CPCG06F16/275G06F16/951
Inventor 杨洋
Owner 德比软件(上海)有限公司