Supercharge Your Innovation With Domain-Expert AI Agents!

Job flow configuration method and device, electronic device and storage medium

A configuration method and job flow technology, applied in the field of data warehouse development, can solve problems such as inefficiency and time-consuming

Pending Publication Date: 2021-10-22
CITY CLOUD TECH HANGZHOU CO LTD
View PDF0 Cites 2 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0023] The embodiment of the present application provides a workflow configuration method, device, electronic device, and storage medium, aiming at the time-consuming and inefficient problems caused by manually modifying the configuration files required for model online, and can realize fast and accurate script configuration online

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Job flow configuration method and device, electronic device and storage medium
  • Job flow configuration method and device, electronic device and storage medium
  • Job flow configuration method and device, electronic device and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0045] This application aims to propose a way to use Python technology to analyze the existing configuration instances in the workflow scheduling engine, obtain the common parts that do not change in job loading as common templates, and obtain the parts that will change in job loading As a non-common template, only the parameter content of the non-public template is changed during the configuration process, so as to realize the technical solution of efficiently modifying the configuration file when the model goes online. During implementation, obtain a fixed template in Oozie for each configuration file When modifying, first import the fixed template, assign the field value of the common template to the corresponding parameter in the fixed template, change the parameter content of each fixed keyword in the non-common template, and assign the obtained field value to the corresponding parameter in the fixed template , and then complete the efficient configuration process of the t...

Embodiment 2

[0088] Based on the same idea, refer to image 3 , the application also proposes a workflow configuration device, including:

[0089] The first template generating module 301 is configured to obtain the first timing scheduling file, the first job coordination file, and the first workflow file provided by the workflow scheduling engine, and generate a fixed template;

[0090] The second template generation module 302 is used to obtain the second timing scheduling file, the second job coordination file and the second workflow file in the project history job, analyze the fixed keywords in each file, and according to the fixed keywords in the job Whether to generate common templates and non-common templates for common parts when loading;

[0091] Assignment module 303, configured to analyze each of the fixed keywords in the non-common template and assign a value, and write the non-common template;

[0092] A file generation module 304, configured to refer to the fixed template, ...

Embodiment 3

[0094] This embodiment also provides an electronic device, refer to Figure 4 , including a memory 404 and a processor 402, the memory 404 stores a computer program, and the processor 402 is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0095] Specifically, the processor 402 may include a central processing unit (CPU), or an Application Specific Integrated Circuit (ASIC for short), or may be configured to implement one or more integrated circuits in the embodiments of the present application.

[0096] Wherein, the memory 404 may include a mass memory 404 for data or instructions. For example without limitation, the memory 404 may include a hard disk drive (HardDiskDrive, referred to as HDD), a floppy disk drive, a solid state drive (SolidStateDrive, referred to as SSD), flash memory, optical disk, magneto-optical disk, magnetic tape or Universal Serial Bus (UniversalSerialBus, abbreviated as a USB) drive or a combina...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention provides a job flow configuration method and device, an electronic device and a storage medium, and the method comprises the following steps: obtaining a first timing scheduling file, a first job coordination file and a first workflow file provided by Oozie, and generating a fixed template; acquiring a second timing scheduling file, a second job coordination file and a second workflow file in the project historical job, analyzing fixed keywords in each file, and generating a common template and a non-common template for the common part according to whether the fixed keywords are used for loading the job; analyzing each fixed keyword in the non-generality template, assigning a value, and writing the value into the non-generality template; and generating a target configuration file by quoting the fixed template, the generality template and the non-generality template. Aiming at the problems of time consumption and low efficiency caused by manually modifying the configuration file required by model online at present, the script configuration online can be quickly and accurately realized.

Description

technical field [0001] The present application relates to the technical field of data warehouse development, in particular to a workflow configuration method, device, electronic device and storage medium. Background technique [0002] In the field of data warehouse development, models need to be loaded periodically, and the common types are offline and real-time. The offline is divided into: minutes, hours, days, months, years and other cycles. Real-time is the cycle of seconds and milliseconds. [0003] Data warehouse models need to write online job scripts when they need to be loaded periodically. The online job script includes: basic job information, job dependency information, and job flow priority execution information. [0004] At present, many enterprises do not have self-developed data development platforms, nor have they purchased other development platforms. The technology platforms of the data warehouses they use are all selected and used according to the big d...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F16/245G06F16/28G06F40/186G06F40/205
CPCG06F16/245G06F16/283G06F40/186G06F40/205
Inventor 徐剑炯吴晶晶马浩王强曹鹏寅
Owner CITY CLOUD TECH HANGZHOU CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com