Supercharge Your Innovation With Domain-Expert AI Agents!

Data processing task cleaning method and device

A technology of data processing and cleaning methods, applied in the field of big data, can solve the problems of lack of versatility, the efficiency of manual statistical management data processing tasks is difficult to match the rate of repeated development of data processing tasks, and the reliability is low, so as to reduce manual labor. Participate in, avoid metadata quality instability, and improve the effect of cleaning efficiency

Pending Publication Date: 2021-05-11
BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
View PDF0 Cites 1 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

However, as the amount of data processing tasks increases, the efficiency of manual statistical management of data processing tasks is difficult to match the rate of repeated development of data processing tasks
In addition, at present, the identification of data processing tasks is mainly based on metadata, but because the quality of metadata depends on developers maintaining information such as field annotations, business rules, and processing standards, the reliability is low and it is not universal

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing task cleaning method and device
  • Data processing task cleaning method and device
  • Data processing task cleaning method and device

Examples

Experimental program
Comparison scheme
Effect test

Embodiment Construction

[0041] Exemplary embodiments of the present invention are described below in conjunction with the accompanying drawings, which include various details of the embodiments of the present invention to facilitate understanding, and they should be regarded as exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the invention. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.

[0042] figure 1 is a schematic flow diagram of the main flow of the cleaning method for data processing tasks according to an embodiment of the present invention, such as figure 1 As shown, the cleaning method of the data processing task may specifically include the following steps:

[0043] Step S101, acquiring SQL running scripts of the first task to be cleaned up and ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing task cleaning method and device, and relates to the technical field of data processing. A specific embodiment of the method comprises the steps of obtaining SQL running scripts of a first to-be-cleaned task and a second to-be-cleaned task; according to one or more preset task elements and an extraction rule of the task elements, respectively extracting values of one or more task elements of the first to-be-cleaned task and the second to-be-cleaned task from the SQL running script; based on a text similarity algorithm, calculating the similarity of the values of the task elements corresponding to the first to-be-cleared task and the second to-be-cleared task, so as to determine the similarity of the first to-be-cleared task and the second to-be-cleared task; and if the similarity between the first to-be-cleared task and the second to-be-cleared task is greater than a threshold similarity, clearing the second to-be-cleared task. According to the embodiment, manual participation is reduced, and the task cleaning efficiency is improved.

Description

technical field [0001] The invention relates to the technical field of big data, in particular to a method and device for clearing data processing tasks. Background technique [0002] With the development of Internet technology, data resources have increased dramatically, and big data has become the basic resource for the daily operation of enterprises. Distributed storage and computing tools provide convenient tools for enterprises to apply big data. However, with the increase of enterprise scale, there are more and more big data application scenarios, and the number of development teams is increasing. The phenomenon of repeated development of data processing tasks has appeared, which has caused problems such as excessive pressure on enterprise servers and waste of resources. The post-governance model leads to governance difficulties for data processing tasks. [0003] At present, for the problem of repeated development of data processing tasks, the common method is to man...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
Patent Type & Authority Applications(China)
IPC IPC(8): G06F9/48G06F16/242
CPCG06F9/485G06F16/2433Y02D10/00
Inventor 焦文健王海旭王建辉陈希
Owner BEIJING WODONG TIANJUN INFORMATION TECH CO LTD
Features
  • R&D
  • Intellectual Property
  • Life Sciences
  • Materials
  • Tech Scout
Why Patsnap Eureka
  • Unparalleled Data Quality
  • Higher Quality Content
  • 60% Fewer Hallucinations
Social media
Patsnap Eureka Blog
Learn More

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2025 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com