Looking for breakthrough ideas for innovation challenges? Try Patsnap Eureka!

Data processing flow optimization method and system and storage medium

A technology for data processing and process optimization, applied in the field of data processing, can solve the problems of long calculation time and low calculation efficiency of data processing process

Pending Publication Date: 2021-07-06
SHANDONG YINGXIN COMP TECH CO LTD
View PDF8 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0003] The technical problem mainly solved by the present invention is to provide a data processing flow optimization method, system and storage medium, which can solve the problems of low calculation efficiency and long calculation time of data processing flow

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • Data processing flow optimization method and system and storage medium
  • Data processing flow optimization method and system and storage medium
  • Data processing flow optimization method and system and storage medium

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0051] Embodiment 1 provides a data processing flow optimization method, such as figure 1 shown, including the following steps:

[0052] In the S100 step, the data processing task includes a preprocessing task, an inference task and a postprocessing task; for the data processing task, mark the start time points in the data processing task according to the preprocessing task, the inference task and the postprocessing task respectively; preprocessing The task is generally the initialization part of the system code, such as in C++ / Python, it is generally the initialization part of the main() function; the reasoning task is reasoning based on the deep learning reasoning framework, and the running time of the task is a reasoning framework such as TensorFlow / TensorRT / PyTorch. There are obvious recognition statements in the program, such as Session.run(data) for TensorFlow, doinference(data) for TensorRT, and net(data) for PyTorch; post-processing tasks are generally running codes af...

Embodiment 2

[0068] Embodiment 2 provides a data processing flow optimization system, such as figure 2 As shown, the data processing flow optimization system 100 includes:

[0069] Task identification unit 1: used to add identifications to the preprocessing tasks, reasoning tasks and post-processing tasks in the data processing application, so that the boundaries of each task are clear, and it is convenient to determine the running time of each task and set the queue;

[0070] Thread estimation unit 2: used to respectively determine the running time of the pre-processing task, the reasoning task and the post-processing task according to the identifier, and respectively estimate the pre-processing task, the reasoning task and the post-processing task according to the running time. Describe the number of threads required for post-processing tasks; among them, since the inference task is mainly handled by the GPU, the CPU only needs to allocate one thread to be responsible for scheduling the...

Embodiment 3

[0076] Embodiment 3 provides a computer-readable storage medium, the storage medium is used to store the computer software instructions used to implement the data processing flow optimization method described in the above embodiment 1, which includes instructions for performing the above-mentioned data processing flow optimization The program designed by the method; specifically, the executable program can be built into the data processing flow optimization system 100, so that the data processing flow optimization system 100 can realize the data processing of the first embodiment by executing the built-in executable program Process optimization methods.

[0077] In addition, the computer-readable storage medium provided in this embodiment may use any combination of one or more readable storage media, where the readable storage medium includes electrical, optical, electromagnetic, infrared or semiconductor systems, devices or devices, or Any combination of the above.

[0078] ...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

PUM

No PUM Login to View More

Abstract

The invention discloses a data processing flow optimization method, which comprises the following steps: 1, respectively adding identifiers for a preprocessing task, a reasoning task and a post-processing task; 2, respectively determining the running time lengths of the preprocessing task, the reasoning task and the post-processing task according to the identifiers, and according to the running duration, respectively estimating the number of threads needed by the preprocessing task, the reasoning task and the post-processing task; and 3, dividing the preprocessing task, the reasoning task and the post-processing task into a plurality of threads according to the number of the threads, and executing data processing operation through a plurality of threads without waiting. By means of the mode, optimization of the data processing flow is achieved, and the calculation efficiency and the hardware resource utilization rate are improved.

Description

technical field [0001] The present invention relates to the technical field of data processing, in particular to a data processing flow optimization method, system and storage medium. Background technique [0002] Data processing tasks include preprocessing tasks, inference tasks, and postprocessing tasks; in practical applications, the processing equipment of the prior art is a heterogeneous server of CPU+GPU; among them, the preprocessing tasks are completed by the CPU, and the inference tasks are completed by the GPU. The post-processing task is completed by CPU or GPU; while the existing processing flow is serial processing, the device is sequentially called for processing in the order of pre-processing task, reasoning task and post-processing task; the defect of the existing technology is that it needs to wait for the previous processing One task is completed, resulting in low processing efficiency and long processing time. Contents of the invention [0003] The tech...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More

Application Information

Patent Timeline
no application Login to View More
IPC IPC(8): G06F9/48G06F9/50G06T1/20
CPCG06F9/4881G06F9/5027G06T1/20
Inventor 王鹏飞
Owner SHANDONG YINGXIN COMP TECH CO LTD
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Patsnap Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Patsnap Eureka Blog
Learn More
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic, Popular Technical Reports.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap|About US| Contact US: help@patsnap.com