A cross-platform adaptive data processing workflow system and method

A data processing and workflow technology, applied in the field of computer systems, can solve problems such as difficult debugging for performance users, and achieve high-performance cross-platform data processing, high scalability, and optimal global performance

Active Publication Date: 2022-08-02
SHANDONG COMP SCI CENTNAT SUPERCOMP CENT IN JINAN
View PDF15 Cites 0 Cited by
  • Summary
  • Abstract
  • Description
  • Claims
  • Application Information

AI Technical Summary

Problems solved by technology

[0006] (2) Users need to write additional script files to achieve collaboration between different platforms, such as data transmission, data format conversion, etc. This part of coding is an additional burden for data scientists, and many people still complete this part manually Work;
[0007] (3) The user needs to debug the computing platform of the algorithm to achieve the desired performance and effect. Part of the performance degradation is caused by the user code, but part of the reason is the platform limitation, and it is difficult for the user to debug this part of the performance

Method used

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View more

Image

Smart Image Click on the blue labels to locate them in the text.
Viewing Examples
Smart Image
  • A cross-platform adaptive data processing workflow system and method
  • A cross-platform adaptive data processing workflow system and method
  • A cross-platform adaptive data processing workflow system and method

Examples

Experimental program
Comparison scheme
Effect test

Embodiment 1

[0043] Embodiment 1 of the present application provides a cross-platform adaptive data processing workflow system.

[0044] A cross-platform adaptive data processing workflow system, comprising:

[0045] The client, configured to build workflows based on API calls and canvas drag-and-drop, pass the built workflows through Istio The security gateway sends it to the server;

[0046] The server is configured to receive the workflow constructed by the client based on the server interface, and optimize and adapt the computing environment of the workflow logic operator based on the operator computing platform adapter.

[0047] As one or more implementation manners, the cross-platform adaptive data processing workflow system adopts such as figure 1 The system architecture shown includes three layers: presentation layer, service layer, and data layer. Among them, the display layer provides API The list is called by the user and provides a visual drag-and-drop canvas front end. ...

Embodiment 2

[0059] The second embodiment of the present application provides a cross-platform adaptive data processing workflow method, which adopts the cross-platform adaptive data processing workflow system provided in the first embodiment.

[0060] User use based on Kubernetes The cross-platform adaptive data processing workflow method of the framework uses relevant operators in accordance with the framework-specific rules when constructing the workflow. In this embodiment, the operators include data object operators, logical operator operators, flow control operators, etc. The data object operators abstract the data to be processed into standardized data objects, which define data structures, data Scale, data path and other information; logical operator operator is a standardized encapsulation of commonly used data processing algorithms and data processing interfaces provided by various platforms. The operator only describes information such as function, usage, input and output rules...

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

PUM

No PUM Login to view more

Abstract

The present application belongs to the technical field of computer systems, and provides a cross-platform adaptive data processing workflow system and method, including a client, which is configured to construct a workflow based on application programming interface calling and canvas dragging, and convert the constructed workflow Workflow through Istio The security gateway is sent to the server; the server is configured to receive the workflow constructed by the client based on the server interface, and optimize and adapt the computing environment of the workflow logic operator based on the operator computing platform adapter. This application is based on Kubernetes microservices architecture, using Istio As the only channel between the client and the server, the security gateway realizes the processing of cross-platform adaptive data workflow.

Description

technical field [0001] The present application belongs to the technical field of computer systems, and in particular relates to a cross-platform adaptive data processing workflow system and method. Background technique [0002] The statements in this section merely provide background information related to the present application and do not necessarily constitute prior art. [0003] With the development of Internet information technology, people's production and life generate more and more complex data. In order to use these data more efficiently and mine the value existing in the data, people have studied a large number of algorithms such as data cleaning, data correction, data standardization, and data filling. However, there is currently no platform that can handle all data processing tasks. Data scientists often need to work across platforms and step by step when processing data, such as using relational databases to store structured data, graph data using graph computi...

Claims

the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to view more

Application Information

Patent Timeline
no application Login to view more
Patent Type & Authority Patents(China)
IPC IPC(8): G06F8/76G06F8/74
CPCG06F8/76G06F8/74
Inventor 杨美红王英龙张俭潘景山赵志刚王春晓武鲁杨帅李传涛
Owner SHANDONG COMP SCI CENTNAT SUPERCOMP CENT IN JINAN
Who we serve
  • R&D Engineer
  • R&D Manager
  • IP Professional
Why Eureka
  • Industry Leading Data Capabilities
  • Powerful AI technology
  • Patent DNA Extraction
Social media
Try Eureka
PatSnap group products

Browse by: Latest US Patents, China's latest patents, Technical Efficacy Thesaurus, Application Domain, Technology Topic.

© 2024 PatSnap. All rights reserved.Legal|Privacy policy|Modern Slavery Act Transparency Statement|Sitemap