Pipeline type data processing method, equipment and storage medium
A data processing and pipeline technology, applied in the field of data processing, can solve problems such as difficult program maintenance, not easy, data processing logic errors, etc., and achieve the effect of solving difficult maintenance
Inactive Publication Date: 2020-05-12
SHANDONG INSPUR GENESOFT INFORMATION TECH CO LTD
View PDF3 Cites 3 Cited by
- Summary
- Abstract
- Description
- Claims
- Application Information
AI Technical Summary
Problems solved by technology
[0002] Today, with the rapid development of big data and cloud computing technology, in order to break the island of information, more sources of data are gathered into a unified display system for analysis and decision-making. Because these data come from different systems, usually a It is a good choice to use the processing program to normalize these data, but this is only applicable to the immutable system. With the rapid development of information technology today, new requirements are constantly being raised, and the immutable data normalization program has to face frequent revisions. More and more complex data processing logic is added to the program, making the program difficult to maintain; with the increase of data processing logic, the program begins to deal with performance problems, but it is necessary to find out which process the performance problem occurs in and Not easy; when an error occurs during data processing, it is impossible to know that there is a problem in executing the data processing logic that caused the error
Method used
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
View moreImage
Smart Image Click on the blue labels to locate them in the text.
Smart ImageViewing Examples
Examples
Experimental program
Comparison scheme
Effect test
Embodiment Construction
[0045] In order to make the object, technical solution and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings.
[0046] It should be noted that all expressions using "first" and "second" in the embodiments of the present invention are to distinguish two entities with the same name but different parameters or parameters that are not the same, see "first" and "second" It is only for the convenience of expression, and should not be construed as a limitation on the embodiments of the present invention, which will not be described one by one in the subsequent embodiments.
[0047] According to one aspect of the present invention, an embodiment of the present invention proposes a pipeline data processing method, such as figure 1 As shown, it may include steps: S1, create a pipeline section and a pipeline section descrip...
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More PUM
Login to View More Abstract
The invention discloses a pipeline type data processing method. The method comprises the steps of: creating pipeline sections and a pipeline section description file; acquiring to-be-processed data; determining the corresponding pipeline section description file according to the to-be-processed data; determining a plurality of pipeline sections for processing the to-be-processed data according tothe pipeline section description file; and processing the to-be-processed data by utilizing the plurality of pipeline sections. The invention further discloses computer equipment and a readable storage medium. According to the method provided by the invention, the pipeline composition and the processing circulation rule are described through the pipeline section description file, and the problemsthat a data processing program is difficult to maintain and the like are solved.
Description
technical field [0001] The invention relates to the field of data processing, in particular to a pipeline data processing method, device and storage medium. Background technique [0002] Today, with the rapid development of big data and cloud computing technology, in order to break the island of information, more sources of data are gathered into a unified display system for analysis and decision-making. Because these data come from different systems, usually a It is a good choice to use the processing program to normalize these data, but this is only applicable to the immutable system. With the rapid development of information technology today, new requirements are constantly being raised, and the immutable data normalization program has to face frequent revisions. More and more complex data processing logic is added to the program, making the program difficult to maintain; with the increase of data processing logic, the program begins to deal with performance problems, but...
Claims
the structure of the environmentally friendly knitted fabric provided by the present invention; figure 2 Flow chart of the yarn wrapping machine for environmentally friendly knitted fabrics and storage devices; image 3 Is the parameter map of the yarn covering machine
Login to View More Application Information
Patent Timeline
Login to View More Patent Type & Authority Applications(China)
IPC IPC(8): G06F8/70G06F8/73
CPCG06F8/70G06F8/73
Inventor 步显文
Owner SHANDONG INSPUR GENESOFT INFORMATION TECH CO LTD