Flow data processing method and device, computer equipment, readable storage medium and program product
Through parallel thread structured processing and standardized flow data, combined with time series model, the problem of low flow data processing efficiency is solved, and efficient data analysis and prediction is achieved.
Patent Information
- Application Number
- CN202411300215.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2025-01-03
AI Technical Summary
In the prior art, the flow data processing efficiency is low, making it difficult to effectively analyze flow data with high frequency, fine-grained and widely covered flow data.
The sub-flow data is structured through parallel threads, key indicator values are extracted, and sampled and standardized based on the preset step size to generate flow data processing results, and time series analysis is performed using a pre-trained data processing model.
It improves the efficiency of flow data analysis, reduces the data dimension, and can accurately extract key information from a large number of scattered data to generate high-quality flow data processing results.
Smart Images
Figure CN119250976A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of big data, and in particular to a pipeline data processing method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art
[0002] Streaming data can provide high-frequency, fine-grained and widely covered data information, but with the rapid development of big data technology and the increasing complexity of social and economic activities, it is becoming increasingly difficult to analyze and process streaming data.
[0003] At present, the relevant technologies for processing flow data mainly rely on market research and statistical data. Although these methods are effective, they are inefficient in processing flow data. Summary of the invention
[0004] Based on this, it is necessary to provide a pipeline data processing method, device, computer equipment, computer-readable storage medium and computer program product that can efficiently process pipeline data in response to the above-mentioned technical problems.
[0005] In a first aspect, the present application proposes a pipeline data processing method, the method comprising:
[0006] Acquire pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data;
[0007] Performing structured processing on a plurality of sub-pipeline data respectively through different parallel threads to obtain key indicator values;
[0008] Respectively obtaining each preset step length corresponding to each key indicator value, and sampling each key indicator value based on the preset step length to obtain standardized flow data;
[0009] Based on the standardized pipeline data, a pipeline data processing result is generated.
[0010] In one embodiment, the structured processing of the plurality of sub-pipeline data by different parallel threads to obtain key indicator values includes:
[0011] Each of the threads classifies each of the sub-pipeline data according to preset category parameters to obtain a plurality of pipeline data sets, and aggregates the sub-pipeline data in each of the pipeline data sets to obtain the key indicator value.
[0012] In one embodiment, the method further comprises:
[0013] A mapping relationship between the pipeline data to be processed and time is obtained based on the pipeline data processing result, and a target pipeline data processing result at a target time is obtained based on the mapping relationship.
[0014] In one embodiment, the mapping relationship between the pipeline data to be processed and time is obtained based on the pipeline data processing result, and the target pipeline data processing result at the target time is obtained based on the mapping relationship through a pre-set data processing model; the training process of the data processing model includes:
[0015] Acquire historical sample data; the historical sample data includes a number of time series flow data;
[0016] The historical sample data is input into an initial model, the initial model calculates the correlation between each of the time series flow data, and determines the parameters of the initial model according to the correlation to obtain the data processing model.
[0017] In one embodiment, the step of sampling each of the key indicator values based on the preset step size to obtain standardized flow data includes:
[0018] Sampling each of the key indicator values based on the preset step size to obtain a sampled indicator value;
[0019] Data standardization is performed on each of the sampling index values to obtain the standardized flow data.
[0020] In one embodiment, generating a pipeline data processing result based on the standardized pipeline data includes:
[0021] Weighting each of the standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result;
[0022] The initial pipeline processing result is denoised to obtain the pipeline data processing result.
[0023] In a second aspect, the present application proposes a pipeline data processing device, the device comprising:
[0024] An acquisition module is used to acquire pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data;
[0025] A structuring module, used for performing structured processing on the sub-flow data to obtain key indicator values;
[0026] A data standardization module, used for performing calculations based on the key indicator values to obtain an initial industry health;
[0027] The target calculation module is used to predict the initial industry health to obtain the target industry health.
[0028] In a third aspect, the present application proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.
[0029] In a fourth aspect, the present application proposes a computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the above method when executed by a processor.
[0030] In a fifth aspect, the present application proposes a computer program product, comprising a computer program, which implements the steps of the above method when executed by a processor.
[0031] The above-mentioned pipeline data processing method, device, computer equipment, computer-readable storage medium and computer program product extract multiple key indicators through different threads, and each thread is responsible for extracting key indicators of different dimensions. In addition, by performing structured processing on a number of sub-pipeline data, a large amount of scattered and isolated sub-pipeline data can be aggregated, and the data dimension can be reduced. Therefore, through multi-threading and dimensionality reduction processing, the analysis efficiency of the data to be processed can be greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 A schematic diagram of a flow chart of a pipeline data processing method in one embodiment;
[0034] Figure 2 A schematic diagram showing a result of pipeline data processing in one embodiment;
[0035] Figure 3 is a structural block diagram of a pipeline data processing device in one embodiment;
[0036] Figure 4 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0038] In one embodiment, Figure 1 As shown, a pipeline data processing method is provided. This embodiment takes the method applied to a terminal as an example for illustration. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0039] Step S102, obtaining pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data.
[0040] Optionally, the corresponding flow data to be processed is obtained according to the object to be analyzed. For example, the data can be filtered according to the specific analysis object. The analysis object can be a specific enterprise, industry, or time period. For example, assuming that the flow data of enterprise A is to be processed, the flow data of enterprise A can be filtered out from the flow data to be processed.
[0041] Optionally, the transaction data to be processed can be obtained from the corresponding database. The transaction data to be processed is usually stored in a relational database, and the transaction data to be processed is obtained by querying the relational database. It should be noted that the transaction data usually includes fields such as transaction period, transaction amount, transaction type, account information, etc. These data have clear rows and columns, which are very suitable for storage in relational databases, such as MySQL, PostgreSQL, etc. These databases are good at processing structured data and can well represent data in tabular form. In addition, relational databases provide a powerful SQL query language that can perform complex data screening, filtering, sorting and aggregation.
[0042] Optionally, the flow data to be processed includes multi-dimensional data, that is, a plurality of sub-flow data. Exemplarily, the flow data to be processed may include sales revenue flow data, salary payment flow data, and water and electricity fee flow data.
[0043] Step S104, performing structured processing on a plurality of sub-pipeline data respectively through different parallel threads to obtain key indicator values.
[0044] Since there are a large number of scattered and isolated sub-flow data in the flow data to be processed, the sub-flow data will be structured in this embodiment so as to extract key indicators that can characterize the flow data to be processed from the flow data to be processed.
[0045] Optionally, since each sub-pipeline data is a relatively independent subset suitable for parallel processing, in this embodiment, each sub-pipeline data is further structured processed through different parallel threads to obtain key indicator values.
[0046] Furthermore, the key indicator values may include multiple ones, respectively evaluating different dimensions. For example, the key indicator values may include a water flow direction indicator value, a water flow index value, etc. Therefore, each thread may be responsible for processing the extraction of key indicator values of different dimensions. The tasks to be performed by each thread may be dynamically assigned to the thread by the task scheduler.
[0047] Optionally, each thread may classify the sub-pipeline data, and calculate the sub-pipeline data belonging to each category to obtain key indicator values that can characterize each dimension.
[0048] Step S106, respectively obtaining the preset step lengths corresponding to the key indicator values, and respectively sampling the key indicator values based on the preset step lengths to obtain standardized flow data.
[0049] The preset step size refers to the interval set in the time or data dimension, which is used for timing or quantitative sampling. The time step size can be daily, weekly, or monthly; the data volume step size can collect key indicators once every N transaction records or fund flows.
[0050] Optionally, for each sub-flow data, based on a preset step length, each key indicator value is obtained respectively. Within each step length interval, the key indicator value within the current time period or data block is calculated.
[0051] Optionally, the purpose of standardization is to convert indicators with different scales, units or distributions into data with the same scale so that reasonable analysis, comparison and modeling can be carried out.
[0052] Optionally, the normalization process may be indexation, minimum-maximum normalization, Z-Score normalization, etc.
[0053] Step S108, generating a pipeline data processing result based on the standardized pipeline data.
[0054] Optionally, the result of the flow data processing may be a risk assessment result or a flow data trend analysis.
[0055] Optionally, risk assessment results for flow data can be generated by weighting the standardized key indicators.
[0056] Optionally, the future inflow trend can be predicted using a time series model by normalizing the data.
[0057] In the above embodiment, multiple key indicators are extracted through different threads, and each thread is responsible for extracting key indicators of different dimensions. In addition, by performing structured processing on a number of sub-pipeline data, a large amount of scattered and isolated sub-pipeline data can be aggregated, and the data dimension can be reduced. Therefore, through multi-threading and dimensionality reduction processing, the analysis efficiency of the data to be processed can be greatly improved.
[0058] In one embodiment, the above-mentioned structured processing of several sub-pipeline data is performed by different parallel threads to obtain key indicator values, including: each thread classifies each sub-pipeline data according to preset category parameters to obtain several pipeline data sets, and aggregates the sub-pipeline data in each pipeline data set to obtain the key indicator value.
[0059] In the process of structuring, each thread classifies each sub-pipeline data according to preset category parameters.
[0060] For example, thread A is responsible for calculating the flow direction index value, and thread B is responsible for the flow rate index value. Then, thread A performs structured processing on the sub-flow data according to the preset category parameters of the flow direction index value, and thread B performs structured processing on the sub-flow data according to the preset category parameters of the flow rate index value.
[0061] Optionally, each thread may read each sub-flow data in the flow data to be processed by a label, and classify the sub-flow data according to the label of each sub-flow data and a preset category parameter to obtain a flow data set corresponding to each key indicator value.
[0062] Exemplarily, assuming that the classification is performed according to the preset category parameters of the flow direction index value, the preset category parameters may be the inflow and outflow flow data of each associated party. The thread calculating the flow direction index value obtains the associated party inflow flow data and the associated party outflow flow data according to the category parameters.
[0063] After the sub-pipeline data are classified, each pipeline data set still includes multiple data. Therefore, in this embodiment, the sub-pipeline data in each pipeline data set are also aggregated to obtain key indicator values that can characterize the pipeline data set. The goal of aggregation is to calculate the key indicator values that can effectively characterize the set by comprehensively analyzing all the sub-pipeline data in the data set.
[0064] Optionally, the sub-flow data may be aggregated by comprehensive calculation, mean calculation, standard deviation, etc. Exemplarily, the comprehensive inflow flow data and the sum of outflow flow data may be calculated respectively, and then the difference between the inflow flow data and the outflow flow data may be calculated, and the difference may be used as the flow direction indicator value.
[0065] Exemplarily, key indicators may include execution capability indicators, outcome output indicators, resource allocation efficiency indicators, and associated entity retention indicators.
[0066] Among them, the execution capability index is used to measure the ability and efficiency of task completion objectives. For example, in the manufacturing industry, the execution capability index may include the output and production time of the production line. For example, at the economic level, the execution capability may affect the production cost and profit margin of the enterprise.
[0067] Exemplarily, the flow data set includes inflow data related to execution capability and outflow data related to execution capability. Among them, the inflow data related to execution capability reflects the inflow of funds or resources related to the execution task, such as raw materials or labor received in production activities. The outflow data related to execution capability reflects the outflow of funds or resources related to the execution task, such as raw materials consumed in the production process or labor costs paid. Further, a first difference is obtained based on the inflow data related to execution capability and the outflow data related to capability, and the ratio of the first difference to the inflow data related to execution capability is calculated to obtain the execution capability index.
[0068] Among them, output indicators are used to measure the actual output of the production process compared with the expected goals or standards. In the manufacturing industry, output indicators can include the completion of the project and the quality of the final deliverables. The completion of the project refers to the degree of realization of the production task, while the quality of the final deliverable reflects the degree to which the product meets the standards. At the economic level, output indicators affect revenue and profitability.
[0069] Exemplarily, the flow data set includes inflow data related to the output of achievements and outflow data related to the output of achievements. Among them, the inflow data related to the output of achievements reflects the inflow of funds or resources related to the output of achievements, such as the funds, materials or equipment required for the project. The outflow data related to the output of achievements reflects the outflow of funds or resources related to the output of achievements, such as the materials consumed in the production process, the labor costs paid or the equipment maintenance costs. Further, a second difference is obtained based on the inflow data related to the output of achievements and the outflow data related to the output of achievements, and the ratio of the second difference to the inflow data related to the output of achievements is calculated to obtain the output of achievements index.
[0070] Among them, the resource allocation efficiency index measures the efficiency of the company's investment and income management of financial assets after the accumulation of profits. The company invests surplus funds in financial products, bonds or other financial instruments in order to obtain additional income. This indicator does not involve the use of raw materials and labor, but focuses on the efficiency and return of the company's allocation of financial assets.
[0071] Exemplarily, the flow data set includes financial subscription flow data and financial redemption flow data. Among them, financial subscription flow data reflects the capital investment of the enterprise for financial asset allocation, such as the principal expenditure for purchasing financial products or other financial investment tools. Financial redemption flow data reflects the funds recovered by the enterprise from financial assets, including the principal plus income after the maturity of financial products. Furthermore, by calculating the difference between the financial subscription amount and the financial redemption amount, the net financial purchase amount is obtained, and the net financial purchase amount is calculated by the ratio of the total capital inflow generated by operating activities to obtain the resource allocation efficiency index, so as to evaluate the investment return efficiency obtained by the enterprise through financial asset allocation.
[0072] Among them, the related entity retention index is used to evaluate the retention and continued cooperation of key entities that have direct or indirect control relationships with manufacturing enterprises. In the manufacturing industry, the related entity retention index can include the transfer rate of manufacturing enterprises to related entities and the related entity retention rate. The related entity transfer rate refers to the excess profits of manufacturing enterprises paid to related entities in various ways, so that they are separated from the manufacturing enterprise system and exist independently in the related entities, while the related entity retention rate reflects the situation in which the excess profits are retained in the related entities for a long time after leaving the manufacturing enterprises, forming the external capital accumulation of the manufacturing enterprises. At the economic level, the related entity retention index reflects the transfer and accumulation of excess profits of manufacturing enterprises in non-financial aspects.
[0073] Exemplarily, the flow data set includes associated entity related inflow data and associated entity related outflow data. Among them, the associated entity related inflow data reflects the inflow of resources or funds brought about by cooperation with the associated entity, such as the transfer of funds from the family of the business owner. The associated entity related outflow data reflects the outflow of funds or resources related to the associated entity, such as the funds obtained by the business owner from the business or the profit share distributed to the partners. Further, a fourth difference is obtained based on the associated entity related inflow data and the associated entity related outflow data, and the ratio of the fourth difference to the output-related inflow data is calculated to obtain the associated entity retention index.
[0074] In the above embodiment, by performing aggregation processing on the classified data sets, the features of each set can be accurately extracted from each flow data set.
[0075] In one embodiment, the method further includes: obtaining a mapping relationship between the pipeline data to be processed and time based on the pipeline data processing result, and obtaining a target pipeline data processing result at a target time based on the mapping relationship.
[0076] Among them, the target time refers to a future time period, which can be the next day, the next week, or the next month, and is set according to the actual usage scenario; the target flow data processing result refers to the flow processing result corresponding to the future target time period.
[0077] Optionally, the pipeline data processing results are input into a pre-trained data processing model, and the data processing model obtains a mapping relationship between the pipeline data to be processed and time based on the pipeline data processing results, and obtains the target pipeline data processing results at the target time based on the mapping relationship.
[0078] In this way, valuable prediction information can be extracted from time series data to obtain the target flow data processing results.
[0079] Among them, the training process of the data processing model includes: obtaining historical sample data; the historical sample data includes several time series flow data; the historical sample data is input into the initial model, the initial model calculates the correlation between each time series flow data, and determines the parameters of the initial model according to the correlation to obtain the data processing model.
[0080] Optionally, the historical sample data is input into the initial model, which calculates the correlation between the time series and adjusts the parameters of the initial model according to the calculated correlation. After the parameters are adjusted, the initial model is converted into the final data processing model, which can accurately capture the relationship between the data.
[0081] Among them, the initial model can be an AR (autoregressive), MA (moving average) and ARMA (autoregressive moving average) model.
[0082] For example, if the initial model is an AR model, after the historical flow data is input into the AR model, the autoregressive order p is selected. The autoregressive order p indicates how many past time points are used for prediction. Then, the model parameters are estimated by the least squares method or other optimization algorithms to minimize the error of the model fitting the historical data. The training goal is to find the optimal parameters so that the model can accurately predict future data points.
[0083] For example, if the initial model is an MA model, after the historical flow data is input into the MA model, the moving average order q is selected. The moving average order indicates how many past error terms are used for prediction. Finally, the parameters are estimated by minimizing the sum of squares of the prediction errors, correcting the impact of historical errors and optimizing the model fitting effect.
[0084] For example, if the initial model is an ARMA model, after inputting the historical flow data into the ARMA model, the autoregressive order p and the moving average order q are selected to simultaneously consider the influence of historical data and errors. Finally, the maximum likelihood estimation or other optimization algorithms are used to estimate the parameters and find the optimal model parameters so that the model can achieve the best fit to the historical data.
[0085] In this way, by training the model to accurately predict future pipeline data processing results, users can better utilize the potential information of time series data in data analysis and decision-making.
[0086] Furthermore, in one embodiment, the above-mentioned sampling of each key indicator value based on a preset step size to obtain standardized flow data includes: sampling each key indicator value based on a preset step size to obtain a sampled indicator value; and performing data standardization on each sampled indicator value to obtain standardized flow data.
[0087] Exemplarily, each key indicator value is sampled at a preset time step. For example, if the preset time step is monthly, the key indicator values are extracted month by month to obtain sampled indicator values.
[0088] Optionally, the data normalization method used in this embodiment is indexation processing. Exemplarily, the specific formula is shown in combination with formula (1).
[0089]
[0090] Among them, β ki is the standardized flow data corresponding to the i-th preset time step, α ki is the key indicator value corresponding to the i-th preset time step. For example, if the preset step is one month, then β ki It is the standardized flow data corresponding to the i-th preset time step.
[0091] In the above embodiment, through the process of sampling and standardization, the key indicator values are converted into standardized flow data with consistent characteristics to avoid deviations in the calculation results.
[0092] Furthermore, in one embodiment, the above-mentioned generation of pipeline data processing results based on standardized pipeline data includes: weighting each standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result; denoising the initial pipeline processing result to obtain a pipeline data processing result.
[0093] Optionally, each standardized flow data is weighted according to a preset weight. The weight corresponding to each standardized flow data is different and is set according to the actual usage scenario.
[0094] Exemplarily, weighting can be performed according to formula (2), and the specific formula is as follows:
[0095]
[0096] Among them, CSI iis the initial pipeline data processing result corresponding to the i-th preset time step, k is the k-th functional index, δ k is the weight.
[0097] Optionally, the denoising used in this embodiment reduces data fluctuations through a smoothing index, which is essentially to reduce short-term fluctuations or abnormal changes in the time series, making the overall trend more stable.
[0098] Exemplarily, denoising is performed according to formula (3), and the specific formula is shown in (3):
[0099]
[0100] Among them, CSIΔ is the smoothing index.
[0101] In the above embodiment, by smoothing the index and reducing the average deviation, short-term random fluctuations can be eliminated and the data can be made smoother.
[0102] In one embodiment, the pipeline data processing result is normalized, and the normalized pipeline data processing result is displayed.
[0103] Optional CSI i * Zoom in closer to 100 to obtain a CSI value that is easy to observe, and draw an image of the pipeline data processing results.
[0104] Exemplary, combined Figure 2 As shown, Figure 2 FIG. 1 is a schematic diagram showing the results of pipeline data processing in an embodiment. Figure 2 As shown, the CSI calculated by the present application is more accurate than the control data.
[0105] Based on the same inventive concept, the embodiment of the present application also provides a pipeline processing device for implementing the pipeline data processing method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more pipeline data processing device embodiments provided below can refer to the limitations of the pipeline data processing method above, and will not be repeated here.
[0106] In an exemplary embodiment, Figure 3 As shown, a pipeline data processing device is provided, comprising: an acquisition module, a structuring module, a data standardization module and a target calculation module, wherein:
[0107] The acquisition module 310 is used to acquire pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data.
[0108] The structuring module 320 is used to perform structured processing on the plurality of sub-pipeline data respectively through different parallel threads to obtain key indicator values.
[0109] The data standardization module 330 is used to respectively obtain the preset step lengths corresponding to the key indicator values, and to sample the key indicator values based on the preset step lengths to obtain standardized flow data.
[0110] The target calculation module 340 is used to generate a pipeline data processing result based on the standardized pipeline data.
[0111] In one embodiment, the structured module includes:
[0112] The extraction unit is used for each of the threads to classify each of the sub-pipeline data according to preset category parameters to obtain a plurality of pipeline data sets, and to aggregate the sub-pipeline data in each of the pipeline data sets to obtain the key indicator value.
[0113] In one embodiment, the above device further comprises:
[0114] The prediction module is used to obtain the mapping relationship between the pipeline data to be processed and time based on the pipeline data processing result, and obtain the target pipeline data processing result at the target time based on the mapping relationship.
[0115] In one embodiment, the prediction module includes:
[0116] The sample data acquisition unit is used to acquire historical sample data; the historical sample data includes a number of time series flow data.
[0117] The training unit is used to input the historical sample data into the initial model. The initial model calculates the correlation between the flow data of each time series, and determines the parameters of the initial model according to the correlation to obtain the data processing model.
[0118] In one embodiment, the data standardization module includes:
[0119] The sampling unit is used to sample each key indicator value based on a preset step size to obtain a sampled indicator value.
[0120] The standardization unit is used to standardize the data of each sampling indicator value to obtain standardized flow data.
[0121] In one embodiment, the target calculation module includes:
[0122] The weighting unit is used to weight each standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result.
[0123] The denoising unit is used to denoise the initial pipeline processing result to obtain the pipeline data processing result.
[0124] Each module in the above-mentioned pipeline data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each module above.
[0125] In an exemplary embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as shown in FIG. Figure 4 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store pipeline data to be processed. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a pipeline data processing method is implemented.
[0126] Those skilled in the art will understand that Figure 4 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0127] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data; performing structured processing on the plurality of sub-pipeline data through different parallel threads to obtain key indicator values; obtaining preset step sizes corresponding to the key indicator values respectively, and sampling the key indicator values based on the preset step sizes to obtain standardized pipeline data; and generating pipeline data processing results based on the standardized pipeline data.
[0128] In one embodiment, when the processor executes the computer program, the following steps are further implemented: each thread classifies each sub-pipeline data according to a preset category parameter to obtain a plurality of pipeline data sets, and aggregates the sub-pipeline data in each pipeline data set to obtain a key indicator value.
[0129] In one embodiment, when the processor executes the computer program, the following steps are also implemented: based on the pipeline data processing result, a mapping relationship between the pipeline data to be processed and time is obtained, and based on the mapping relationship, a target pipeline data processing result at the target time is obtained.
[0130] In one embodiment, when the processor executes the computer program, the following steps are also implemented: obtaining historical sample data; the historical sample data includes a number of time series flow data; the historical sample data is input into the initial model, the initial model calculates the correlation between each time series flow data, and determines the parameters of the initial model based on the correlation to obtain a data processing model.
[0131] In one embodiment, when the processor executes the computer program, the following steps are also implemented: sampling each key indicator value based on a preset step size to obtain a sampled indicator value; and performing data standardization on each sampled indicator value to obtain standardized flow data.
[0132] In one embodiment, when the processor executes the computer program, the following steps are also implemented: weighting each standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result; denoising the initial pipeline processing result to obtain a pipeline data processing result.
[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data; performing structured processing on the plurality of sub-pipeline data through different parallel threads to obtain key indicator values; obtaining preset step sizes corresponding to the key indicator values respectively, and sampling the key indicator values based on the preset step sizes to obtain standardized pipeline data; and generating pipeline data processing results based on the standardized pipeline data.
[0134] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: each thread classifies each sub-pipeline data according to preset category parameters to obtain a number of pipeline data sets, and aggregates the sub-pipeline data in each pipeline data set to obtain a key indicator value.
[0135] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the pipeline data processing result, a mapping relationship between the pipeline data to be processed and time is obtained, and based on the mapping relationship, a target pipeline data processing result at the target time is obtained.
[0136] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining historical sample data; the historical sample data includes a number of time series flow data; the historical sample data is input into the initial model, the initial model calculates the correlation between each time series flow data, and determines the parameters of the initial model based on the correlation to obtain a data processing model.
[0137] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sampling each key indicator value based on a preset step size to obtain a sampled indicator value; and performing data standardization on each sampled indicator value to obtain standardized flow data.
[0138] In one embodiment, when the computer program is executed by the processor, the following steps are further implemented: weighting each standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result; denoising the initial pipeline processing result to obtain a pipeline data processing result. In one embodiment, when the computer program is executed by the processor, the following steps are further implemented:
[0139] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data; performing structured processing on the plurality of sub-pipeline data through different parallel threads to obtain key indicator values; obtaining preset step sizes corresponding to the key indicator values respectively, and sampling the key indicator values based on the preset step sizes to obtain standardized pipeline data; and generating pipeline data processing results based on the standardized pipeline data.
[0140] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: each thread classifies each sub-pipeline data according to preset category parameters to obtain a number of pipeline data sets, and aggregates the sub-pipeline data in each pipeline data set to obtain a key indicator value.
[0141] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: based on the pipeline data processing result, a mapping relationship between the pipeline data to be processed and time is obtained, and based on the mapping relationship, a target pipeline data processing result at the target time is obtained.
[0142] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: obtaining historical sample data; the historical sample data includes a number of time series flow data; the historical sample data is input into the initial model, the initial model calculates the correlation between each time series flow data, and determines the parameters of the initial model based on the correlation to obtain a data processing model.
[0143] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: sampling each key indicator value based on a preset step size to obtain a sampled indicator value; and performing data standardization on each sampled indicator value to obtain standardized flow data.
[0144] In one embodiment, when the computer program is executed by the processor, the following steps are also implemented: weighting each standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result; denoising the initial pipeline processing result to obtain a pipeline data processing result. A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to a memory, a database or other medium used in the embodiments provided in the present application may include at least one of a non-volatile memory and a volatile memory. Non-volatile memory may include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory may include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but is not limited thereto.
[0145] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0146] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A pipeline data processing method, characterized in that: The method comprises: Acquire pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data; Performing structured processing on a plurality of sub-pipeline data respectively through different parallel threads to obtain key indicator values; Respectively obtaining each preset step length corresponding to each key indicator value, and sampling each key indicator value based on the preset step length to obtain standardized flow data; Based on the standardized pipeline data, a pipeline data processing result is generated.
2. The method according to claim 1, characterized in that The structured processing of the plurality of sub-pipeline data is performed by different parallel threads to obtain key indicator values, including: Each of the threads classifies each of the sub-pipeline data according to preset category parameters to obtain a plurality of pipeline data sets, and aggregates the sub-pipeline data in each of the pipeline data sets to obtain the key indicator value.
3. The method according to claim 1, characterized in that The method further comprises: A mapping relationship between the pipeline data to be processed and time is obtained based on the pipeline data processing result, and a target pipeline data processing result at a target time is obtained based on the mapping relationship.
4. The method according to claim 3, characterized in that The mapping relationship between the pipeline data to be processed and time is obtained based on the pipeline data processing result, and the target pipeline data processing result at the target time is obtained based on the mapping relationship through a preset data processing model; The training process of the data processing model includes: Acquire historical sample data; the historical sample data includes a number of time series flow data; The historical sample data is input into an initial model, the initial model calculates the correlation between each of the time series flow data, and determines the parameters of the initial model according to the correlation to obtain the data processing model.
5. The method according to claim 1, characterized in that: The step of sampling each of the key indicator values based on the preset step length to obtain standardized flow data includes: Sampling each of the key indicator values based on the preset step size to obtain a sampled indicator value; Data standardization is performed on each of the sampling index values to obtain the standardized flow data.
6. The method according to claim 1, characterized in that The step of generating a pipeline data processing result based on the standardized pipeline data includes: Weighting each of the standardized pipeline data according to a preset weight to obtain an initial pipeline data processing result; The initial pipeline processing result is denoised to obtain the pipeline data processing result.
7. A pipeline data processing device, characterized in that: The device comprises: An acquisition module is used to acquire pipeline data to be processed; the pipeline data to be processed includes a plurality of sub-pipeline data; A structuring module, used for performing structured processing on a plurality of sub-pipeline data respectively through different parallel threads to obtain key indicator values; A data standardization module, used to respectively obtain each preset step length corresponding to each key indicator value, and sample each key indicator value based on the preset step length to obtain standardized flow data; The target calculation module is used to generate a pipeline data processing result based on the standardized pipeline data.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.