Device data stream configuration method, device, device and storage medium

Abstract

The present application discloses a device data stream configuration method, apparatus, equipment and storage medium, which relates to the field of data stream processing technology, and is applied to a data stream engine including a preset resource monitoring center, a preset load analyzer and a preset resource scheduler, including: monitoring a target device to obtain indicators of the device to be processed; processing the indicators of the device to be processed using a target smoothing formula constructed based on the characteristics of several initial smoothing processing algorithms, and analyzing the obtained target device indicators to obtain resource analysis results; analyzing the resource analysis results to predict the load of the target device using a target load learning model constructed using the obtained time series analysis results to obtain load prediction results; processing the load prediction results to obtain an initial data stream, and adjusting the initial data stream to obtain a target data stream, so as to configure the target device using the target data stream. This can improve the data processing speed of the device.

Description

Technical Field

[0001] The present invention relates to the technical field of data stream processing, and in particular to a device data stream configuration method, apparatus, device and storage medium. Background Art

[0002] With the advent of the big data era, data stream processing systems are facing an increasing amount of data and increasingly complex data processing requirements. At present, existing data stream processing systems usually adopt static resource allocation strategies and cannot dynamically adjust resource allocation according to real-time data load and processing requirements, resulting in resource waste and processing bottlenecks. As a result, the system lacks the ability to adapt to different application scenarios and cannot be flexibly adjusted according to actual needs.

[0003] As can be seen from the above, how to improve the data processing speed of the device during the device data flow configuration process is a problem that needs to be solved urgently. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a device data flow configuration method, apparatus, device and storage medium, which can improve the data processing speed and data processing efficiency of the device. The specific scheme is as follows:

[0005] In a first aspect, the present application provides a device data flow configuration method, which is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer, and a preset resource scheduler, and the method includes:

[0006] Calling the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed;

[0007] Based on the characteristics of several initial smoothing processing algorithms, a target smoothing formula is constructed, the target smoothing formula is used to process the device index to be processed, and the obtained target device index is sent to a preset processing center in the preset resource monitoring center for analysis to obtain a resource analysis result;

[0008] Calling a preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then using the target load learning model to predict the load condition of the target device to obtain a load prediction result;

[0009] The load prediction result is processed using a preset deep learning model to obtain an initial data stream, and the initial data stream is adjusted using a preset data stream adjustment algorithm to obtain a target data stream, so that the preset resource scheduler configures the target device using the target data stream.

[0010] Optionally, the calling of the preset resource monitoring center to monitor the device indicator corresponding to the target device to obtain the device indicator to be processed includes:

[0011] The preset monitoring tool in the preset resource monitoring center is called to monitor the device indicators corresponding to the device to obtain the device indicators to be processed; the device indicators include CPU usage, memory usage, storage I / O and network bandwidth.

[0012] Optionally, constructing a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, and processing the device indicator to be processed using the target smoothing formula includes:

[0013] Based on the algorithm characteristics of the moving average algorithm and the exponential smoothing algorithm, and in combination with the current moment, the index of the device to be processed, the number of indicators of the device to be processed, and the indicator weight, a target smoothing formula is constructed to process the device index to be processed using the target smoothing formula;

[0014] Among them, the value corresponding to the current moment is positively correlated with the indicator weight.

[0015] Optionally, the using the target smoothing formula to process the target device indicator, and sending the obtained target device indicator to a preset processing center in the preset resource monitoring center for analysis, includes:

[0016] Processing the device index to be processed by using the target smoothing formula to obtain a first intermediate device index;

[0017] Using a preset polynomial merging rule, the quadratic polynomial and the cubic polynomial in the initial formula corresponding to the preset interpolation algorithm are merged to obtain a 2.5-power polynomial;

[0018] Combining the 2.5-power polynomial with the initial formula according to a preset formula combination rule to obtain a target combination formula, and using the target combination formula to calculate the first intermediate device indicator to obtain a second intermediate device indicator;

[0019] The data type corresponding to the second intermediate state device indicator is converted from time domain data to frequency domain data using a preset Fourier transform formula to obtain the target device indicator, so as to send the target device indicator to a preset processing center in the preset resource monitoring center for analysis.

[0020] Optionally, calling a preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then using the target load learning model to predict the load of the target device to obtain a load prediction result, includes:

[0021] Calling a preset input gate, a preset forget gate, and a preset output gate in a preset time series analysis model in the preset load analyzer to analyze the resource analysis result to obtain a time series analysis result;

[0022] The time series analysis result is used to train the initial load learning model to obtain a target load learning model, and the target load learning model is used to predict the load condition of the target device to obtain a load prediction result.

[0023] Optionally, the using a preset deep learning model to process the load prediction result to obtain an initial data stream, and using a preset data stream adjustment algorithm to adjust the initial data stream to obtain a target data stream, so that the preset resource scheduler uses the target data stream to configure the target device, including:

[0024] Generate a plurality of initial data streams using a preset deep learning model and based on the load prediction result issued by the preset load analyzer, and evaluate each of the initial data streams using a preset evaluation model to obtain a corresponding evaluation result, and then determine whether the evaluation result is greater than a preset threshold;

[0025] If the evaluation result is greater than the preset threshold, the data stream corresponding to the evaluation result is set as the target data stream, so that the preset resource scheduler configures the target device using each of the target data streams;

[0026] If the evaluation result is not greater than the preset threshold, the preset gradient descent method is used to adjust the corresponding initial data stream based on each evaluation result, and then the process jumps to the step of evaluating each initial data stream using the preset evaluation model to obtain the corresponding evaluation result, and then determining whether the evaluation result is greater than the preset threshold.

[0027] Optionally, the adjusting the initial data stream by using a preset data stream adjustment algorithm to obtain a target data stream, so that the preset resource scheduler configures the target device by using the target data stream, includes:

[0028] Each of the initial data streams is adjusted using a preset data stream adjustment algorithm to obtain a corresponding target data stream, and the preset load analyzer is called to send each of the target data streams to the preset resource scheduler, so that the preset resource scheduler determines the corresponding type and quantity of computer resources based on each of the target data streams, and configures the target device based on the type and quantity of computer resources.

[0029] In a second aspect, the present application provides a device data flow configuration apparatus, which is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer, and a preset resource scheduler, and the apparatus includes:

[0030] The device indicator determination module is used to call the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed;

[0031] A resource analysis result determination module is used to construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the device indicators to be processed, and send the obtained target device indicators to a preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results;

[0032] A load prediction result determination module, used to call a preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then use the target load learning model to predict the load of the target device to obtain a load prediction result;

[0033] A data flow adjustment module is used to process the load prediction results using a preset deep learning model to obtain an initial data flow, and to adjust the initial data flow using a preset data flow adjustment algorithm to obtain a target data flow, so that the preset resource scheduler can configure the target device using the target data flow.

[0034] In a third aspect, the present application provides an electronic device, including:

[0035] Memory, used to store computer programs;

[0036] The processor is used to execute the computer program to implement the aforementioned device data flow configuration method.

[0037] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program, wherein the computer program implements the aforementioned device data flow configuration method when executed by a processor.

[0038] As can be seen from the above, before configuring the device data flow, the present application needs to call the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed; construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the device indicators to be processed, and send the obtained target device indicators to the preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results; call the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to use the obtained time series analysis results to construct a target load learning model, and then use the target load learning model to predict the load of the target device to obtain a load prediction result; use the preset deep learning model to process the load prediction result to obtain an initial data flow, and use the preset data flow adjustment algorithm to adjust the initial data flow to obtain a target data flow, so that the preset resource scheduler can use the target data flow to configure the target device.

[0039] It can be seen that the present application first monitors the device indicators corresponding to the target device by calling the preset resource monitoring center to obtain the device indicators to be processed; then, constructs a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, uses the target smoothing formula to process the device indicators to be processed, and sends the obtained target device indicators to the preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results; further, calls the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to construct a target load learning model using the obtained time series analysis results, and then uses the target load learning model to predict the load of the target device to obtain the load prediction results; uses the preset deep learning model to process the load prediction results to obtain the initial data stream. Finally, the preset data stream adjustment algorithm is used to adjust the initial data stream to obtain the target data stream, so that the preset resource scheduler uses the target data stream to configure the target device. In this way, the data processing speed and efficiency of the device are improved, thereby further enhancing the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0041] Figure 1 A flow chart of a device data stream configuration method disclosed in this application;

[0042] Figure 2 A schematic diagram of a specific device data flow configuration process disclosed in this application;

[0043] Figure 3 A schematic diagram of a process of calling a preset load analyzer to process indicators of a device to be processed disclosed in the present application;

[0044] Figure 4 A schematic diagram of a process for optimizing a data stream using a deep learning model disclosed in the present application;

[0045] Figure 5 This is a schematic diagram of the structure of a device data flow configuration apparatus disclosed in this application;

[0046] Figure 6 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0048] With the advent of the big data era, data stream processing systems are facing an increasing amount of data and increasingly complex data processing requirements. At present, existing data stream processing systems usually adopt static resource allocation strategies, which cannot dynamically adjust resource allocation according to real-time data load and processing requirements, resulting in resource waste and processing bottlenecks, which leads to the system's lack of adaptability to different application scenarios and inability to flexibly adjust according to actual needs. To this end, the present application provides a device data stream configuration method that can improve the data processing speed of the device during the device data stream configuration process.

[0049] See also Figure 1 As shown, an embodiment of the present invention discloses a device data flow configuration method, which is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer and a preset resource scheduler, and the method includes:

[0050] Step S11: calling the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed.

[0051] In this embodiment, a preset resource monitoring center (resource monitoring module), a preset load analyzer, and a preset resource scheduler are introduced to configure the data flow of the device, and the flow diagram is as follows: Figure 2As shown: First, the preset resource monitoring center can monitor the device information of the target device and send the obtained device information to be processed to the preset load analyzer. Then, the preset load analyzer is called to perform load analysis on the device information, obtain the load analysis result, and determine the target data flow based on the load analysis result. Finally, the resource scheduler is called and the computer resources of the target device are configured based on the target data flow.

[0052] Furthermore, the preset resource monitoring center relies on the use of pre-integrated monitoring tools to collect indicators such as the CPU (Central Processing Unit) usage, memory usage, storage I / O (Input / Output Equipment) and network bandwidth of the target device in real time. Subsequently, the above indicators are sent to the designated preset central processing center to further analyze the overall situation of the computer resources of the target device using the preset central processing center. Specifically, calling the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed may include: calling the preset monitoring tool in the preset resource monitoring center to monitor the device indicators corresponding to the device to obtain the device indicators to be processed; the device indicators include CPU usage, memory usage, storage I / O and network bandwidth.

[0053] In a specific implementation, for a single CPU core, the calculation formula for CPU usage is as follows:

[0054] ;

[0055] in, It refers to the time actually spent by the CPU to process tasks within the preset time. is the overall sampling time. For a multi-core CPU, the overall CPU utilization can be calculated by calculating the average utilization of all cores, and the calculation formula is as follows:

[0056] ;

[0057] in, is the utilization rate of each CPU, is the identifier corresponding to the CPU, is the number of CPUs. In addition, the calculation formulas for memory usage and I / O are the same as that for CPU usage.

[0058] Step S12: construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the equipment indicators to be processed, and send the obtained target equipment indicators to the preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results.

[0059] In this embodiment, the load analyzer predicts the load pattern of the data stream based on the time series and machine learning model. The analyzer can identify data peaks and valleys as well as potential anomalies. Time series analysis mainly analyzes and extracts patterns and trends in time series. Based on the results of time series analysis, a corresponding machine learning model can be constructed to predict future load conditions, which is of great significance for planning resource allocation in advance and avoiding performance bottlenecks. The flow diagram of processing the indicators of the pending devices in the preset load analyzer is shown in the figure below. Figure 3 As shown. First, the indicators of the equipment to be processed are preprocessed to improve data quality. Specifically, the target smoothing formula is constructed based on the characteristics of several initial smoothing processing algorithms, and the target smoothing formula is used to process the indicators of the equipment to be processed, which may include: based on the algorithm characteristics of the moving average algorithm and the exponential smoothing algorithm, and combined with the current moment, the indicators of the equipment to be processed, the number of indicators of the equipment to be processed, and the indicator weight, a target smoothing formula is constructed to process the indicators of the equipment to be processed using the target smoothing formula; wherein the numerical value corresponding to the current moment is positively correlated with the indicator weight.

[0060] In a specific implementation, the preprocessing steps used in this embodiment include two methods: smoothing and missing value processing. Among them, the smoothing process mainly uses the moving average method and the exponential smoothing method to remove noise in the data, and the moving average method is adopted. Before the moment The data is processed and the expression is as follows:

[0061] ;

[0062] in, is the estimated data at time t, is the number of device indicators, The data corresponding to the current moment.

[0063] Furthermore, the specific formula expression of exponential smoothing is as follows:

[0064] ;

[0065] in, , moving average and exponential smoothing methods, for Real data at all times, for Real data at all times, , They are , The weight coefficient can be set based on user needs.

[0066] It is worth mentioning that since the moving average method takes into account Before the moment data, but not considered Before the moment The influence of each data on the result is different, and the exponential smoothing method only considers Moment and The data at the moment has a limited amount of data, and there may still be burrs in the process of smoothing. Therefore, the embodiment of the present application combines the advantages of the above two methods and proposes a new method, and the specific formula is expressed as follows:

[0067] ;

[0068] in, For the corresponding The weight coefficient of the moment.

[0069] Furthermore, considering that the closer The greater the impact of noise on the data, the greater the impact of noise on the data. Time has come time, Gradually increase.

[0070] In this embodiment, missing value processing mainly uses interpolation to supplement missing data points. In addition, in order to further improve the smoothness and continuity of the data, the embodiment of the present application can use cubic spline interpolation to construct a piecewise defined polynomial function to fit the data points. Specifically, the use of the target smoothing formula to process the device indicator to be processed, and sending the obtained target device indicator to the preset processing center in the preset resource monitoring center for analysis, may include: using the target smoothing formula to process the device indicator to be processed to obtain a first intermediate device indicator; using a preset polynomial merging rule to merge the quadratic polynomial and the cubic polynomial in the initial formula corresponding to the preset interpolation algorithm to obtain a 2.5-power polynomial; combining the 2.5-power polynomial with the initial formula according to a preset formula combination rule to obtain a target combination formula, and using the target combination formula to calculate the first intermediate device indicator to obtain a second intermediate device indicator; using a preset Fourier transform formula to convert the data type corresponding to the second intermediate device indicator from time domain data to frequency domain data to obtain the target device indicator, and sending the target device indicator to the preset processing center in the preset resource monitoring center for analysis.

[0071] In a specific implementation, for each data interval , the cubic spline difference is calculated as follows:

[0072] ;

[0073] For each pair of adjacent data points and , first we need to calculate the second-order derivative of each segment , and then calculate according to the above second-order derivative ,in, They are the coefficients of each power, which can be set based on user needs, and the calculation method of the coefficients of each power is as follows:

[0074] ;

[0075] ; ;

[0076] .

[0077] Furthermore, in this embodiment, considering that the above formula needs to use more data points for difference calculation, this puts forward higher requirements on the data quality of the original data. Therefore, the embodiment of the present application combines the second power and the third power in the above formula to reduce data dependence and reduce the amount of related calculations. That is, the obtained formula is as follows:

[0078] .

[0079] Step S13, calling the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to use the obtained time series analysis results to build a target load learning model, and then use the target load learning model to predict the load condition of the target device to obtain a load prediction result.

[0080] In this embodiment, after obtaining the preprocessed data, the embodiment of the present application needs to use the Fourier transform method to convert the data format corresponding to the preprocessed data from time domain data to frequency domain data, so as to use the preset frequency domain analysis technology to analyze the periodic components of the data whose data format is frequency domain data, and the Fourier transform formula is as follows:

[0081] ;

[0082] in, is a function in the time domain, representing a signal that changes over time. represents a function in the frequency domain, indicating the amplitude and phase of different frequency components, is a time variable, representing a time point in the time domain, is a frequency variable, indicating the frequency point in the frequency domain, is a complex exponential function, Is an imaginary unit.

[0083] In this embodiment, an LSTM neural network is required to perform serial modeling operations on complex nonlinear time series. Among them, the LSTM (Long Short-Term Memory) neural network is a special RNN (Recurrent Neural Network) structure, which uses the LSTM neural network to process long sequence data. Specifically, calling the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to use the obtained time series analysis results to build a target load learning model, and then using the target load learning model to predict the load of the target device to obtain the load prediction result, can include: calling the preset input gate, preset forget gate and preset output gate in the preset time series analysis model in the preset load analyzer to analyze the resource analysis results to obtain the time series analysis results; using the time series analysis results to train the initial load learning model to obtain the target load learning model, so as to use the target load learning model to predict the load of the target device to obtain the load prediction result. That is, using the LSTM neural network to predict the load changes of the target device in the future period of time, so as to understand the performance hotspots in the target device, and give priority to accelerating the performance hotspots.

[0084] Step S14: Use a preset deep learning model to process the load prediction result to obtain an initial data stream, and use a preset data stream adjustment algorithm to adjust the initial data stream to obtain a target data stream, so that the preset resource scheduler uses the target data stream to configure the target device.

[0085] In this embodiment, the process of optimizing the data stream using the deep learning model is as follows: Figure 4As shown in the figure: First, a set of initial data flows is set according to the load changes output by the LSTM model. Then, a mathematical model is established to evaluate each data flow generated, and the evaluation results are used as the basis for judging each data flow. Finally, the data flow is adjusted using the evaluation results. Specifically, the process of processing the load prediction result using a preset deep learning model to obtain an initial data stream, and adjusting the initial data stream using a preset data stream adjustment algorithm to obtain a target data stream, so that the preset resource scheduler uses the target data stream to configure the target device, may include: using a preset deep learning model and generating a plurality of initial data streams based on the load prediction result issued by the preset load analyzer, and using a preset evaluation model to evaluate each of the initial data streams to obtain a corresponding evaluation result, and then judging whether the evaluation result is greater than a preset threshold; if the evaluation result is greater than the preset threshold, setting the data stream corresponding to the evaluation result as the target data stream, so that the preset resource scheduler uses each of the target data streams to configure the target device; if the evaluation result is not greater than the preset threshold, adjusting the corresponding initial data stream based on each of the evaluation results using a preset gradient descent method, and then jumping to the step of evaluating each of the initial data streams using a preset evaluation model to obtain a corresponding evaluation result, and then judging whether the evaluation result is greater than the preset threshold.

[0086] In a specific implementation, the data stream may be tuned using a gradient descent method to converge along the direction of the gradient change, and then when the performance of the generated data stream is within a preset range, the data stream tuning process is terminated.

[0087] That is, when the application is running on the host, the embodiment of the present application needs to automatically identify the computationally intensive parts of the target device in the intermediate representation of the computation graph, and retain the program graph and place it on the data flow hardware, then obtain telemetry data from the hardware, and do it recursively to optimize computation and memory when the program is running.

[0088] In this embodiment, it is necessary to use the resource monitoring module and the data load analyzer to process the target device indicators to obtain the target device indicators, so as to call the preset resource scheduler and dynamically allocate the computer resources corresponding to the target device according to the target device indicators. Specifically, the use of the preset data flow adjustment algorithm to adjust the initial data flow to obtain the target data flow, so that the preset resource scheduler uses the target data flow to configure the target device, can include: using the preset data flow adjustment algorithm to adjust each of the initial data flows to obtain the corresponding target data flow, and calling the preset load analyzer to send each of the target data flows to the preset resource scheduler respectively, so that the preset resource scheduler determines the corresponding computer resource type and computer resource quantity based on each of the target data flows, and configures the target device based on the computer resource type and the computer resource quantity. In a specific implementation, the embodiment of the present application can increase the number of CPU cores when the data load is high, and reduce the number of cores when the load is low to save energy.

[0089] It can be seen that the embodiment of the present application monitors the device indicators corresponding to the target device by calling the preset resource monitoring center to obtain the device indicators to be processed; constructs a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, uses the target smoothing formula to process the device indicators to be processed, and sends the obtained target device indicators to the preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results; calls the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to construct a target load learning model using the obtained time series analysis results, and then uses the target load learning model to predict the load of the target device to obtain a load prediction result; uses the preset deep learning model to process the load prediction results to obtain an initial data stream, and uses the preset data stream adjustment algorithm to adjust the initial data stream to obtain a target data stream, so that the preset resource scheduler uses the target data stream to configure the target device. In this way, the data processing speed and efficiency of the device are improved.

[0090] Accordingly, see Figure 5 As shown, the present application also provides a device data flow configuration device, which is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer and a preset resource scheduler, and the device includes:

[0091] The device indicator determination module 11 is used to call the preset resource monitoring center to monitor the device indicator corresponding to the target device to obtain the device indicator to be processed;

[0092] The resource analysis result determination module 12 is used to construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the device index to be processed, and send the obtained target device index to the preset processing center in the preset resource monitoring center for analysis to obtain the resource analysis result;

[0093] The load prediction result determination module 13 is used to call the preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then use the target load learning model to predict the load of the target device to obtain a load prediction result;

[0094] The data flow adjustment module 14 is used to process the load prediction result using a preset deep learning model to obtain an initial data flow, and adjust the initial data flow using a preset data flow adjustment algorithm to obtain a target data flow, so that the preset resource scheduler uses the target data flow to configure the target device.

[0095] As can be seen from the above, before configuring the device data flow, the embodiment of the present application first needs to monitor the device indicators corresponding to the target device by calling the preset resource monitoring center to obtain the device indicators to be processed; then, construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the device indicators to be processed, and send the obtained target device indicators to the preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results; further, call the preset time series analysis model in the preset load analyzer to analyze the resource analysis results, so as to use the obtained time series analysis results to construct a target load learning model, and then use the target load learning model to predict the load of the target device to obtain the load prediction result; use the preset deep learning model to process the load prediction result to obtain the initial data flow. Finally, use the preset data flow adjustment algorithm to adjust the initial data flow to obtain the target data flow, so that the preset resource scheduler uses the target data flow to configure the target device. In this way, the data processing speed and efficiency of the device are improved.

[0096] In some specific implementations, the device indicator determination module 11 may specifically include:

[0097] The device indicator monitoring unit is used to call the preset monitoring tool in the preset resource monitoring center to monitor the device indicators corresponding to the device to obtain the device indicators to be processed; the device indicators include CPU usage, memory usage, storage I / O and network bandwidth.

[0098] In some specific implementations, the resource analysis result determination module 12 may specifically include:

[0099] The target formula determination unit is used to construct a target smoothing formula based on the algorithm characteristics of the moving average algorithm and the exponential smoothing algorithm, and in combination with the current moment, the index of the device to be processed, the number of indicators of the device to be processed, and the indicator weight, so as to use the target smoothing formula to process the device indicator to be processed; wherein the numerical value corresponding to the current moment is positively correlated with the indicator weight.

[0100] In some specific implementations, the resource analysis result determination module 12 may specifically include:

[0101] A first device index determination unit, configured to process the device index to be processed using the target smoothing formula to obtain a first intermediate device index;

[0102] A polynomial merging unit, used to merge a quadratic polynomial and a cubic polynomial in an initial formula corresponding to a preset interpolation algorithm using a preset polynomial merging rule to obtain a 2.5-power polynomial;

[0103] a second device index determination unit, configured to combine the 2.5-power polynomial with the initial formula according to a preset formula combination rule to obtain a target combination formula, and to calculate the first intermediate device index using the target combination formula to obtain a second intermediate device index;

[0104] A data type conversion unit is used to convert the data type corresponding to the second intermediate state device indicator from time domain data to frequency domain data using a preset Fourier transform formula to obtain the target device indicator, so as to send the target device indicator to a preset processing center in the preset resource monitoring center for analysis.

[0105] In some specific implementations, the load prediction result determination module 13 may specifically include:

[0106] A sequence analysis result determination unit, used for calling a preset input gate, a preset forget gate and a preset output gate in a preset time series analysis model in the preset load analyzer to analyze the resource analysis result to obtain a time series analysis result;

[0107] The load prediction result determination unit is used to train the initial load learning model using the time series analysis result to obtain a target load learning model, and to use the target load learning model to predict the load condition of the target device to obtain a load prediction result.

[0108] In some specific implementations, the data flow adjustment module 14 may specifically include:

[0109] An evaluation result judgment unit, used to generate a plurality of initial data streams using a preset deep learning model and based on the load prediction result issued by the preset load analyzer, and to evaluate each of the initial data streams using a preset evaluation model to obtain a corresponding evaluation result, and then to determine whether the evaluation result is greater than a preset threshold;

[0110] a target data stream determining unit, configured to set the data stream corresponding to the evaluation result as the target data stream if the evaluation result is greater than the preset threshold, so that the preset resource scheduler configures the target device using each of the target data streams;

[0111] The data flow adjustment subunit is used to adjust the corresponding initial data flow based on each evaluation result using the preset gradient descent method if the evaluation result is not greater than the preset threshold, and then jump to the step of evaluating each initial data flow using the preset evaluation model to obtain the corresponding evaluation result, and then determine whether the evaluation result is greater than the preset threshold.

[0112] In some specific implementations, the data flow adjustment module 14 may specifically include:

[0113] The target device configuration unit is used to adjust each of the initial data streams using a preset data stream adjustment algorithm to obtain a corresponding target data stream, and call the preset load analyzer to send each of the target data streams to the preset resource scheduler respectively, so that the preset resource scheduler determines the corresponding computer resource type and computer resource quantity based on each of the target data streams, and configures the target device based on the computer resource type and the computer resource quantity.

[0114] Furthermore, the present application also discloses an electronic device. Figure 6 This is a structural diagram of an electronic device 20 according to an exemplary embodiment, and the content in the figure cannot be regarded as any limitation on the scope of use of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the device data flow configuration method disclosed in any of the aforementioned embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0115] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0116] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0117] The operating system 221 is used to manage and control the hardware devices on the electronic device 20 and the computer program 222, which can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program that can be used to complete the device data flow configuration method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 can further include a computer program that can be used to complete other specific tasks.

[0118] Furthermore, the present application also discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned disclosed device data flow configuration method. The specific steps of the method can refer to the corresponding contents disclosed in the aforementioned embodiments, and will not be repeated here.

[0119] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0120] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0121] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0122] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0123] The technical solution provided by the present application is introduced in detail above. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for general technicians in this field, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A device data flow configuration method, characterized in that: The method is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer and a preset resource scheduler, and comprises: Calling the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed; Based on the characteristics of several initial smoothing processing algorithms, a target smoothing formula is constructed, the target smoothing formula is used to process the device index to be processed, and the obtained target device index is sent to a preset processing center in the preset resource monitoring center for analysis to obtain a resource analysis result; Calling a preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then using the target load learning model to predict the load condition of the target device to obtain a load prediction result; Processing the load prediction result using a preset deep learning model to obtain an initial data stream, and adjusting the initial data stream using a preset data stream adjustment algorithm to obtain a target data stream, so that the preset resource scheduler configures the target device using the target data stream; Among them, the use of a preset data flow adjustment algorithm to adjust the initial data flow to obtain a target data flow includes: using a preset evaluation model to evaluate each of the initial data flows to obtain a corresponding evaluation result, and then determining whether the evaluation result is greater than a preset threshold; if the evaluation result is not greater than the preset threshold, using a preset gradient descent method and adjusting the corresponding initial data flow based on each of the evaluation results.

2. The device data flow configuration method according to claim 1, characterized in that: The calling of the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed includes: The preset monitoring tool in the preset resource monitoring center is called to monitor the device indicators corresponding to the device to obtain the device indicators to be processed; the device indicators include CPU usage, memory usage, storage I / O and network bandwidth.

3. The device data flow configuration method according to claim 1, characterized in that: The target smoothing formula is constructed based on the characteristics of several initial smoothing processing algorithms, and the target smoothing formula is used to process the index of the device to be processed, including: Based on the algorithm characteristics of the moving average algorithm and the exponential smoothing algorithm, and in combination with the current moment, the index of the device to be processed, the number of indicators of the device to be processed, and the indicator weight, a target smoothing formula is constructed to process the device index to be processed using the target smoothing formula; Among them, the value corresponding to the current moment is positively correlated with the indicator weight.

4. The device data flow configuration method according to claim 1, characterized in that: The step of processing the target device indicator by using the target smoothing formula and sending the obtained target device indicator to a preset processing center in the preset resource monitoring center for analysis includes: Processing the device index to be processed by using the target smoothing formula to obtain a first intermediate device index; Using a preset polynomial merging rule, the quadratic polynomial and the cubic polynomial in the initial formula corresponding to the preset interpolation algorithm are merged to obtain a 2.5-power polynomial; Combining the 2.5-power polynomial with the initial formula according to a preset formula combination rule to obtain a target combination formula, and using the target combination formula to calculate the first intermediate device indicator to obtain a second intermediate device indicator; The data type corresponding to the second intermediate state device indicator is converted from time domain data to frequency domain data using a preset Fourier transform formula to obtain the target device indicator, so as to send the target device indicator to a preset processing center in the preset resource monitoring center for analysis.

5. The device data flow configuration method according to claim 1, characterized in that: The calling of the preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then using the target load learning model to predict the load of the target device to obtain a load prediction result, includes: Calling a preset input gate, a preset forget gate, and a preset output gate in a preset time series analysis model in the preset load analyzer to analyze the resource analysis result to obtain a time series analysis result; The time series analysis result is used to train the initial load learning model to obtain a target load learning model, and the target load learning model is used to predict the load condition of the target device to obtain a load prediction result.

6. The device data flow configuration method according to claim 1, characterized in that: The using a preset deep learning model to process the load prediction result to obtain an initial data stream, and using a preset data stream adjustment algorithm to adjust the initial data stream to obtain a target data stream, so that the preset resource scheduler uses the target data stream to configure the target device, including: Generate a plurality of initial data streams using a preset deep learning model and based on the load prediction result issued by the preset load analyzer, and evaluate each of the initial data streams using a preset evaluation model to obtain a corresponding evaluation result, and then determine whether the evaluation result is greater than a preset threshold; If the evaluation result is greater than the preset threshold, the data stream corresponding to the evaluation result is set as the target data stream, so that the preset resource scheduler configures the target device using each of the target data streams; If the evaluation result is not greater than the preset threshold, the preset gradient descent method is used to adjust the corresponding initial data stream based on each evaluation result, and then the process jumps to the step of evaluating each initial data stream using the preset evaluation model to obtain the corresponding evaluation result, and then determining whether the evaluation result is greater than the preset threshold.

7. The device data flow configuration method according to claim 1, characterized in that: The adjusting the initial data stream by using a preset data stream adjustment algorithm to obtain a target data stream so that the preset resource scheduler configures the target device by using the target data stream includes: Each of the initial data streams is adjusted using a preset data stream adjustment algorithm to obtain a corresponding target data stream, and the preset load analyzer is called to send each of the target data streams to the preset resource scheduler, so that the preset resource scheduler determines the corresponding type and quantity of computer resources based on each of the target data streams, and configures the target device based on the type and quantity of computer resources.

8. A device data flow configuration apparatus, characterized in that: The device is applied to a data flow engine including a preset resource monitoring center, a preset load analyzer and a preset resource scheduler, and comprises: The device indicator determination module is used to call the preset resource monitoring center to monitor the device indicators corresponding to the target device to obtain the device indicators to be processed; A resource analysis result determination module is used to construct a target smoothing formula based on the characteristics of several initial smoothing processing algorithms, use the target smoothing formula to process the device indicators to be processed, and send the obtained target device indicators to a preset processing center in the preset resource monitoring center for analysis to obtain resource analysis results; A load prediction result determination module, used to call a preset time series analysis model in the preset load analyzer to analyze the resource analysis result, so as to construct a target load learning model using the obtained time series analysis result, and then use the target load learning model to predict the load of the target device to obtain a load prediction result; A data stream adjustment module, used to process the load prediction result using a preset deep learning model to obtain an initial data stream, and adjust the initial data stream using a preset data stream adjustment algorithm to obtain a target data stream, so that the preset resource scheduler configures the target device using the target data stream; Among them, the device is also used to adjust the initial data flow using a preset data flow adjustment algorithm to obtain a target data flow, including: using a preset evaluation model to evaluate each of the initial data flows to obtain a corresponding evaluation result, and then judging whether the evaluation result is greater than a preset threshold; if the evaluation result is not greater than the preset threshold, using a preset gradient descent method and adjusting the corresponding initial data flow based on each of the evaluation results.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the device data flow configuration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: Used to store a computer program, wherein when the computer program is executed by a processor, the steps of the device data flow configuration method according to any one of claims 1 to 7 are implemented.

Images