Data processing method and device, equipment, storage medium and product

Abstract

The invention discloses a data processing method and device, equipment, a storage medium and a product. The method comprises the steps of obtaining window time delay of a previous window; predicting the data transmission time delay of the current window; according to a comparison result of the data transmission time delay of the current window and the window time delay of the previous window, adjusting the window time delay of the previous window to obtain the window time delay of the current window; and processing data in the current window based on the window time delay of the current window. According to the embodiment of the invention, the window time delay setting is more scientific and reasonable.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a data processing method, apparatus, device, storage medium and product. Background Technology

[0002] With the rapid development of the Internet and Internet of Things industries and the surge in various types of data, effectively processing massive amounts of real-time data has become a key challenge for enterprises. Real-time data processing has a wide range of applications, including instant risk assessment in financial transactions, real-time equipment status monitoring in smart manufacturing, and traffic flow management in smart cities, all demonstrating the importance of real-time decision support.

[0003] Currently, Flink and Spark are mainstream frameworks for real-time data processing. Their core approach often employs a "window + watermark" mechanism to balance real-time performance and data integrity. The window calculation in these frameworks primarily relies on the watermark, and determining the watermark value requires considering both event time and window latency. Since event time is unpredictable, setting the watermark value depends heavily on the window latency; however, excessively large or small window latency can negatively impact the overall performance of the real-time system.

[0004] In existing technologies, window latency is often set based on the experience of development engineers, lacking a scientific and reasonable basis, which ultimately leads to poor real-time performance and integrity in real-time data processing. Summary of the Invention

[0005] This application provides a data processing method, apparatus, device, storage medium, and product to solve the problem in the prior art where relying on the experience of development engineers to set window delays leads to poor real-time performance and integrity in real-time data processing.

[0006] To achieve the above objectives, embodiments of this application provide a data processing method, including: Get the window delay of the previous window; Predict the data transmission latency of the current window; Based on the comparison between the data transmission delay of the current window and the window delay of the previous window, the window delay of the previous window is adjusted to obtain the window delay of the current window. The data within the current window is processed based on the window delay of the current window.

[0007] As an improvement to the above scheme, the prediction of the data transmission delay of the current window includes: At least one prediction model is used to predict the data transmission latency of the current window; wherein the prediction model is trained based on data information from historical windows. The data transmission delay of the current window predicted by the at least one prediction model is fused to obtain the data transmission delay of the current window.

[0008] As an improvement to the above scheme, the at least one prediction model includes: a first prediction model, which is trained through the following steps: The historical performance information of the data source node, the historical performance information of the data processing node, and the historical common information of the nodes are obtained; wherein, the historical common information of the nodes includes at least: the historical network information between the data source node and the data processing node, and the data volume of the historical window; The historical performance information of the data source node and the common historical information of the node are input into the first intermediate layer of the first prediction model to obtain the first feature output by the first intermediate layer. The historical performance information of the data processing node and the common historical information of the node are input into the second intermediate layer of the first prediction model to obtain the second feature output by the second intermediate layer. The first feature and the second feature are fused to obtain a fused feature; The first prediction model is obtained by training using the fused features.

[0009] As an improvement to the above scheme, the step of training using the fused features to obtain the first prediction model includes: Based on the fusion features, a loss function is constructed; wherein the loss function is used to measure the difference between the predicted value and the true value of the first prediction model; The first prediction model is obtained by training using the loss function.

[0010] As an improvement to the above scheme, the at least one prediction model includes: a second prediction model, which is trained through the following steps: The second prediction model is trained using the data transmission latency of the historical window; Calculate the relative mean absolute error of the second prediction model; If the relative average absolute error is greater than a preset threshold, then the second prediction model is fine-tuned.

[0011] As an improvement to the above scheme, the fine-tuning training of the second prediction model includes: Add an adjustable parameter matrix to the second prediction model; The adjustable parameter matrix is ​​decomposed into a low-rank matrix; Freeze the basic parameters of the second prediction model, and use the data transmission delay of the historical window to perform backpropagation to obtain the low-rank matrix.

[0012] As an improvement to the above scheme, the step of adjusting the window delay of the previous window based on the comparison result of the data transmission delay of the current window and the window delay of the previous window to obtain the window delay of the current window includes: Calculate the difference between the data transmission latency of the current window and the window latency of the previous window; The difference is scaled to obtain the change value; The window delay of the previous window is adjusted based on the change value to obtain the window delay of the current window.

[0013] To achieve the above objectives, embodiments of this application also provide a data processing apparatus, including: The acquisition module is used to obtain the window delay of the previous window; The prediction module is used to predict the data transmission latency of the current window; An adjustment module is used to adjust the window delay of the previous window based on a comparison between the data transmission delay of the current window and the window delay of the previous window, so as to obtain the window delay of the current window. The processing module is used to process the data within the current window based on the window delay of the current window.

[0014] To achieve the above objectives, embodiments of this application also provide a data processing device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the data processing method as described above when executing the computer program.

[0015] To achieve the above objectives, embodiments of this application also provide a computer-readable storage medium, the computer-readable storage medium including a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data processing method as described above.

[0016] To achieve the above objectives, embodiments of this application also provide a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data processing method described above.

[0017] Compared with existing technologies, the data processing method, apparatus, device, storage medium, and product provided in this application embodiment obtain the window latency of the previous window; predict the data transmission latency of the current window; adjust the window latency of the previous window based on the comparison result of the data transmission latency of the current window and the window latency of the previous window to obtain the window latency of the current window; and process the data in the current window based on the window latency of the current window. Therefore, this application embodiment fully combines the data transmission latency of the current window and adaptively and dynamically adjusts the window latency without relying on manual experience, making the window latency setting more scientific and reasonable, thereby effectively improving the real-time performance and completeness of real-time data processing. Attached Figure Description

[0018] Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a real-time data processing procedure provided in an embodiment of this application; Figure 3 This is a structural block diagram of a first prediction model provided in an embodiment of this application; Figure 4 This is a structural block diagram of a data processing apparatus provided in an embodiment of this application; Figure 5 This is a structural block diagram of a data processing device provided in an embodiment of this application. Detailed Implementation

[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0020] In the description of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0021] In this application description, the terms "exemplary" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or designs. Specifically, the use of terms such as "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0022] In this application description, the terms "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus. The term "based on" means "at least partially based on." The term "according to" means "at least partially according to." The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments." The term "and / or" means at least one of the connected objects, such as A and / or B, indicating three cases: including only A, only B, and both A and B. Unless otherwise stated, the term "multiple" means two or more.

[0023] It's worth noting that in real-time data processing, data is typically processed sequentially. However, due to network latency, insufficient computing resources, and other factors, some data may be delayed or out of order, resulting in poor real-time performance or completeness. Current frameworks like Flink and Spark employ a "window + watermark" approach to balance data real-time performance and completeness. Figure 2 : Window: refers to a time range within a window; for example... Figure 2 The current window is in the range of 12:00:00-12:00:10.

[0024] Water level: A marker that measures the progress of an event over time. The water level can be viewed as a special data record. Water level = maximum time of the current window event - window delay.

[0025] Triggering condition: When the water level exceeds the window end time, the window calculation is triggered, the current window ends, and the next window is opened.

[0026] like Figure 2 The principle of real-time data processing is as follows: 1. Assuming the system's window delay time is set to 10 seconds, when event A arrives at the window, since the event time (12:00:11) is the current maximum event time entering the window, the water level = 12:00:11 - 10 seconds = 12:00:01. The water level is inside the window, which does not meet the window calculation conditions, and the window calculation is not triggered.

[0027] 2. When event B arrives at the window, event B is not the current maximum event time (less than A). The water level is 12:00:11 - 10 seconds = 12:00:01. The water level is within the window, so the window calculation is not triggered.

[0028] 3. When event C is entered, since event C is the current maximum event time, the water level = 12:00:12 - 10 seconds = 12:00:02. The water level is within the window, so the window calculation is not triggered.

[0029] 4. When event D arrives at the window, event D is not the current maximum event time (less than C). The water level is 12:00:12 - 10 seconds = 12:00:02. The water level is within the window, so the window calculation is not triggered.

[0030] 5. When event E arrives at the window, since event E is the current maximum event time, and the water level = 12:00:25 - 10 seconds = 12:00:15, the water level is outside the window at this time, satisfying the window calculation triggering rule: water level ≥ window end time, the window immediately triggers calculation (specifically, calculation of real-time data within the current window), and event F is discarded. In other words, events A through E belong to the current window, and when the window triggers calculation, it will process events A through E within the current window.

[0031] The above window calculation triggers mainly rely on the water level, and the determination of the water level value needs to be combined with the event time and window delay. Since the event time is uncontrollable, the setting of the water level value actually depends more on the window delay, but if the window delay is too large or too small, it will affect the overall performance of the real-time system.

[0032] Based on this, the data processing method provided in this application fully combines the data transmission latency of the current window and adaptively and dynamically adjusts the window latency without relying on manual experience, making the window latency setting more scientific and reasonable, thereby effectively improving the real-time performance and completeness of real-time data processing.

[0033] See Figure 1 , Figure 1 This is a flowchart of a data processing method provided in an embodiment of this application. The data processing method includes: Get the window delay of the previous window; It is worth noting that the window delay of the (t-1)th window can be preset by the user, or it can be obtained by adjusting the window delay of the (t-2)th window based on the comparison between the data transmission delay of the (t-1)th window and the window delay of the (t-2)th window. Predict the data transmission latency of the current window; It is worth noting that the data transmission latency of the current window refers to the time consumed from when the data belonging to the current window is sent from the data source node to when it is received by the data processing node. This application does not limit the prediction method; any method can be used for prediction. For example, the data transmission latency of the current window can be predicted using a learning algorithm, or it can be based on historical window data transmission latency statistics, real-time network bandwidth monitoring results, etc. The specific method can be flexibly selected according to the actual application scenario.

[0034] Based on the comparison between the data transmission delay of the current window and the window delay of the previous window, the window delay of the previous window is adjusted to obtain the window delay of the current window. It is worth noting that this application determines whether the window delay of the previous window needs to be adjusted by combining the comparison results of the data transmission delay of the current window and the window delay of the previous window, thereby obtaining the window delay of the current window.

[0035] For example, if the data transmission latency of the current window is greater than the window latency of the previous window, it means that the current window needs a larger window latency to adapt to the data transmission requirements. Therefore, the window latency of the previous window is increased and used as the window latency of the current window.

[0036] If the data transmission delay of the current window equals the window delay of the previous window, it means that the window delay of the previous window needs to be adapted to the data transmission requirements of the current window. In this case, the window delay of the previous window is directly used as the window delay of the current window.

[0037] If the data transmission delay of the current window is less than the window delay of the previous window, it means that the current data transmission does not require the original large window delay. Therefore, the window delay of the previous window is reduced to obtain the window delay of the current window.

[0038] The data within the current window is processed based on the window delay of the current window.

[0039] In this embodiment of the application, after confirming the window delay of the current window, real-time data can be processed according to the "window + water level" mechanism. For example, when real-time data arrives at the current window, the water level is calculated as: maximum event time of the current window - window delay of the current window. If the water level is within the current window, the calculation is not triggered; if the water level is not within the current window, the calculation is triggered.

[0040] The embodiments of this application fully integrate the data transmission latency of the current window and adaptively and dynamically adjust the window latency without relying on manual experience, making the window latency setting more scientific and reasonable, thereby effectively improving the real-time performance and completeness of real-time data processing.

[0041] In one optional embodiment, predicting the data transmission latency of the current window includes: At least one prediction model is used to predict the data transmission latency of the current window; wherein the prediction model is trained based on data information from historical windows. The data transmission delay of the current window predicted by the at least one prediction model is fused to obtain the data transmission delay of the current window.

[0042] It is worth noting that the prediction model trained based on historical window data in this application embodiment, and fused with at least one data transmission latency predicted by the prediction model, can accurately predict the data transmission latency of the current window, thereby providing a reliable basis for dynamic adjustment of window latency and effectively improving the adaptability of window latency. The fusion method can employ weighted summation.

[0043] Specifically, the data information of the historical window includes, but is not limited to, the amount of data in the historical window, data transmission latency, etc. This information is used to train a machine learning model to obtain a prediction model, which is used to predict the data transmission latency of the current window. This application embodiment does not impose specific limitations on the training process.

[0044] For example, two prediction models, A and B, can be trained using different model structures or different training data to provide a multi-dimensional reference perspective for predicting the data transmission latency of the current window. Model A is used to predict the data transmission latency A of the current window; Model B is used to predict the data transmission latency B of the current window. By weighted summation, the data transmission latency A and B of the current window are fused to obtain a more accurate data transmission latency for the current window. This more accurate latency is then compared with the latency of the previous window, and adjustments are made based on the comparison result to obtain the final window latency for the current window.

[0045] In one optional embodiment, the at least one prediction model includes: a first prediction model, which is trained through the following steps: The historical performance information of the data source node, the historical performance information of the data processing node, and the historical common information of the nodes are obtained; wherein, the historical common information of the nodes includes at least: the historical network information between the data source node and the data processing node, and the data volume of the historical window; The historical performance information of the data source node and the common historical information of the node are input into the first intermediate layer of the first prediction model to obtain the first feature output by the first intermediate layer. The historical performance information of the data processing node and the common historical information of the node are input into the second intermediate layer of the first prediction model to obtain the second feature output by the second intermediate layer. The first feature and the second feature are fused to obtain a fused feature; The first prediction model is obtained by training using the fused features.

[0046] It is worth noting that in real-time data processing scenarios, the main factors affecting data transmission latency are: first, the performance of the data source node and the data processing node, such as server performance information including CPU, memory, disk, and network; the higher the server performance, the lower the data transmission latency. Second, the window data size; the larger the window data size, the higher the data latency. Therefore, this application embodiment considers these two aspects of information to train the first prediction model, thereby reducing window latency when resources are sufficient and the load is low, and increasing window latency when system resources are insufficient and the load is high, to ensure the integrity and real-time performance of real-time data.

[0047] In one optional embodiment, training the first prediction model using the fused features includes: Based on the fusion features, a loss function is constructed; wherein the loss function is used to measure the difference between the predicted value and the true value of the first prediction model; The first prediction model is obtained by training using the loss function.

[0048] The embodiments of this application utilize a loss function to quantify the difference between the predicted value and the true value of the first prediction model and conduct training, which can effectively improve the prediction accuracy of the first prediction model and make its predictions for scenarios such as data transmission latency more realistic.

[0049] In one specific embodiment, a first prediction model is trained through the following steps, the structural block diagram of which is shown below. Figure 3 As shown.

[0050] (1) Input of the first prediction model The Zabbix monitoring tool was used to collect real-time parameters such as CPU, memory, disk, network, and window data volume of the data source node and data processing node. These parameters were then used as input to construct the first prediction model. The model input parameters are shown in Table 1. Table 1 Model Input Parameters

[0051] (2) Generate the intermediate layer of the first prediction model This application assumes that data transmission latency exhibits a regularity with the system's real-time operating status and the amount of real-time data, and uses this as the basis for training the first prediction model. The structure of the first prediction model is as follows: Figure 2 As shown.

[0052] First intermediate layer: Input of the first intermediate layer The main factors are the performance of the data source node and the amount of data in the window. , can be represented as: ; in, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between them. express The deviation value.

[0053] The first intermediate layer output is After sigmoid activation function f The result of the calculation is (i.e., the first characteristic), as shown in the following formula: ; Second intermediate layer: Input of the second intermediate layer The main factors are the performance of the data processing nodes and the amount of data in the window. , can be represented as: ; in, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between, express and Weight parameters between them. express The deviation value.

[0054] The output of the second intermediate layer is After sigmoid activation function f The result of the calculation is (i.e., the second characteristic), as shown in the following formula: ; (3) Output of the first prediction model: Fusion features It can be represented as: ; in, Indicates the output of the first intermediate layer The weight parameters, Indicates the output of the second intermediate layer The weight parameters. express The deviation value, These are intermediate parameters used for model training and have no practical significance.

[0055] Fusion features After the activation function, the output is... Represented as: .

[0056] (4) Loss function To train the first prediction model, the model is considered most accurate when the loss function between the model's predicted values ​​and the actual values ​​is minimized. Assume we have... A window, loss function Represented as: ; in, The first model predicted the first i The predicted data transmission latency for each window. For the first i The actual value of data transmission latency for each window is only when... The model is most accurate when the value is minimized. First, the various weight parameters and deviation values ​​mentioned above are randomly generated. Then, the optimal weight parameters and deviation values ​​are calculated iteratively using the backpropagation gradient descent method.

[0057] (5) Generate the first prediction model After steps (1) to (4) above, a first prediction model is generated. For any given historical performance information of the data source node, the historical performance information of the data processing node, and the common historical information of the nodes, the first prediction model can be used to predict the data transmission delay, as shown in the following formula: .

[0058] In the formula, Dm() represents the output of the first prediction model.

[0059] In an optional embodiment, the at least one prediction model includes: a second prediction model, which is trained through the following steps: The second prediction model is trained using the data transmission latency of the historical window; Calculate the relative mean absolute error of the second prediction model; If the relative average absolute error is greater than a preset threshold, then the second prediction model is fine-tuned.

[0060] It is worth noting that, as the real-time data processing system operates, from the perspective of the single variable of data transmission delay, the data transmission delay exhibits a time-series trend, and the data transmission delay at the next moment can be predicted from historical data transmission delays. This application's embodiment, based on the time-series characteristics of data transmission delay, uses a second prediction model... It can predict the data transmission latency of the next window based on the data transmission latency of the historical window.

[0061] The second prediction model is a basic large model. While the basic large model has a certain generalization ability, its ability to predict time series data in vertical scenarios needs further improvement. Therefore, in the real-time data transmission latency prediction scenario, this application embodiment performs fine-tuning based on historical samples. This allows the fine-tuned second prediction model to maintain the generalization ability of the original large model while improving the accuracy of model prediction in real-time data processing vertical scenarios.

[0062] Specifically, the relative mean absolute error (RME) of the second prediction model can be obtained by comparing the predicted value and the actual value. When the RME of the second prediction model is less than or equal to a preset threshold, it indicates that the error is within an acceptable range, and no fine-tuning is required; the second prediction model can be directly used to predict the data transmission delay of the current window. If the RME of the second prediction model is greater than the preset threshold, it indicates that the generalization ability of the second prediction model is poor, and fine-tuning is required. The fine-tuned second prediction model can then be used to predict the data transmission delay of the current window. In one optional embodiment, fine-tuning the second prediction model includes: Add an adjustable parameter matrix to the second prediction model; The adjustable parameter matrix is ​​decomposed into a low-rank matrix; Freeze the basic parameters of the second prediction model, and use the data transmission delay of the historical window to perform backpropagation to obtain the low-rank matrix.

[0063] It is worth noting that while the parameter matrix of the second prediction model has a high dimensionality, only a small portion of the parameters play a major role in practical applications in specific scenarios. Based on this characteristic, the basic parameters of the second prediction model (i.e., the preset parameters in the original parameters) can be frozen, and an adjustable parameter matrix can be added while keeping the basic parameters unchanged. , making ,in, These are the fine-tuned parameters. These are the basic parameters. Using low-dimensional matrix factorization approximation, new technologies are applied in real-time data processing scenarios. In this embodiment, during fine-tuning, the low-rank matrix is ​​calculated through backpropagation, which significantly reduces the amount of parameter calculation and improves the efficiency of model fine-tuning.

[0064] In one specific embodiment, the second prediction model is specifically a basic large model that can utilize t Time to t +r time window data transmission delay prediction t The data transmission delay at time +r+1, i.e., utilizing the... t Data transmission delay prediction for the t+rth window. t The data transmission delay for +r+1 windows is as follows: ; In the formula, Indicates the first t The data transmission delay of +r+1 windows, where gpt() represents the output value of the second prediction model. Indicates the first t Data transmission delay for +r windows.

[0065] For example, historical data transmission latency of data source nodes is collected to form historical samples as shown in Table 2. It is assumed that the real-time data processing system has... n One data transmission channel, with m The data transmission latency of the first historical window is recorded in the table. j The first transmission channel i The actual value of data transmission latency for each window.

[0066] Table 2 Historical Samples

[0067] The second prediction model was trained using the historical samples collected above.

[0068] The relative mean absolute error rMAE of the second prediction model is calculated as follows:

[0069] In the formula, r refers to the data transmission latency of the future window predicted using r historical windows. It refers to the first j The first transmission channel i The actual value of data transmission latency for each window. This refers to the prediction of the second prediction model. j The first transmission channel i The predicted value of data transmission delay for each window. Of course, the relative mean absolute error can also be calculated using other formulas, not limited to this one.

[0070] when If the value exceeds a preset threshold, fine-tuning is performed. The second prediction model is fine-tuned through the following steps.

[0071] (1) Assume that the basic parameter W of the second prediction model is an a To improve fine-tuning efficiency, matrix b is added while keeping the original W unchanged. W, such that WB = W + W can significantly improve the efficiency of fine-tuning the model while using fewer resources; WB is a new model parameter for the second prediction model.

[0072] (2) To further improve fine-tuning efficiency and save computing resources, W is decomposed into a low-rank matrix. The decomposition of W means that two smaller a's are needed. b represents a larger matrix W. Using the number of rows of a and W is the same, and the number of columns in b is the same. With W being the same, the above matrix decomposition is denoted as... .if If W has 10,000 rows and 10,000 columns, then 100,000,000 parameters need to be stored. If the hyperparameters a and b are 8, then a has 10,000 rows and 8 columns, and b has 8 rows and 10,000 columns, that is, 10,000 parameters need to be stored. 8+8 10,000 = 160,000 parameters, which is about 625 times less than 100,000,000 parameters, significantly reducing the demand for computing resources.

[0073] (3) Using the historical samples (Table 2), the data transmission delay at any given time is taken as the dependent variable. The data transmission latency of each window is used as the independent variable in the training. W. Since all the original W parameters are known. W is an unknown variable. Using backpropagation iteration, matrices a and b are obtained, thus yielding... W.

[0074] (4) When using the fine-tuned second prediction model, the fine-tuned second prediction model can be used in the real-time data transmission delay prediction scenario. In other scenarios, the fine-tuned second prediction model can still be used for prediction. Through the embodiments of this application, the generalization ability of the original large model in multiple industries is retained, the prediction accuracy in the real-time data processing scenario is improved, and the computing power requirement is greatly reduced during the fine-tuning training process.

[0075] In one specific embodiment, the data transmission latency of the current window is predicted using the first prediction model described above. The data transmission latency of the current window is predicted using the second prediction model described above. By calculating the average of the two values, the data transmission latency of the current window is obtained. Then, the data transmission latency of the current window is used to adjust the window latency of the previous window. This makes the window latency setting more scientific and accurate. When system resources are sufficient and the load is low, the window latency is reduced; when system resources are insufficient and the load is high, the window latency is appropriately increased to ensure the integrity and real-time performance of real-time data.

[0076] In an optional embodiment, adjusting the window delay of the previous window based on a comparison between the data transmission delay of the current window and the window delay of the previous window to obtain the window delay of the current window includes: Calculate the difference between the data transmission latency of the current window and the window latency of the previous window; The difference is scaled to obtain the change value; The window delay of the previous window is adjusted based on the change value to obtain the window delay of the current window.

[0077] This application embodiment uses the difference in data transmission latency between the current window and the previous window for adjustment, which can achieve fine-grained adjustment of window latency, making the matching degree between window latency and data transmission latency higher and the adjustment more reasonable.

[0078] Specifically, through Calculate the window delay of the current window. In the formula, This indicates the window delay of the previous window. This indicates the data transmission delay of the previous window. This represents a preset scaling factor. Of course, it can be adjusted in other ways, and the embodiments in this application are not limited to this formula.

[0079] See Figure 4 , Figure 4 This is a structural block diagram of a data processing apparatus 10 provided in an embodiment of this application. The data processing apparatus 10 includes: Module 11 is used to obtain the window delay of the previous window; Prediction module 12 is used to predict the data transmission latency of the current window; The adjustment module 13 is used to adjust the window delay of the previous window based on the comparison result of the data transmission delay of the current window and the window delay of the previous window, so as to obtain the window delay of the current window. The processing module 14 is used to process the data in the current window based on the window delay of the current window.

[0080] Optionally, predicting the data transmission delay of the current window includes: At least one prediction model is used to predict the data transmission latency of the current window; wherein the prediction model is trained based on data information from historical windows. The data transmission delay of the current window predicted by the at least one prediction model is fused to obtain the data transmission delay of the current window.

[0081] Optionally, the at least one prediction model includes: a first prediction model, which is trained through the following steps: The historical performance information of the data source node, the historical performance information of the data processing node, and the historical common information of the nodes are obtained; wherein, the historical common information of the nodes includes at least: the historical network information between the data source node and the data processing node, and the data volume of the historical window; The historical performance information of the data source node and the common historical information of the node are input into the first intermediate layer of the first prediction model to obtain the first feature output by the first intermediate layer. The historical performance information of the data processing node and the common historical information of the node are input into the second intermediate layer of the first prediction model to obtain the second feature output by the second intermediate layer. The first feature and the second feature are fused to obtain a fused feature; The first prediction model is obtained by training using the fused features.

[0082] Optionally, training the first prediction model using the fused features includes: Based on the fusion features, a loss function is constructed; wherein the loss function is used to measure the difference between the predicted value and the true value of the first prediction model; The first prediction model is obtained by training using the loss function.

[0083] Optionally, the at least one prediction model includes: a second prediction model, which is trained through the following steps: The second prediction model is trained using the data transmission latency of the historical window; Calculate the relative mean absolute error of the second prediction model; If the relative average absolute error is greater than a preset threshold, then the second prediction model is fine-tuned.

[0084] Optionally, fine-tuning the second prediction model includes: Add an adjustable parameter matrix to the second prediction model; The adjustable parameter matrix is ​​decomposed into a low-rank matrix; Freeze the basic parameters of the second prediction model, and use the data transmission delay of the historical window to perform backpropagation to obtain the low-rank matrix.

[0085] Optionally, adjusting the window delay of the previous window based on a comparison between the data transmission delay of the current window and the window delay of the previous window to obtain the window delay of the current window includes: Calculate the difference between the data transmission latency of the current window and the window latency of the previous window; The difference is scaled to obtain the change value; The window delay of the previous window is adjusted based on the change value to obtain the window delay of the current window.

[0086] It is worth noting that the working process of each module in the data processing device 10 described in this application embodiment can refer to the working process of the data processing method described in the above embodiment and achieve the same beneficial effect, and will not be repeated here.

[0087] Furthermore, this application also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the data processing method as described in any of the above embodiments.

[0088] Furthermore, this application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the data processing method as described in any of the above embodiments.

[0089] See Figure 5 , Figure 5This is a structural block diagram of a data processing device 20 provided in an embodiment of this application. The data processing device 20 includes: a processor 21, a memory 22, and a computer program stored in the memory 22 and executable on the processor 21. When the processor 21 executes the computer program, it implements the steps in the above-described data processing method embodiments. Alternatively, when the processor 21 executes the computer program, it implements the functions of each module / unit in the above-described device embodiments.

[0090] For example, the computer program may be divided into one or more modules / units, which are stored in the memory 22 and executed by the processor 21 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the data processing device 20.

[0091] The data processing device 20 may include, but is not limited to, a processor 21 and a memory 22. Those skilled in the art will understand that the schematic diagram is merely an example of the data processing device 20 and does not constitute a limitation on the data processing device 20. It may include more or fewer components than illustrated, or combine certain components, or different components. For example, the data processing device 20 may also include input / output devices, network access devices, buses, etc.

[0092] The processor 21 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor 21 is the control center of the data processing device 20, connecting all parts of the data processing device 20 via various interfaces and lines.

[0093] The memory 22 can be used to store the computer programs and / or modules. The processor 21 implements various functions of the data processing device 20 by running or executing the computer programs and / or modules stored in the memory 22 and calling the data stored in the memory 22. The memory 22 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory 22 may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, memory, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0094] If the modules / units integrated in the data processing device 20 are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by the processor 21, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc.

[0095] It should be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided in this application, the connection relationships between modules indicate that they have communication connections, which can be specifically implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0096] The above description is the preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications are also considered to be within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, include: Get the window delay of the previous window; Predict the data transmission latency of the current window; Based on the comparison between the data transmission delay of the current window and the window delay of the previous window, the window delay of the previous window is adjusted to obtain the window delay of the current window. The data within the current window is processed based on the window delay of the current window.

2. The data processing method as described in claim 1, characterized in that, The prediction of the data transmission latency of the current window includes: At least one prediction model is used to predict the data transmission latency of the current window; wherein the prediction model is trained based on data information from historical windows. The data transmission delay of the current window predicted by the at least one prediction model is fused to obtain the data transmission delay of the current window.

3. The data processing method as described in claim 2, characterized in that, The at least one prediction model includes: a first prediction model, which is trained through the following steps: The historical performance information of the data source node, the historical performance information of the data processing node, and the historical common information of the nodes are obtained; wherein, the historical common information of the nodes includes at least: the historical network information between the data source node and the data processing node, and the data volume of the historical window; The historical performance information of the data source node and the common historical information of the node are input into the first intermediate layer of the first prediction model to obtain the first feature output by the first intermediate layer. The historical performance information of the data processing node and the common historical information of the node are input into the second intermediate layer of the first prediction model to obtain the second feature output by the second intermediate layer. The first feature and the second feature are fused to obtain a fused feature; The first prediction model is obtained by training using the fused features.

4. The data processing method as described in claim 3, characterized in that, The step of training the first prediction model using the fused features includes: Based on the fusion features, a loss function is constructed; wherein the loss function is used to measure the difference between the predicted value and the true value of the first prediction model; The first prediction model is obtained by training using the loss function.

5. The data processing method as described in claim 2, characterized in that, The at least one prediction model includes: a second prediction model, which is trained through the following steps: The second prediction model is trained using the data transmission latency of the historical window; Calculate the relative mean absolute error of the second prediction model; If the relative average absolute error is greater than a preset threshold, then the second prediction model is fine-tuned and trained.

6. The data processing method as described in claim 5, characterized in that, The fine-tuning training of the second prediction model includes: Add an adjustable parameter matrix to the second prediction model; The adjustable parameter matrix is ​​decomposed into a low-rank matrix; Freeze the basic parameters of the second prediction model, and use the data transmission delay of the historical window to perform backpropagation to obtain the low-rank matrix.

7. The data processing method as described in claim 1, characterized in that, The step of adjusting the window delay of the previous window based on a comparison between the data transmission delay of the current window and the window delay of the previous window to obtain the window delay of the current window includes: Calculate the difference between the data transmission latency of the current window and the window latency of the previous window; The difference is scaled to obtain the change value; The window delay of the previous window is adjusted based on the change value to obtain the window delay of the current window.

8. A data processing apparatus, characterized in that, include: The acquisition module is used to obtain the window delay of the previous window; The prediction module is used to predict the data transmission latency of the current window; An adjustment module is used to adjust the window delay of the previous window based on a comparison between the data transmission delay of the current window and the window delay of the previous window, so as to obtain the window delay of the current window. The processing module is used to process the data within the current window based on the window delay of the current window.

9. A data processing device, characterized in that, The system includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the data processing method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program; wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the data processing method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, It includes a computer program / instruction that, when executed by a processor, implements the data processing method as described in any one of claims 1 to 7.

Images