Task data management method and device, equipment, medium and product

CN118897724APending Publication Date: 2024-11-05BEIJING JINGDONG YUANSHENG TECH CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411186982.8
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-27
Publication Date
2024-11-05

Smart Images

  • Figure CN118897724A_ABST
    Figure CN118897724A_ABST
Patent Text Reader

Abstract

The embodiment of the invention discloses a task data management method and device, equipment and a medium, and the method comprises the steps: obtaining the current queue index data of a target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of task data corresponding to the target message queue; performing serialization processing on the current queue index data, the current operator state data and the current state storage data with corresponding historical data to obtain target to-be-analyzed data; inputting the target data to be analyzed into a pre-trained data skew prediction model to obtain a data skew occurrence probability prediction value and a data skew degree prediction value; and performing scheduling management on the target task data in the message queue according to the predicted value of the data skew occurrence probability and the predicted value of the data skew degree. According to the technical scheme provided by the embodiment of the invention, the real-time performance, accuracy and high reliability of data processing in the distributed subtask can be ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] Embodiments of the present invention relate to the field of computer technology, and in particular, to a task data management method, apparatus, device, medium and product. Background Art

[0002] When the business system processes business data through a distributed task processing system, in order to ensure the sequential processing and accuracy of the data, it is necessary to distribute the business data according to certain dimensions and send the business data of the same dimension or several related dimensions to the same distributed subtask processing system.

[0003] However, in the process of implementing the present invention, it is found that there are at least the following technical problems in the prior art:

[0004] If the amount of data in a certain dimension is very large, the amount of task data that the corresponding distributed subtask processing system has to process will be very large, which will cause data skew. If serious data skew occurs, it will lead to a shortage of computing resources in a single subtask processing system, causing the entire real-time stream computing task data snapshot to time out and fail, resulting in frequent task restarts and data not being processed in a timely manner. Summary of the invention

[0005] The embodiments of the present invention provide a task data management method, apparatus, device, medium and product, which can schedule task data in a timely manner based on the prediction results of data skew, thereby ensuring the real-time, accuracy and high reliability of task data calculation tasks.

[0006] In a first aspect, an embodiment of the present invention provides a task data management method, the method comprising:

[0007] Obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue;

[0008] The current queue indicator data, current operator state data, and current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed;

[0009] Input the target data to be analyzed into the pre-trained data tilt prediction model to obtain the predicted value of the probability of data tilt and the predicted value of the degree of data tilt;

[0010] The target task data in the message queue is scheduled and managed according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0011] In a second aspect, an embodiment of the present invention provides a task data management device, the device comprising:

[0012] The task analysis data acquisition module is used to obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue;

[0013] The analysis data serialization module is used to serialize the current queue indicator data, the current operator state data and the current state storage data with the corresponding historical data to obtain the target data to be analyzed;

[0014] A data tilt prediction module is used to input the target data to be analyzed into a pre-trained data tilt prediction model to obtain a predicted value of the probability of data tilt and a predicted value of the degree of data tilt;

[0015] The task data management module is used to schedule and manage the target task data in the message queue according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0016] In a third aspect, an embodiment of the present invention further provides a computer device, the computer device comprising:

[0017] one or more processors;

[0018] A memory for storing one or more programs;

[0019] When one or more programs are executed by one or more processors, the one or more processors implement the task data management method provided by any embodiment of the present invention.

[0020] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a task data management method as provided in any embodiment of the present invention.

[0021] In a fifth aspect, the embodiments of the present disclosure further provide a computer program product, including a computer program, which, when executed by a processor, implements the task data management method provided by any embodiment of the present invention.

[0022] The embodiments of the above invention have the following advantages or beneficial effects:

[0023] The embodiment of the present invention obtains the current queue index data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; the current queue index data, the current operator state data and the current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed; the target data to be analyzed is input into the pre-trained data tilt prediction model to obtain the data tilt probability prediction value and the data tilt degree prediction value; the target task data in the message queue is scheduled and managed according to the data tilt probability prediction value and the data tilt degree prediction value. The technical solution of the embodiment of the present invention solves the problem that data tilt may occur during the processing of distributed subtasks, and can predict the data tilt situation, and based on the prediction results of the data tilt situation, timely schedule the task data to ensure the real-time, accuracy and high reliability of the task data calculation task. Ensure the real-time, accuracy and high reliability of data processing in distributed subtasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 is a flow chart of a task data management method provided by an embodiment of the present invention;

[0025] Figure 2 is a flow chart of a task data management method provided by an embodiment of the present invention;

[0026] Figure 3 It is a schematic diagram of a task data scheduling example provided by an embodiment of the present invention;

[0027] Figure 4 It is a flowchart of a task data management method provided by an embodiment of the present invention;

[0028] Figure 5 It is a structural diagram of a task data management device provided by an embodiment of the present invention;

[0029] Figure 6 It is a functional module schematic diagram of a distributed subsystem provided by an embodiment of the present invention;

[0030] Figure 7 is a schematic diagram of the working process of a real-time computing module provided by an embodiment of the present invention;

[0031] Figure 8 This is a schematic diagram of the working process of a data indicator monitoring module provided by an embodiment of the present invention;

[0032] Fig. 9 It is a schematic diagram of an example of a distributed subsystem data tilt processing process provided by an embodiment of the present invention;

[0033] Fig.10It is a structural schematic diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0035] Figure 1 The present invention provides a flowchart of a task data management method, which can be applied to the scheduling and management of task data in distributed computing tasks, especially when the amount of task data in different distributed subtasks is unknown. The method can be executed by a task data management device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0036] like Figure 1 As shown, the task data management method of this embodiment includes the following steps:

[0037] S110, obtaining current queue indicator data of the target message queue, current operator state data corresponding to the target message queue, and current state storage data of task data corresponding to the target message queue.

[0038] In a distributed task cluster, each distributed subtask processing system consumes task data from the corresponding message queue. The message body of each message in the message queue contains the task data object to be processed.

[0039] The target message queue may be a subject message queue subscribed by a distributed subtask processing system that currently manages task data.

[0040] The current queue indicator data of the target message queue may be at least one of the indicator data such as message queue topic partition, partition throughput, partition data year-on-year growth, message body size, and message production speed, consumption speed, and backlog.

[0041] The acquisition of queue indicator data can be achieved by periodically calling the message queue Rest API (Rest style application program interface) to obtain the pre-registered indicator data interface to obtain data indicators such as message queue topic partition, partition throughput, partition data year-on-year and month-on-month, and message body size. Through the data relationship between the upstream data source corresponding to the task data and the downstream consumer, based on the topic of the task data and the client identifier corresponding to the distributed subtask processing system, you can query and obtain indicator data such as the production speed, consumption speed and backlog of the message.

[0042] The current operator status data corresponding to the target message queue includes the operator status data of each operator that performs calculations on the task data, such as windowing, filtering, conversion, processing, aggregation, and diversion operators, which correspond to keyby, map, flatmap, union, window, and process operators in Flink tasks.

[0043] To obtain the current operator status data corresponding to the target message queue, you can call the Flink Rest API regularly to obtain the registered indicator interface, thereby obtaining custom indicators. Among them, the custom indicators include indicators of custom Flink operator computing resources, as well as indicators of custom business dimensions, such as order dimensions, merchant dimensions, and warehouse dimensions. When calling the Flink Rest API, you can use the distributed subtask identifier + operator as the calling function input parameter to obtain data such as the data busyness, computing resource pressure, and data circulation degree of the task-specified operator.

[0044] The current state storage data of the task data corresponding to the target message queue includes but is not limited to the data such as the throughput of each state (data item) after the task data is processed, the state update frequency, the state size, and the remote storage cluster index. You can also obtain the current state storage data by calling the corresponding state data information Rest API to obtain the registered index interface, thereby obtaining custom indicators such as the state throughput, state update frequency, state size, and remote storage cluster index.

[0045] In addition, you can also obtain the status of multiple real-time data streams of the operator through the state type, operator topology and state unique descriptor of the real-time computing task operator, and call the state API regularly with the operator + real-time stream state descriptor as input parameters to obtain the read and write throughput, indicators and other conditions of the state. Among them, the state types include MapState, ValueState, ListState, etc.; the operator topology includes keyby+flatmap, keyby+process, keyby+map, etc., among which the keyby dimension can obtain a state value of the finest granularity, such as warehouse code + order number; the state unique descriptor corresponds to the unique state of the real-time task and is globally unique.

[0046] S120 , serialize the current queue indicator data, the current operator state data, and the current state storage data with the corresponding historical data to obtain target data to be analyzed.

[0047] The historical data described herein refers to data corresponding to each item of the current queue indicator data, the current operator state data, and the current state storage data, and is obtained before obtaining the current queue indicator data, the current operator state data, and the current state storage data.

[0048] The historical data to be acquired can be determined according to the sequence length corresponding to the preset target data to be analyzed. For example, in a data sequence, including data acquired ten times in a row, then the data acquired currently can be combined with the same data item data acquired nine times before to obtain a serialization result.

[0049] By combining each item of the current queue indicator data, the current operator state data, and the current state storage data with the corresponding historical data, data serialization can be completed to obtain the target data to be analyzed.

[0050] The reason why data serialization is needed is that in the process of data skew prediction, data features can be extracted by analyzing temporally related data, and then data skew events can be predicted.

[0051] S130, inputting the target data to be analyzed into a pre-trained data skew prediction model to obtain a predicted value of the probability of data skew occurrence and a predicted value of the degree of data skew.

[0052] Among them, the pre-trained data skew prediction model can be a neural network model including two output heads, which can analyze the input target data to be analyzed and output two numerical values: a predicted value of the probability of data skew occurrence and a predicted value of the degree of data skew.

[0053] S140 . Scheduling and managing target task data in the message queue according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0054] When the predicted value of the probability of data skew is greater than a preset probability threshold, it can be considered that data skew will occur, and task data scheduling management is required to avoid serious data skew.

[0055] The predicted value of data skew can be used as a reference for task data scheduling management to confirm the computing resources required during task data scheduling.

[0056] Therefore, more computing resources can be allocated to consume unconsumed task messages in the target message queue and process the task data quickly to ensure the real-time processing of large amounts of business data.

[0057] The technical solution of this embodiment is to obtain the current queue index data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; the current queue index data, the current operator state data and the current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed; the target data to be analyzed is input into the pre-trained data tilt prediction model to obtain the data tilt probability prediction value and the data tilt degree prediction value; the target task data in the message queue is scheduled and managed according to the data tilt probability prediction value and the data tilt degree prediction value. The technical solution of the embodiment of the present invention solves the problem that data tilt may occur during the processing of distributed subtasks, and can predict the data tilt situation, and based on the prediction results of the data tilt situation, timely schedule the task data to ensure the real-time, accuracy and high reliability of the task data calculation task. Ensure the real-time, accuracy and high reliability of data processing in distributed subtasks.

[0058] Figure 2 The flowchart of a task data management method provided by an embodiment of the present invention, this embodiment and the task data management method in the above embodiment belong to the same inventive concept, and further describes the process of task data scheduling management. The method can be executed by a task data management device, which can be implemented by software and / or hardware and integrated in a computer device with application development function.

[0059] like Figure 2 As shown, the task data management method of this embodiment includes the following steps:

[0060] S210: Acquire current queue indicator data of the target message queue, current operator state data corresponding to the target message queue, and current state storage data of task data corresponding to the target message queue.

[0061] S220 , serialize and combine each data item in the current queue indicator data, the current operator state data, and the current state storage data with the same data item in the corresponding historical data in chronological order.

[0062] S230, performing data format conversion on the serialized combination result corresponding to each data item to obtain a serialized combination result in a preset format; integrating the serialized combination result in the preset format corresponding to each data item to obtain target data to be analyzed.

[0063] In an optional implementation, the target data to be analyzed may also include data items obtained by processing the acquired current queue indicator data of the target message queue and some data items in the current operator state data corresponding to the target message queue, for example, further calculating the indicator data in the current queue indicator data, such as the year-on-year comparison, mean, correlation, maximum value and / or percentage of a certain data item, multiple indicator calculation values, etc.

[0064] Accordingly, the serialized combination results of the preset format corresponding to each data item and the calculated indicator data can be summarized and integrated, and the format can be converted into a data format that can be recognized by the data tilt prediction model, such as JSON format, or a serialized data transfer object (DTO).

[0065] S240: Input the target data to be analyzed into a pre-trained data skew prediction model to obtain a predicted value of the probability of data skew occurrence and a predicted value of the degree of data skew.

[0066] The data skew prediction model performs predictive analysis on the sequence data in the target data to be analyzed to determine whether the data will be skewed. The data skew prediction model will output a value between 0 and 1 for each set of sequence feature data. The larger the value, the greater the probability of "data calculation skew", and the corresponding prediction result of the degree of skew will also be output.

[0067] S250: When the predicted value of the probability of data skew occurring is greater than a preset probability threshold, determine to perform task scheduling.

[0068] In the technical solution of this embodiment, the preset probability threshold can be set to 0.7. When the predicted value of the probability of data skew occurring is greater than 0.7, it can be considered that data skew will occur and task scheduling is required.

[0069] S260. Determine the data computing resource requirements according to the data skew amount corresponding to the data skew degree prediction value, the message body size in the target message queue, and the operator computing complexity corresponding to the task data in the message body.

[0070] Data skew can be understood as the amount of data in a distributed subtask processing system that exceeds its normal processing capacity.

[0071] The predicted value of the data skew degree may correspond to a data skew amount, which may be parameter data obtained by quantifying the data skew amount when a historical data skew event occurs.

[0072] In addition, when determining the demand for computing resources, in addition to the amount of data skew, you also need to consider the size of the message body in the target message queue and the computational complexity of the operator corresponding to the task data in the message body. The larger the message body, the larger the amount of data corresponding to the single task data, and the more computing resources are required. The greater the computational complexity of the operator, the more computing resources are required.

[0073] Computing resources can be the CPU, memory, etc. of a computer device.

[0074] S270: Allocate the unconsumed task data in the message queue to computing resources that meet the data computing resource requirements.

[0075] Accordingly, after determining the required computing resources, we can start to reschedule and allocate the task data, and allocate the unconsumed task data in the message queue to computing resources that meet the data computing resource requirements, so that the task data can be processed in a timely manner.

[0076] Specifically, a data snapshot (save point) of the current computing task processing status can be triggered to save the real-time data snapshot of the operator computing task corresponding to the task data in the message queue, and suspend the operator computing task; after the task is suspended, the operator computing task is split or merged according to the real-time data snapshot to obtain the corresponding task splitting result or task merging result; according to the task splitting result or task merging result, the corresponding task data is sent to the computing resource, so that the computing resource starts to execute a new operator computing task at the corresponding data location in the real-time data snapshot according to the task splitting result or task merging result.

[0077] In a specific task data allocation scheduling example, you can refer to Figure 3 The process shown. The old task state refers to the state when the task is suspended before the task data is redistributed. Offset represents the offset of the message queue. Then, after waiting for the savepoint (sp) to be completed, the task is suspended to ensure that there is no data loss. Then the task data is split and merged, and the computing resources (cluster parameter settings) for subsequent task data processing are set. Finally, the set cluster device can start a new task according to sp. The new task refers to a task that is started by the configured computing resources to calculate the task data that may be tilted.

[0078] The technical solution of this embodiment is to obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; each data item in the current queue indicator data, the current operator state data and the current state storage data is serialized and combined with the same data item in the corresponding historical data in chronological order; the serialized combination result corresponding to each data item is converted into a data format to obtain a serialized combination result in a preset format; the serialized combination result in a preset format corresponding to each data item is integrated to obtain the target data to be analyzed; the target data to be analyzed is input into a pre-trained data tilt prediction model to obtain a data tilt probability prediction value and a data tilt degree prediction value; when the data tilt probability prediction value is greater than a preset probability threshold, determine to perform task scheduling; determine the data computing resource requirements according to the data tilt amount corresponding to the data tilt degree prediction value, the message body size in the target message queue and the operator calculation complexity corresponding to the task data in the message body. Allocate the unconsumed task data in the message queue to the computing resources that meet the data computing resource requirements. The technical solution of the embodiment of the present invention solves the problem of data skew that may occur during the processing of distributed subtasks. The data skew situation can be predicted, and based on the prediction result of the data skew situation, the task data can be scheduled in time to ensure the real-time, accuracy and high reliability of the task data calculation task. The real-time, accuracy and high reliability of data processing in distributed subtasks are ensured.

[0079] Figure 4 The flowchart of a task data management method provided by an embodiment of the present invention is a flowchart of a task data management method provided by an embodiment of the present invention. The task data management method in this embodiment and the task data management method in the above embodiment belong to the same inventive concept, and further illustrates the process. The method can be executed by a task data management device, which can be implemented by software and / or hardware and integrated into a computer device with application development function.

[0080] like Figure 4 As shown, the task data management method of this embodiment includes the following steps:

[0081] S310: Obtain a preset model training data sample set.

[0082] Among them, the preset model training data sample set includes the queue indicator data sequence of the message queue when data skew occurs, the corresponding operator state data sequence, and the corresponding task data state storage data sequence.

[0083] The preset model training data sample set may include positive samples and negative samples, that is, sample data when data skew occurs and sample data when data skew does not occur. Positive samples can be formed by serializing the operating data and corresponding indicator data corresponding to historical data skew events in time series.

[0084] S320: Input samples in the preset model training data sample set into the data tilt prediction model to be trained to obtain a model output result.

[0085] Specifically, the data skew prediction model to be trained can be a model built based on a long short-term memory (LSTM) network, which includes a sequence model of multiple long short-term memory network layers and a fully connected layer, wherein each long short-term memory network layer includes a long short-term memory network unit with a preset output dimension, and a flag specifying whether the data is skewed.

[0086] These LSTM layers can capture the temporal dependencies in the input sequence and generate task-relevant feature representations. The dropout layer is used for regularization to prevent overfitting. The return sequence of the last LSTM layer is set to False. After the last LSTM layer, a fully connected layer is added, which uses a sigmoid activation function for binary classification.

[0087] S330: Calculate the model training loss based on the model output result and the corresponding sample data label, and reversely update the data tilt prediction model to be trained according to the model training loss to complete the model training.

[0088] In the process of updating the parameters of the data skew prediction model, the Adam optimizer can be used to adaptively adjust the learning rate and train the model by minimizing the binary cross entropy loss function.

[0089] S340: Obtain current queue indicator data of the target message queue, current operator state data corresponding to the target message queue, and current state storage data of task data corresponding to the target message queue.

[0090] S350: Serialize the current queue indicator data, the current operator state data, and the current state storage data with the corresponding historical data to obtain target data to be analyzed.

[0091] S360: Input the target data to be analyzed into a pre-trained data skew prediction model to obtain a predicted value of the probability of data skew and a predicted value of the degree of data skew.

[0092] S370. Schedule and manage the target task data in the message queue according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0093] The technical solution of this embodiment uses the serialization of the relevant indicator numbers when the historical data tilt occurs as samples to train a data tilt prediction model, and inputs the target data to be analyzed into the pre-trained data tilt prediction model in the application to obtain the predicted value of the probability of data tilt and the predicted value of the degree of data tilt; when the predicted value of the probability of data tilt is greater than the preset probability threshold, it is determined to perform task scheduling; according to the data tilt amount corresponding to the predicted value of the degree of data tilt, the size of the message body in the target message queue, and the operator calculation complexity corresponding to the task data in the message body, the data computing resource requirements are determined. The unconsumed task data in the message queue is allocated to the computing resources that meet the data computing resource requirements. The technical solution of the embodiment of the present invention solves the problem that data tilt may occur during the processing of distributed subtasks, and can predict the data tilt situation, and based on the prediction results of the data tilt situation, timely schedule the task data to ensure the real-time, accuracy and high reliability of the task data calculation task. Ensure the real-time, accuracy and high reliability of data processing in distributed subtasks.

[0094] Figure 5 A structural schematic diagram of a task data management device provided in an embodiment of the present invention. This embodiment can be applied to scheduling and management scenarios of task data in distributed computing tasks, especially in situations where the amount of task data in different distributed subtasks is unknown. The task data management device can be implemented by software and / or hardware and integrated into a computer terminal device with application development capabilities.

[0095] like Figure 5 As shown, the task data management device includes: a task analysis data acquisition module 410, an analysis data serialization module 420, a data tilt prediction module 430 and a task data management module 440.

[0096] Among them, the task analysis data acquisition module 410 is used to obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; the analysis data serialization module 420 is used to serialize the current queue indicator data, the current operator state data and the current state storage data with the corresponding historical data respectively to obtain the target data to be analyzed; the data skew prediction module 430 is used to input the target data to be analyzed into a pre-trained data skew prediction model to obtain the data skew probability prediction value and the data skew degree prediction value; the task data management module 440 is used to schedule and manage the target task data in the message queue according to the data skew probability prediction value and the data skew degree prediction value.

[0097] The technical solution of this embodiment is to obtain the current queue index data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; the current queue index data, the current operator state data and the current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed; the target data to be analyzed is input into the pre-trained data tilt prediction model to obtain the data tilt probability prediction value and the data tilt degree prediction value; the target task data in the message queue is scheduled and managed according to the data tilt probability prediction value and the data tilt degree prediction value. The technical solution of the embodiment of the present invention solves the problem that data tilt may occur during the processing of distributed subtasks, and can predict the data tilt situation, and based on the prediction results of the data tilt situation, timely schedule the task data to ensure the real-time, accuracy and high reliability of the task data calculation task. Ensure the real-time, accuracy and high reliability of data processing in distributed subtasks.

[0098] In an optional implementation, the analysis data serialization module 420 is specifically used for:

[0099] For each data item in the current queue indicator data, the current operator state data, and the current state storage data, serialize and combine them with the same data item in the corresponding historical data in chronological order;

[0100] Perform data format conversion on the serialized combination result corresponding to each data item to obtain a serialized combination result in a preset format;

[0101] The serialized combination results of the preset format corresponding to each data item are integrated to obtain the target data to be analyzed.

[0102] In an optional implementation, the task data management module 440 is specifically used to:

[0103] When the predicted value of the probability of data skew occurring is greater than a preset probability threshold, determining to perform task scheduling;

[0104] Determine the data computing resource requirements based on the data skew amount corresponding to the data skew prediction value, the message body size in the target message queue, and the operator computing complexity corresponding to the task data in the message body;

[0105] Allocate the unconsumed task data in the message queue to computing resources that meet the data computing resource requirements.

[0106] In an optional implementation, the task data management module 440 may be further configured to:

[0107] Save the real-time data snapshot of the operator computing task corresponding to the task data in the message queue, and suspend the operator computing task;

[0108] Split or merge operator computing tasks according to real-time data snapshots to obtain corresponding task splitting results or task merging results;

[0109] According to the task splitting result or the task merging result, the corresponding task data is sent to the computing resources, so that the computing resources start to execute new operator computing tasks at the corresponding data locations in the real-time data snapshot according to the task splitting result or the task merging result.

[0110] In an optional implementation, the analysis data serialization module 420 may also be used to:

[0111] Before obtaining the target data to be analyzed, data statistics are performed on the data items that need to be statistically analyzed in the current queue indicator data, the current operator state data, and the current state storage data to obtain corresponding target indicator data.

[0112] In an optional implementation, the task data management device further includes a model training module for training a data tilt prediction model, wherein the specific training process includes:

[0113] Obtain a preset model training data sample set, wherein the preset model training data sample set includes a queue indicator data sequence of the message queue when data skew occurs, a corresponding operator state data sequence, and a corresponding task data state storage data sequence;

[0114] Input samples in the preset model training data sample set into the data tilt prediction model to be trained to obtain the model output result;

[0115] The model training loss is calculated based on the model output results and the corresponding sample data labels, and the data tilt prediction model to be trained is reversely updated according to the model training loss to complete the model training.

[0116] In an optional embodiment, the data skew prediction model to be trained includes a sequence model of multiple long short-term memory network layers and a fully connected layer, wherein each long short-term memory network layer includes a long short-term memory network unit with a preset output dimension.

[0117] The task data management device provided in the embodiment of the present invention can execute the task data management method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0118] In a specific distributed subsystem example, in order to achieve the purpose of real-time full-link data stability and availability, the following settings are set: Figure 6 The functional modules shown are used to predict and process data skew.

[0119] exist Figure 6 It specifically includes real-time computing module, state storage module, data indicator monitoring module, LSTM module, instruction conversion module and scheduling module. During the computing task, the data in the data source-message queue is written to the storage medium or the new message queue through the "real-time computing module".

[0120] Among them, the working logic process of the "real-time calculation module" is as follows: Figure 7 As shown. Specifically: connect the message queue to the Flink data source, perform logical processing through operator operations such as windowing, filtering, conversion, processing, aggregation, and diversion, and then directly store the processed data or send it to the downstream (message queue, remote storage) in the form of a message queue. The above operator operations correspond to the keyby, map, flatmap, union, window, process and other operators in the Flink task. Each operator has complex calculation and processing logic, which requires caching historical data for a period of time and persisting it to remote storage, and calculating with new stream data, such as merging, addition, subtraction, cumulative multiplication and other multiple calculations. One or more cross operations.

[0121] "Data indicator monitoring module" Figure 8 In the data analysis and summary calculation module, the indicator information of the "data source-message queue", "real-time calculation module" and "state storage module" is regularly pulled / received, and the data is processed, serialized and structured, and sent to the "LSTM module".

[0122] The "LSTM module" is used to identify severe data skew and determine the degree of skewness in historical and real-time sequence information, and send the results of "whether it is skewed" and "degree of skewness" to the "instruction calculation module" and "instruction conversion module".

[0123] After receiving the monitoring data from the LSTM module, the "instruction conversion module" performs instruction calculation operations, outputs whether to perform task data scheduling data, that is, task change, parameter change, state change and other operation data, and converts them into relevant instructions and sends them to the "scheduling module". Cluster change instructions can be executable Flink Rest Api or Flink CLI syntax strings.

[0124] After the "scheduling module" receives the instruction issued by the "instruction conversion module", it calls the Rest Api or executes the CLI to execute or change the relevant instructions according to the queue, and performs the change scheduling of the "real-time computing module", "data source-message queue" and "state storage module". Specifically, the "scheduling module" obtains the "data indicator monitoring module" to obtain: message queue and Flink task operation history and real-time data, cluster status data, task status data and other data. According to the time series data generated by the serialization of these data, the data processing task plan and the degree of data tilt for the next time interval can be predicted, and then, based on the obtained predicted data tilt degree and data processing plan, the data processing strategy can be determined. The data processing strategy can be to execute the startup data tilt operation plan and not to execute the switch. Among them, when the data processing strategy is to execute the startup data tilt operation plan, according to the predicted data tilt degree, one of the following schemes can be executed: first, automatically call the RestAPI, according to the amount of data tilted, the standby cluster is resourced, and the state is copied from the state and remote storage through the data primary key (such as warehouse, merchant code, etc.) where the data tilt occurs, and transferred to the standby cluster. Second, when switching data streams, a snapshot operation of the task status will be performed first, and the current data stream displacement will be saved together with the task status to ensure that after the original task and multiple subtasks are switched, the data stream processing still starts from the data stream displacement and data snapshot status at the beginning of the previous one or more task switches.

[0125] Fig. 9 It shows that based on Figure 6 The complete process of scheduling and managing task data in each functional module. According to the data tilt prediction of the original task, split it to obtain the corresponding two subtasks. During the execution of the two subtasks, predict the data tilt of the subtasks, and merge or maintain the tasks according to the data tilt prediction of each subtask.

[0126] Fig.10 A schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Fig.10 A block diagram of an exemplary computer device 12 suitable for use in implementing embodiments of the present invention is shown. Fig.10The computer device 12 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention. The computer device 12 can be any terminal device with computing capabilities, such as an intelligent controller and server, a mobile phone and other terminal devices.

[0127] like Fig.10 As shown, the computer device 12 is in the form of a general-purpose computing device. The components of the computer device 12 may include, but are not limited to: one or more processors or processing units 16, a system memory 28, and a bus 18 that connects various system components (including the system memory 28 and the processing unit 16).

[0128] Bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, an accelerated graphics port, a processor or a local bus using any of a variety of bus architectures. By way of example, these architectures include, but are not limited to, an Industry Standard Architecture (ISA) bus, a Micro Channel Architecture (MAC) bus, an Enhanced ISA bus, a Video Electronics Standards Association (VESA) local bus, and a Peripheral Component Interconnect (PCI) bus.

[0129] The computer device 12 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 12, including volatile and non-volatile media, removable and non-removable media.

[0130] The system memory 28 may include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. The computer device 12 may further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, the storage system 34 may be used to read and write non-removable, non-volatile magnetic media ( Fig.10 not shown, usually called a "hard drive"). Although Fig.10 Not shown in the figure, a disk drive for reading and writing to a removable non-volatile disk (e.g., a "floppy disk"), and an optical disk drive for reading and writing to a removable non-volatile optical disk (e.g., a CD-ROM, a DVD-ROM, or other optical media) may be provided. In these cases, each drive may be connected to the bus 18 via one or more data medium interfaces. The system memory 28 may include at least one program product having a set (e.g., at least one) of program modules that are configured to perform the functions of the various embodiments of the present invention.

[0131] A program / utility 40 having a set (at least one) of program modules 42 may be stored, for example, in system memory 28, such program modules 42 including, but not limited to, an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment. Program modules 42 generally perform the functions and / or methods of the embodiments described herein.

[0132] The computer device 12 may also communicate with one or more external devices 14 (e.g., keyboards, pointing devices, displays 24, etc.), one or more devices that enable a user to interact with the computer device 12, and / or any device that enables the computer device 12 to communicate with one or more other computing devices (e.g., network cards, modems, etc.). Such communication may be performed via an input / output (I / O) interface 22. Furthermore, the computer device 12 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 20. As shown, the network adapter 20 communicates with the other modules of the computer device 12 via the bus 18. It should be understood that although Fig.10 Not shown, other hardware and / or software modules may be used in conjunction with computer device 12, including but not limited to microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0133] The processing unit 16 executes various functional applications and data processing by running the programs stored in the system memory 28, for example, implementing the task data management method provided in the embodiment of the present invention, which includes:

[0134] Obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue;

[0135] The current queue indicator data, current operator state data, and current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed;

[0136] Input the target data to be analyzed into the pre-trained data tilt prediction model to obtain the predicted value of the probability of data tilt and the predicted value of the degree of data tilt;

[0137] The target task data in the message queue is scheduled and managed according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0138] An embodiment of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When the program is executed by a processor, the task data management method provided by any embodiment of the present invention is implemented. The method includes:

[0139] Obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue;

[0140] The current queue indicator data, current operator state data, and current state storage data are serialized with the corresponding historical data to obtain the target data to be analyzed;

[0141] Input the target data to be analyzed into the pre-trained data tilt prediction model to obtain the predicted value of the probability of data tilt and the predicted value of the degree of data tilt;

[0142] The target task data in the message queue is scheduled and managed according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

[0143] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to: an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0144] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0145] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0146] Computer program code for performing the operation of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] It should be understood by those skilled in the art that the modules or steps of the present invention described above can be implemented by a general-purpose computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, optionally, they can be implemented by a program code executable by a computer device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0148] An embodiment of the present disclosure further provides a computer program product, including a computer program, which, when executed by a processor, implements the task data management method provided in any embodiment of the present disclosure.

[0149] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to perform the computer program code for the disclosed operation, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0150] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A task data management method, characterized in that: include: Obtain current queue indicator data of the target message queue, current operator state data corresponding to the target message queue, and current state storage data of task data corresponding to the target message queue; The current queue indicator data, the current operator state data and the current state storage data are serialized with the corresponding historical data to obtain target data to be analyzed; Inputting the target data to be analyzed into a pre-trained data tilt prediction model to obtain a predicted value of the probability of data tilt and a predicted value of the degree of data tilt; The target task data in the message queue is scheduled and managed according to the predicted value of the probability of data skew occurrence and the predicted value of the degree of data skew.

2. The method according to claim 1, characterized in that The current queue indicator data, the current operator state data, and the current state storage data are serialized with the corresponding historical period data to obtain target data to be analyzed, including: For each data item in the current queue indicator data, the current operator state data and the current state storage data, serialize and combine them with the same data item in the corresponding historical data in chronological order; Performing data format conversion on the serialized combination result corresponding to each data item to obtain a serialized combination result in a preset format; The serialized combination results of the preset format corresponding to each data item are integrated to obtain the target data to be analyzed.

3. The method according to claim 1, characterized in that The scheduling and managing of the target task data in the message queue according to the data skew occurrence probability prediction value and the data skew degree prediction value includes: When the predicted value of the probability of occurrence of data skew is greater than a preset probability threshold, determining to perform task scheduling; Determine the data computing resource requirement according to the data skew amount corresponding to the data skew degree prediction value, the message body size in the target message queue, and the operator calculation complexity corresponding to the task data in the message body; Allocate the unconsumed task data in the message queue to computing resources that meet the data computing resource requirements.

4. The method according to claim 3, characterized in that The allocating the unconsumed task data in the message queue to computing resources that meet the data computing resource requirements includes: Save the real-time data snapshot of the operator computing task corresponding to the task data in the message queue, and suspend the operator computing task; Splitting or merging operator computing tasks according to the real-time data snapshot to obtain corresponding task splitting results or task merging results; According to the task splitting result or the task merging result, the corresponding task data is sent to the computing resource, so that the computing resource starts to execute a new operator computing task at the data location corresponding to the task splitting result or the task merging result in the real-time data snapshot.

5. The method according to claim 2, characterized in that: Before obtaining the target data to be analyzed, the method further includes: Data statistics are performed on the data items that need to be statistically analyzed in the current queue indicator data, the current operator state data, and the current state storage data to obtain corresponding target indicator data.

6. The method according to any one of claims 1 to 5, characterized in that: The training process of the data tilt prediction model includes: Obtain a preset model training data sample set, wherein the preset model training data sample set includes a queue indicator data sequence of a message queue when data skew occurs, a corresponding operator state data sequence, and a corresponding task data state storage data sequence; Input samples in the preset model training data sample set into the data tilt prediction model to be trained to obtain a model output result; The model training loss is calculated based on the model output result and the corresponding sample data label, and the data tilt prediction model to be trained is reversely updated according to the model training loss to complete the model training.

7. The method according to claim 5, characterized in that The data tilt prediction model to be trained includes a sequence model of multiple long short-term memory network layers and a fully connected layer, wherein each long short-term memory network layer includes a long short-term memory network unit with a preset output dimension.

8. A task data management device, characterized in that: include: A task analysis data acquisition module is used to obtain the current queue indicator data of the target message queue, the current operator state data corresponding to the target message queue, and the current state storage data of the task data corresponding to the target message queue; An analysis data serialization module is used to serialize the current queue indicator data, the current operator state data and the current state storage data with the corresponding historical data to obtain target data to be analyzed; A data tilt prediction module is used to input the target data to be analyzed into a pre-trained data tilt prediction model to obtain a predicted value of the probability of data tilt and a predicted value of the degree of data tilt; The task data management module is used to schedule and manage the target task data in the message queue according to the predicted value of the probability of data skew and the predicted value of the degree of data skew.

9. A computer device, characterized in that: The computer device comprises: one or more processors; A memory for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the task data management method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the task data management method as described in any one of claims 1 to 7 is implemented.

11. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the computer program implements the task data management method according to any one of claims 1 to 7.