Data processing method and device, equipment, storage medium and computer program product

Abstract

The embodiment of the invention discloses a data processing method. The method comprises the following steps: calling a master node of a target structure tree comprising a master node corresponding to a master process and a slave node corresponding to a slave process; if the slave nodes of the target structure tree are updated, obtaining a to-be-scheduled data stream from the slave nodes of the target structure tree through the main process of the main node, and determining a target slave node based on the load of the slave nodes of the target structure tree; and distributing the to-be-scheduled data stream to the target slave node through the master process of the master node, and processing the to-be-scheduled data stream through the slave process of the target slave node. Therefore, the load of each slave node is fully considered in the data stream scheduling process after the slave nodes of the target structure tree are updated, and the problem that the load among the slave nodes is unbalanced when the data stream is scheduled after the slave nodes are added or deleted in the related technology is solved. The embodiment of the invention further discloses a data processing device and equipment, a storage medium and a computer program product.

Description

Technical Field

[0001] This application relates to data stream processing technology in the field of computers, and more particularly to a data processing method, apparatus, device, storage medium, and computer program product. Background Technology

[0002] In the education sector, many exam-related video monitoring scenarios are required, such as dance exams for art students and online exams. In these video services, the video stream processing is generally divided into three relatively loosely coupled processes: front-end, mid-end, and back-end. The front-end pushes the video stream, sending real-time video from various cameras to a video streaming server. The mid-end pulls the stream, retrieving it from the server and restoring it to a visual video file. The back-end handles the business logic, processing the restored video streams. For the mid-end pull processing, multiple machines form a pull cluster to pull video streams from multiple servers and retrieve them locally. Then, the stream is frame-segmented, dividing it into individual images based on its original frame rate. Finally, each image is sent to the back-end for actual business processing.

[0003] Therefore, related technologies logically design the cluster in a master-slave structure. The master node acts as a scheduler, allocating each stream to the slave nodes for processing. Real-time heartbeat monitoring via middleware (ZooKeeper) keeps the master aware of the slave nodes' status. Slave nodes respond to the master's scheduling, pulling and processing the allocated streams. The master maintains interval communication with slaves or uses real-time heartbeat monitoring via middleware to know the slave's status. When a slave node crashes or its processing of a particular stream is terminated, the master reschedules the stream. However, due to the limited processing capacity of a single slave node, the number of slave servers increases or decreases as the number of video streams to be processed increases or decreases. Consequently, the number of slave nodes also needs to be adjusted. However, related technologies suffer from load imbalances among slave nodes during data stream scheduling after adding or removing slave nodes. Summary of the Invention

[0004] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, device, storage medium, and computer program product, which solves the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies.

[0005] The technical solution of this application is achieved as follows:

[0006] A data processing method, the method comprising:

[0007] Invoke the main node of the target structure tree; wherein, the target structure tree includes the main node corresponding to the main process and the slave nodes corresponding to the slave processes;

[0008] If the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled from the slave nodes of the target structure tree, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree;

[0009] The master process of the master node distributes the data stream to be scheduled to the target slave node, and the slave process of the target slave node processes the data stream to be scheduled.

[0010] In the above scheme, if the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled from the slave nodes of the target structure tree, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree, including:

[0011] If a new slave node is added to the target structure tree, and the first target parameter determined based on the load of the slave node is greater than a preset threshold, the master process of the master node obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream; wherein, the scheduled data stream includes the first scheduled data stream; and the target slave node includes the first target slave node.

[0012] In the above scheme, the master process of the master node obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream, including:

[0013] The master process of the master node determines the first slave node whose load meets the target load condition from the slave nodes of the target structure tree, and uses the target load prediction model to determine the first data stream from the first slave node;

[0014] Based on the estimated first load of the slave node after each first other slave node carries the first data stream to be scheduled, a first target value is determined; wherein, the first other slave node is the slave node other than the first slave node in the slave node of the target structure tree;

[0015] When the load of the slave node corresponding to the first target value is updated to the load after having the first data flow, the master process of the master node determines the second slave node whose load meets the target load condition from the slave nodes of the target structure tree, and uses the target load prediction model to determine the second data flow from the second slave node;

[0016] A second target value is determined based on the estimated second load of the slave node after each of the second other slave nodes carries the second data stream;

[0017] If the second target value is greater than or equal to the first target value, the slave node corresponding to the first target value is determined as the first target slave node; wherein, the first data stream to be scheduled includes the first data stream.

[0018] In the above scheme, the first target value is determined based on the estimated first load of the slave node after each first other slave node carries the first data stream, including:

[0019] Estimate the first load of each of the first other slave nodes after it has carried the first data stream;

[0020] For each estimation operation, multiple first values ​​are calculated for multiple first loads, and the first target value is determined from the multiple first values.

[0021] The method in the above scheme further includes:

[0022] If the second target value is less than the first target value, when the load of the slave node corresponding to the second target value is updated to the load after having the second data stream, the master process of the master node determines the third slave node whose load meets the target load condition from the slave nodes of the target structure tree, until the Nth target value is determined; wherein, the first data stream to be scheduled includes the second data stream;

[0023] The slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value are designated as the first target slave node.

[0024] The method in the above scheme further includes:

[0025] Retrieve the first sample load from the first sample node in the sample structure tree;

[0026] After allocating the sample data stream to the first sample slave node, the second sample load is obtained after the first sample slave node carries the sample data stream;

[0027] Based on the first sample load and the second sample load, the sample stream load of the sample data stream is determined;

[0028] The initial load prediction model is trained based on the sample flow load to obtain the target load prediction model.

[0029] The method in the above scheme further includes:

[0030] After the first data stream to be scheduled is allocated to the first target slave node through the master process of the master node, the second target parameter is determined based on the load of the slave nodes in the target structure tree.

[0031] If the second target parameter is greater than the preset threshold, the master process of the master node obtains the second scheduled data stream from the second scheduled slave node determined by the load of the slave node based on the target structure tree, and determines the second target slave node based on the estimated load of the slave node after carrying the second scheduled data stream; wherein, the scheduled data stream also includes the second scheduled data stream; the target slave node also includes the second target slave node.

[0032] The method in the above scheme further includes:

[0033] If a third slave node to be scheduled is deleted in the target structure tree, the target slave node is determined from the other slave nodes in the target structure tree based on their loads, through the master process of the master node.

[0034] A data processing apparatus, the apparatus comprising:

[0035] The invocation unit is used to invoke the main node of the target structure tree; wherein, the target structure tree includes the main node corresponding to the main process and the slave nodes corresponding to the slave processes;

[0036] The acquisition unit is configured to, if the slave nodes of the target structure tree are updated, acquire the data stream to be scheduled from the slave nodes of the target structure tree through the main process of the master node, and determine the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree;

[0037] The processing unit is configured to allocate the data stream to be scheduled to the target slave node through the master process of the master node, and to process the data stream to be scheduled through the slave process of the target slave node.

[0038] A data processing device, the device comprising: a processor, a memory, and a communication bus;

[0039] The communication bus is used to realize the communication connection between the processor and the memory;

[0040] The processor is used to execute the data processing program in the memory to implement the steps of the above-described data processing method.

[0041] A computer-readable storage medium storing one or more programs that can be executed by one or more processors to implement the steps of the data processing method described above.

[0042] A computer program product comprising a computer program that, when executed by a processor, implements the aforementioned data processing method.

[0043] The data processing method, apparatus, device, storage medium, and computer program product provided in the embodiments of this application can invoke the master node of a target structure tree, which includes a master node corresponding to a master process and slave nodes corresponding to slave processes. If the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled from the slave nodes of the target structure tree, determines the target slave node based on the load of the slave nodes, and then allocates the data stream to be scheduled to the target slave node through the master process of the master node. The slave processes of the target slave nodes then process the data stream to be scheduled. In this way, after the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled and the target slave node to which the data stream to be scheduled needs to be scheduled based on the updated load of each slave node. Then, the master process of the master node schedules the data stream to be scheduled to the target slave node. That is, the load of each slave node is fully considered in the data stream scheduling process after the slave nodes of the target structure tree are updated, ensuring that the load among the slave nodes after scheduling is balanced. This solves the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies. Attached Figure Description

[0044] Figure 1 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0045] Figure 2 A schematic diagram of the target structure tree in a data processing method provided in an embodiment of this application;

[0046] Figure 3 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0047] Figure 4 A flowchart illustrating another data processing method provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application; Detailed Implementation

[0050] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application.

[0051] It should be understood that the phrases "embodiments of this application" or "foreign embodiments" throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment is included in at least one embodiment of this application. Therefore, "embodiments of this application" or "in the foreign embodiments" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. In the various embodiments of this application, the sequence numbers of the above-described processes do not imply a sequential order of execution; the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. The sequence numbers of the above-described embodiments are merely descriptive and do not represent the superiority or inferiority of the embodiments.

[0052] Unless otherwise specified, any step in the embodiments of this application performed by the electronic device may be executed by the processor of the electronic device. It is also worth noting that the embodiments of this application do not limit the order in which the electronic device performs the following steps. Furthermore, the methods used to process data in different embodiments may be the same or different methods. It should also be noted that any step in the embodiments of this application can be executed independently by the electronic device; that is, when the electronic device performs any step in the following embodiments, it may not depend on the execution of other steps.

[0053] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0054] This application provides a data processing method, which can be applied to a data processing device, as described above. Figure 1 As shown, the method may include the following steps:

[0055] Step 101: The data processing device calls the main node of the target structure tree.

[0056] The target structure tree includes the master node corresponding to the master process and the slave node corresponding to the slave process.

[0057] In this embodiment of the application, the target structure tree may further include push and pull nodes corresponding to the data stream.

[0058] In this embodiment, the master node of the target structure tree can be used to receive external Hypertext Transfer Protocol (HTTP) requests that instruct the processing of the data stream to be processed. These requests are forwarded by a reverse proxy server, and specific processing operations are performed based on the data processing requests. The master and slave nodes in the target structure tree correspond to machine nodes, i.e., machine nodes include master and slave nodes. Furthermore, the push and pull nodes in the target structure tree are child nodes of the root node.

[0059] In the embodiments of this application, such as Figure 2 As shown, the proposed solution can be applied to a ZooKeeper cluster. To ensure high availability, the ZooKeeper cluster includes at least three servers. Specifically, a pull node, a push node, and a machine node are established on a selected path (root node / ) in ZooKeeper. Under the machine node, a master node and slave nodes are established. Under the master node, working nodes and waiting nodes are established. The working nodes under the master node correspond to the currently working master server, and are ephemeral nodes. The waiting nodes under the master node correspond to the master server that is waiting, and can monitor changes to the working nodes.

[0060] In this embodiment, the application can be applied to a video stream scheduling and processing system, which can be jointly scheduled and processed by a master node and slave nodes. The working node receives requests from external systems to add or delete video streams and schedules the video streams. The waiting node monitors changes in the working node; when a temporary working node disappears, all waiting nodes compete to take over the temporary node. Due to the uniqueness of the working node, only one waiting node becomes the working node. The waiting node that takes over the working node switches to the working state, thus ensuring high availability of the master node cluster. The slave node responds to the master node's requests to add or delete streams, pulls streams, creates and destroys child processes, and reports processing results.

[0061] It should be noted that after the master node process starts, it will compete to become a working node. However, since ZooKeeper does not allow nodes with the same name under a specified path, only one working node exists, and the rest become waiting nodes. After startup, the master node creates ephemeral nodes under the working nodes and also creates ephemeral nodes under the waiting nodes. After the slave node process starts, it will create ephemeral nodes under the slave node. Each time a slave node adds a child process to pull streams, it will create a corresponding ephemeral node under the pull node. All ephemeral nodes under the machine node, except for the working node, are named in the form of PID_IP.

[0062] Step 102: If the slave nodes of the target structure tree are updated, the data processing device obtains the data stream to be scheduled from the slave nodes of the target structure tree through the master process of the master node, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree.

[0063] In this application embodiment, updating the slave nodes of the target structure tree can refer to an increase or decrease in the number of target slave nodes; when this application is applied to a video stream scheduling and processing system, updating the slave nodes of the target structure tree can refer to an increase or decrease in the number of slave nodes.

[0064] Specifically, the number of slave nodes in the target structure tree can be increased or decreased based on the current load of all slave nodes. When the overall load is low, the number of slave nodes needs to be reduced, i.e., the process of a certain slave node is shut down. In this case, the master node needs to schedule the data streams to be scheduled on the shut-down slave nodes. When the overall load is high, slave nodes need to be added to share the load. In this case, the master process of the master node needs to obtain the data streams to be scheduled based on the load of each slave node in the target structure tree after adding slave nodes. After obtaining the data streams to be scheduled, the load of each slave node in the target structure tree after pre-scheduling the data streams to be scheduled to other slave nodes besides the slave node corresponding to the data streams to be scheduled is calculated. Thus, the target slave node is determined based on the obtained load of each slave node.

[0065] Step 103: The data processing device distributes the data stream to be scheduled to the target slave node through the master process of the master node, and processes the data stream to be scheduled through the slave process of the target slave node.

[0066] In this embodiment of the application, the data processing device can process the data stream to be scheduled through a child process in the main process of the target slave node; wherein, the child process in the main process of the target slave node can be a newly created child process.

[0067] It should be noted that after the data stream to be scheduled and the target slave node are determined, the master process of the master node can allocate the data stream to the target slave node through a remote procedure call (RPC).

[0068] In this embodiment, after the master node distributes the data stream to be scheduled to the target slave node through the working node, the master node starts a child process (i.e., a slave process) on the master process of the target slave node. If the child process of the target slave node is successfully established, the master node will create a temporary node for the child process under the pull node and return a successful establishment message to the master node. At the same time, the child process of the target slave node starts to pull the data stream to be scheduled. If an error occurs during the pull process (i.e., the data corresponding to the Uniform Resource Locator (URL) of the data stream has disappeared), the child process of the target slave node will pull the stream again after a period of time. Meanwhile, the child process of the target slave node will monitor whether the master process of the target slave node is running normally. If the master process of the target slave node exits unexpectedly, the child process of the target slave node cannot be controlled by the master process. At this time, the child process of the target slave node will terminate.

[0069] Correspondingly, if the child process fails to be created, a failure message is returned to the master node. Upon receiving the failure message, the master node randomly selects a target slave node from the target tree structure and attempts to create the child process on the master process of the target slave node again. If the child process of the target slave node is successfully created, the data stream to be scheduled is processed through the created child process. If the child process of the target slave node fails to be created again, the process continues to retry. If the failure message is returned every time after several retries, the master node abandons the process of creating the streaming process.

[0070] The data processing method provided in the embodiments of this application can, after the slave nodes in the target structure tree are updated, obtain the data stream to be scheduled and the target slave node to which the data stream needs to be scheduled based on the load of each updated slave node through the master process of the master node, and then schedule the data stream to be scheduled to the target slave node through the master process of the master node. That is, in the data stream scheduling process after the slave nodes in the target structure tree are updated, the load of each slave node is fully considered, ensuring that the load among the slave nodes after scheduling is balanced, and solving the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies.

[0071] Based on the foregoing embodiments, embodiments of this application provide a data processing method, referring to... Figures 3-4 As shown, the method may include the following steps:

[0072] Step 201: The data processing device calls the main node of the target structure tree.

[0073] The target structure tree includes the master node corresponding to the master process and the slave node corresponding to the slave process.

[0074] In this embodiment of the application, the target structure tree may further include push and pull nodes corresponding to the video stream, and the pull node may correspond to the data stream being processed, while the push node may correspond to the unprocessed data stream and the data stream being processed; wherein, the unprocessed data stream may be a newly added data stream; the data stream being processed may include the data stream to be scheduled and the data stream to be deleted.

[0075] In this application embodiment, the solution can be applied to scenarios of preventing cheating in online exams and dance scenarios in art exams, and the data stream can refer to a video stream.

[0076] It should be noted that, as Figure 3 As shown, steps 202 and 204 can be executed after step 201.

[0077] Step 202: If a new slave node is added to the target structure tree, and the first target parameter determined based on the load of the slave node is greater than a preset threshold, the data processing device obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node through the master process of the master node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream.

[0078] The data stream to be scheduled includes the first data stream to be scheduled; the target slave node includes the first target slave node.

[0079] In this embodiment, the first target parameter characterizes the load balancing degree among the slave nodes of the target structure tree. In one possible implementation, the first target parameter may refer to the variance value calculated based on the load of the slave nodes of the target structure tree; the preset threshold may be a critical value determined according to the application scenario.

[0080] In this embodiment, if a new slave node is added to the target structure tree, the master process of the working node of the master node will obtain the load of each new slave node at fixed intervals and calculate the variance value based on the obtained load of each slave node. If the calculated variance value is greater than a preset threshold, it indicates that the load difference between the slave nodes is large and adjustment is required.

[0081] Specifically, the load of each slave node after the addition of a new slave node is calculated, and the first slave node to be scheduled is determined from all slave nodes based on the load of each slave node; further, the load of each data stream carried on the first slave node to be scheduled is calculated by using a target load prediction model, and the first data stream to be scheduled is obtained based on the load of each data stream; then, based on the predicted target structure tree where other slave nodes besides the first slave node to be scheduled carry the load of all slave nodes after the first data stream to be scheduled, the first target slave node is determined.

[0082] It should be noted that we can assume that the first data stream to be scheduled is scheduled to other slave nodes in the target structure tree besides the first slave node to be scheduled. That is, we can assume that the other slave nodes in the target structure tree besides the first slave node to be scheduled are carrying the first data stream to be scheduled. In this case, we estimate the load of all slave nodes in the target structure tree, and then determine the first target slave node based on the estimated load of all slave nodes. In other words, in the process of obtaining the first target slave node, the first data stream to be scheduled is not actually assigned to other slave nodes in the target structure tree besides the first slave node to be scheduled. Therefore, we do not need to perform actual data stream scheduling. The first target slave node is determined based on the estimated load of the slave nodes after they have carried the first data stream to be scheduled. This significantly improves the efficiency of determining the target slave node, and thus improves the efficiency of data processing.

[0083] It should be noted that step 202 can be achieved in the following way:

[0084] Step 202a: The data processing device determines the first slave node whose load meets the target load condition from the slave nodes in the target structure tree through the master process of the master node, and determines the first data stream from the first slave node using the target load prediction model.

[0085] In the embodiments of this application, the target load condition is used to determine whether the first slave node is the slave node with the largest load in the target structure tree; the target load prediction model can be obtained by training the initial load prediction model.

[0086] In this embodiment, the set of slave nodes in the target structure tree can be represented by s; wherein, the set of slave nodes may include all currently running slave nodes. In one feasible implementation, s = {s1, s2, s3, ...}; and, the slave node with the highest load can be determined as the first slave node by calculating the load of each slave node in the target structure tree.

[0087] In this embodiment of the application, the frame rate, resolution and other parameters of each data stream carried on the first slave node of the target structure tree are input into the target load prediction model. After processing the relevant parameters of the input data stream, the target load prediction model can obtain the load of each data stream among all the data streams carried on the first slave node, and determine the data stream with the largest load on the first slave node as the first data stream.

[0088] The target load prediction model can be trained in the following way:

[0089] A1. The data processing device obtains the first sample load of the first sample slave node in the sample slave node of the sample structure tree.

[0090] In this embodiment, the first sample load of the first sample slave node can be the load of the current system when the first sample slave node is first started, that is, the load when the first sample slave node is not carrying data flow. At this time, the load of the first sample slave node can be determined by the load calculation formula. In one possible implementation, the load calculation formula is: L = f(c,m,i,d); where L can represent the load of each slave node in the target structure tree, c can represent the consumption ratio of the central processing unit (CPU), m can represent memory usage, i can represent network input / output (I / O), and d can represent disk input / output; and the weights of CPU consumption ratio, memory usage, network I / O, and disk I / O can be determined according to the application scenario, which is not specifically limited here.

[0091] A2. The data processing device allocates the sample data stream to the first sample slave node, obtains the second sample load after the first sample slave node carries the sample data stream, and determines the sample stream load of the sample data stream based on the first sample load and the second sample load.

[0092] In this embodiment of the application, the second sample load can be the load of the first sample slave node after scheduling, calculated by the load calculation formula after the sample data stream is allocated to the first sample slave node; furthermore, the difference between the first sample load and the second sample load can be calculated, and the calculated difference can be determined as the sample stream load of the sample data stream.

[0093] A3. The data processing equipment trains the initial load prediction model based on the sample stream load to obtain the target load prediction model.

[0094] The initial load prediction model can refer to a fully connected neural network model that can predict the load of data streams; that is, the data processing device can train the fully connected neural network model based on the sample stream load to obtain the target load prediction model.

[0095] Specifically, the frame rate, resolution, resolution width, resolution height, format, encoding rate, audio sampling rate, and audio encoding rate of the data stream can be used as inputs to a fully connected neural network model, with the data stream load as the output, to obtain an initial load prediction model. Then, the load of the data stream is estimated based on this initial model. When the estimated load differs significantly from the sample stream load obtained through the load calculation formula, the parameters of the initial load prediction model are adjusted, and the adjusted model is trained again based on the sample stream load until a target load prediction model is obtained. Furthermore, the trained target load prediction model can accurately predict the load of the data stream; that is, using the target load prediction model to calculate the data stream load makes the calculated load more accurate.

[0096] Step 202b: The data processing device determines the first target value based on the estimated first load of each of the first other slave nodes after carrying the first data stream.

[0097] Wherein, the first other slave node refers to the slave node in the target structure tree other than the first slave node; the first load may refer to the load of each slave node in the target structure tree after the first data stream is pre-scheduled to any of the first other slave nodes.

[0098] In this embodiment of the application, a first load can be obtained by pre-scheduling the first data stream to one of the first other slave nodes, and then a variance value can be obtained by calculating the first load. It should be noted that each time the first data stream is pre-scheduled to a first other slave node, a first load is obtained, and thus a variance value is obtained. Therefore, pre-scheduling the first data stream to each first other slave node will result in multiple first loads, and thus multiple variance values. Accordingly, the smallest variance value among the multiple variance values ​​can be determined as the first target value.

[0099] It should be noted that step 202b can be achieved through steps 202b1 to 202b2:

[0100] Step 202b1: The data processing device estimates the first load of each slave node after it has carried the first data stream.

[0101] In this embodiment, the first data stream is pre-scheduled to each of the first other slave nodes, and the load of the slave nodes in the target structure tree is estimated after each pre-scheduling of the first data stream, so as to obtain multiple first loads of the slave nodes in the target structure tree. It should be noted that pre-scheduling the first data stream to each of the first other slave nodes means assuming that the first data stream is scheduled to each of the first other slave nodes, rather than actually allocating the first data stream to each of the first other slave nodes.

[0102] Step 202b2: The data processing device calculates multiple first values ​​for each estimation operation based on multiple first loads, and determines a first target value from the multiple first values.

[0103] Here, the first value can refer to the variance value obtained by calculating the load of the slave nodes in the target structure tree after each prediction operation. It should be noted that the smaller the variance, the smaller the first value, and the higher the load balancing degree among the slave nodes of the target structure tree.

[0104] In this embodiment, after each estimation operation, i.e., after each pre-scheduling of the first data stream to one of the first other slave nodes, a variance value (i.e., a first value) is calculated on the multiple first loads of the current slave node. It should be noted that each time the first data stream is pre-scheduled to one of the first other slave nodes, a first value is obtained. Therefore, pre-scheduling the first data stream to each of the first other slave nodes will result in multiple first values ​​(i.e., variance values). At this time, the smallest first value (i.e., variance value) is taken as the first target value.

[0105] Step 202c: When the load of the slave node corresponding to the first target value is updated to the load after the first data flow, the data processing device determines the second slave node whose load meets the target load condition from the slave nodes in the target structure tree through the master process of the master node, and determines the second data flow from the second slave node using the target load prediction model.

[0106] The second slave node can refer to the slave node with the largest load in the target structure tree after the load of the slave node corresponding to the first target value is updated to the load after the first data flow.

[0107] It should be noted that updating the load of the slave node corresponding to the first target value to the load after having the first data stream can refer to the load of the slave node corresponding to the first target value after the first data stream is allocated to it; that is, the actual load of the slave node corresponding to the first target value has not changed, only the load of the slave node corresponding to the first target value is pre-updated to the load after having the first data stream.

[0108] In this embodiment of the application, the load of the slave nodes in the pre-scheduled target structure tree can be calculated by the load calculation formula, and the master process of the master node determines the slave node with the largest load in the target structure tree based on the calculated load of the slave nodes, and then the slave node with the largest load is taken as the second slave node; at this time, the load of each data stream on the second slave node can be estimated by the target load prediction model, and the data stream with the largest load on the second slave node is determined as the second data stream.

[0109] Step 202d: The data processing device determines the second target value based on the estimated second load of each second other slave node after it carries the second data stream.

[0110] Wherein, the second other slave nodes are the slave nodes in the target structure tree other than the second slave node; the second load can refer to the load of each slave node in the target structure tree after the second data stream is pre-scheduled to any of the second other slave nodes.

[0111] In this embodiment, the second data stream is pre-scheduled to each of the other second slave nodes, and the load of the target slave node after each pre-scheduling of the second data stream is estimated to obtain multiple second loads of the slave nodes in the target structure tree. It should be noted that pre-scheduling the second data stream to each of the other second slave nodes means assuming the second data stream is scheduled to each of the other second slave nodes, rather than actually allocating the second data stream to each of the other second slave nodes. Then, the average load of the slave nodes in the target structure tree is obtained by calculating the load of each slave node included in the second load. Finally, the variance value, i.e., the second value, is determined based on the number of slave nodes in the target structure tree, the load of each slave node, and the average load of the slave nodes in the target structure tree.

[0112] It should be noted that the second value may refer to the variance obtained after calculating the load of the slave nodes in the target structure tree after each pre-scheduling. The smaller the variance, the smaller the second value, and the higher the load balancing degree among the slave nodes of the target structure tree.

[0113] Furthermore, since each calculation on a second load yields a second value, performing calculations on each second load will result in multiple second values ​​(variance values). At this point, the smallest second value (variance value) is taken as the second target value.

[0114] It should be noted that after step 202d, steps 202e or 202f to 202g can be executed.

[0115] Step 202e: If the second target value is greater than or equal to the first target value, the data processing device determines the slave node corresponding to the first target value as the first target slave node.

[0116] The first data stream to be scheduled includes the first data stream; the first target slave node can refer to the slave node to which the first data stream is scheduled, which is the slave node with the highest load balancing degree among the slave nodes of the target structure tree after the first data stream is scheduled.

[0117] In this embodiment, if the second target value is greater than or equal to the first target value, that is, the variance value obtained by calculating the load of the slave nodes after the second pre-scheduling is greater than or equal to the variance value obtained by calculating the load of the slave nodes after the first pre-scheduling, it means that the load balancing degree among the slave nodes after the second pre-scheduling is lower than that after the first pre-scheduling. This indicates that the first pre-scheduling scheme is already optimal. At this time, there is no need to continue scheduling the data flow, and the first pre-scheduling scheme is taken as the final scheduling scheme. That is, the slave node corresponding to the first target value that makes the load balancing degree among the slave nodes in the target structure tree after pre-scheduling is the first target slave node.

[0118] Step 202f: If the second target value is less than the first target value, when the load of the slave node corresponding to the second target value is updated to the load after the second data flow, the data processing device determines the third slave node whose load meets the target load condition from the slave nodes in the target structure tree through the master process of the master node, until the Nth target value is determined.

[0119] The first data stream to be scheduled also includes the second data stream.

[0120] In this embodiment, if the second target value is less than the first target value, it indicates that the load balancing degree among the slave nodes after the second data flow pre-scheduling is higher than that after the first data flow pre-scheduling. In other words, the scheduling scheme determined after the first pre-scheduling is not optimal, and further data flow pre-scheduling is needed until the optimal scheduling scheme is obtained. That is, after updating the load of the slave node corresponding to the second target value to the load with the second data flow, steps 202a to 202g can be used to determine the slave node with the largest load from the slave nodes in the target structure tree after the second pre-scheduling as the third slave node. Then, based on the load of each data flow carried by the third slave node, the third slave node is selected from the slave nodes. The process involves obtaining a third data stream from the nodes and determining multiple third values ​​based on the estimated load of each of the other third slave nodes in the target tree structure after the third data stream is carried. The smallest third value is then designated as the third target value. Finally, if the third target value is greater than or equal to the second target value, the loop stops, i.e., the data stream scheduling stops. If the third target value is less than the second target value, it indicates that the current scheduling scheme is not optimal, and the process needs to continue iteratively according to steps 202a to 202g until the Nth target value is determined. At this point, the scheduling scheme determined after N iterations is optimal, where N can be a positive integer greater than or equal to 2. The other third slave nodes are the slave nodes in the target tree structure other than the third slave node. It should be noted that the loop can stop after N+1 iterations, where N can be a value determined based on the specific application scenario and historical data to obtain the optimal scheduling scheme.

[0121] Step 202g: The data processing device uses the slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value as the first target slave node.

[0122] It should be noted that, in order to ensure the highest load balancing among the slave nodes of the target structure tree after data flow scheduling, multiple data flows can be scheduled to achieve load balancing among the slave nodes. That is, the data flow to be scheduled can include multiple data flows such as the first data flow and the second data flow, and the first target slave node can include multiple slave nodes such as the slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value.

[0123] Step 202h: After the data processing device distributes the first data stream to be scheduled to the first target slave node through the master process of the master node, it determines the second target parameters based on the load of the slave node in the target structure tree.

[0124] The second objective parameter characterizes the load balancing degree among the slave nodes of the target structure tree after the first target slave node carries the first scheduled data stream. In one implementation, the second objective parameter may refer to the variance value obtained by calculating the load of the slave nodes in the target structure tree after the first scheduled data stream is allocated to the first target slave node.

[0125] In this embodiment, if the second target value is greater than or equal to the first target value, the first data stream is directly allocated to the first target slave node through the master process of the master node; if the second target value is less than the first target value, the first data stream in the first data stream to be scheduled is allocated to the slave node corresponding to the first target value in the first target slave node through the master process of the master node, and the second data stream in the first data stream to be scheduled is allocated to the slave node corresponding to the second target value in the first target slave node, until finally the Nth data stream in the first data stream to be scheduled is allocated to the slave node corresponding to the Nth target value in the first target slave node.

[0126] After allocating the first data stream to be scheduled to the first target slave node, the load of each slave node in the scheduled target structure tree is calculated to obtain the second target parameters.

[0127] Step 202i: If the second target parameter is greater than the preset threshold, the data processing device obtains the second scheduled data stream from the second scheduled slave node determined based on the load of the slave node through the master process of the master node, and determines the second target slave node based on the estimated load of the slave node after carrying the second scheduled data stream.

[0128] The data stream to be scheduled also includes a second data stream to be scheduled; the target slave node also includes a second target slave node.

[0129] In this embodiment of the application, if the second target parameter is greater than the preset threshold, it means that after the first data stream to be scheduled is allocated to the first target slave node, the load difference between the slave nodes in the target structure tree is still very large. At this time, it is necessary to adjust the data stream allocation between the slave nodes again.

[0130] Specifically, the load of each slave node in the target structure tree is calculated, and the second slave node to be scheduled is determined from all slave nodes based on the load of each slave node by the master process of the master node. Further, according to steps 202a to 202h, the load of each data stream carried on the second slave node to be scheduled can be calculated using the target load prediction model, and the data stream with the largest load is determined as the second data stream to be scheduled. Then, based on the predicted load of all slave nodes in the target structure tree after the other slave nodes in the target structure tree except the second slave node to be scheduled carry the second data stream to be scheduled, the second target slave node is determined.

[0131] In this embodiment, when the number of slave nodes in the target structure tree increases, the load of the slave cluster can be evenly distributed without interrupting stream processing. During each batch scheduling, the stream load is first estimated using the target load prediction model. After scheduling, the actual variance is checked. Since the variance of the load change obtained from the predicted scheduling scheme may not meet the preset requirements, if the requirements are not met, the "prediction-scheduling-checking" process is repeated to ensure that the load balancing degree of the slave nodes in the entire target structure tree is reduced to below the preset threshold. At this point, the load among the slave nodes is balanced.

[0132] In other embodiments of this application, such as Figure 4 As shown, steps 203 to 204 can be executed after step 201.

[0133] Step 203: If a third slave node to be scheduled is deleted in the target structure tree, the data processing device determines the target slave node from the other slave nodes in the target structure tree based on the load of the other slave nodes besides the third slave node to be scheduled, through the master process of the master node.

[0134] In this embodiment, when the overall load of the slave nodes in the target structure tree is low, it is necessary to reduce the number of slave nodes to release resources. This requires shutting down the main process of the third slave node to be scheduled, i.e., deleting the third slave node to be scheduled. Correspondingly, the child process of the pull node corresponding to the deleted slave node (i.e., the third slave node to be scheduled) will terminate, and all temporary nodes under the corresponding pull node will also disappear. Since the master node is monitoring the changes of the nodes under the pull node, when the temporary nodes under the pull node disappear, it will trigger the master node to obtain the data stream that exists in the push node but does not exist in the pull node as the data stream to be scheduled. That is, each data stream carried by the deleted slave node is taken as the data stream to be scheduled. At this time, the load of the other slave nodes other than the deleted slave node can be calculated, and the master node's main process can determine the target slave node from the other slave nodes other than the third slave node to be scheduled based on the calculated load.

[0135] Step 204: The data processing device distributes the data stream to be scheduled to the target slave node through the master process of the master node, and processes the data stream to be scheduled through the slave process of the target slave node.

[0136] In this embodiment, regardless of whether the number of slave nodes in the target structure tree increases or decreases, after determining the scheduling scheme for each data stream to be scheduled, the master process of the working node of the master node needs to create a corresponding target child node on the target slave node corresponding to the data stream to be scheduled, and assign each data stream to be scheduled to the corresponding target child node. At this time, the target child node only performs the action of pulling the stream and does not perform actual business processing. At the same time, the working node of the master node will notify the source slave node where the data stream to be scheduled is located to close the source child process of the data stream to be scheduled. If the target slave node detects that the source pulling node corresponding to the data stream to be scheduled has disappeared, it will start to perform actual business processing on the data stream to be scheduled through the slave process of the target child node, instead of the master node first closing the source process of the data stream, then reallocating the data stream to the slave node, and finally the slave node pulling the data stream, as in the related technologies. This solves the problem of frame interruption caused by the data stream scheduling process taking seconds in the related technologies and the inability of business to continue during the process, and ensures that the data stream achieves frame-free switching during the scheduling process.

[0137] It should be noted that before the target child node pulls the stream, the master node also needs to determine whether there is a data stream with the same identifier as the data stream to be scheduled among the push and pull nodes. If there is a data stream with the same identifier as the data stream to be scheduled, the master node returns a processing failure message to the master node; if there is no data stream with the same identifier as the data stream to be scheduled, the master node will create a temporary node corresponding to the data stream to be scheduled under the push node and return a success message to the external system, instead of responding to the external system only after the slave node successfully pulls the stream, as in related technologies, or after the master node saves the request information and returns a success message to the external system before scheduling the data stream to be scheduled.

[0138] In this embodiment, if applied to a video stream scheduling and processing system, the master node, upon receiving a video stream scheduling request, reads the nodes existing under the push node but not under the pull node to form a set of video streams to be processed, and selects a machine with low load to process each video stream in the set accordingly. This achieves rapid response under high concurrency. Because the response is returned immediately after the successful establishment of a temporary node under the push node, the overall process time is shorter, enabling rapid response under high concurrency. This solves the problems of not being able to guarantee 100% pull and slow interface response speed in related technologies. It also avoids the performance degradation of stream balancing scheduling caused by adding a global lock, and the uneven data stream scheduling caused by direct scheduling without adding a global lock, thus improving the processing efficiency of the data stream.

[0139] In other embodiments of this application, when applied to online examination anti-cheating scenarios, the data stream to be processed (i.e., the video stream to be processed) can be processed by frame extraction. Then, the backend anti-cheating model is invoked to detect cheating in multiple frames, including scenarios such as whispering, using a mobile phone, talking, no one being present, and multiple people. If cheating occurs, the slave node reports the detection result to the online examination system. After receiving the reported result, the online examination system notifies the proctor on the proctor's end that a student has cheated, providing a visual prompt.

[0140] In other embodiments of this application, when this application is applied to the dance scenario of art exams, each slave node, after pulling the video stream to be processed and performing frame extraction, will call the art exam dance scoring model to divide the video according to the art exam movements such as leg control, backbend, splits, etc. After division, each movement will be scored. The slave node will report the scoring results to the online examination business system. After receiving the reported results, the online examination business system will give the student's score on the examiner's end for reference.

[0141] It should be noted that the descriptions of the same steps and contents as in other embodiments in this embodiment can be found in the descriptions in other embodiments, and will not be repeated here.

[0142] The data processing method provided in the embodiments of this application can, after the slave nodes in the target structure tree are updated, obtain the data stream to be scheduled and the target slave node to which the data stream to be scheduled needs to be scheduled based on the load of each updated slave node through the master process of the master node, and then schedule the data stream to be scheduled to the target slave node through the master process of the master node. That is, in the data stream scheduling process after the slave nodes in the target structure tree are updated, the load of each slave node is fully considered, ensuring that the load among the slave nodes after scheduling is balanced, and solving the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies.

[0143] Based on the foregoing embodiments, this application provides a data processing apparatus that can be applied to... Figure 1 , 3 In the data processing method provided in the embodiments corresponding to ~4, refer to Figure 5 As shown, the data processing device 3 may include: a calling unit 31, an acquisition unit 32, and a processing unit 33, wherein:

[0144] Calling unit 31 is used to call the main node of the target structure tree; wherein, the target structure tree includes the main node corresponding to the main process and the slave node corresponding to the slave process;

[0145] The acquisition unit 32 is used to acquire the data stream to be scheduled from the slave nodes of the target structure tree through the master process of the master node if the slave nodes of the target structure tree are updated, and to determine the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree.

[0146] The processing unit 33 is used to distribute the data stream to be scheduled to the target slave node through the master process of the master node, and to process the data stream to be scheduled through the slave process of the target slave node;

[0147] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0148] If a new slave node is added to the target structure tree, and the first target parameter is greater than a preset threshold, the master process of the master node obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream; wherein, the scheduled data stream includes the first scheduled data stream; the target slave node includes the first target slave node.

[0149] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0150] The master process of the master node determines the first slave node whose load meets the target load condition from the slave nodes in the target structure tree, and uses the target load prediction model to determine the first data stream from the first slave node;

[0151] Based on the estimated first load of each first other slave node after carrying the first data stream, a first target value is determined; wherein, the first other slave node is the slave node other than the first slave node in the target structure tree;

[0152] When the load of the slave node corresponding to the first target value is updated to the load after the first data flow, the master process of the master node determines the second slave node whose load meets the target load condition from the slave nodes in the target structure tree, and the target load prediction model is used to determine the second data flow from the second slave node.

[0153] Based on the estimated second load of each second other slave node after it carries the second data stream, the second target value is determined; wherein, the second other slave node is the slave node other than the second slave node.

[0154] If the second target value is greater than or equal to the first target value, the slave node corresponding to the first target value is determined as the first target slave node; wherein, the first data stream to be scheduled includes the first data stream.

[0155] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0156] Estimate the first load of each slave node after it has carried the first data stream;

[0157] For each estimation operation, multiple first values ​​are calculated for multiple first loads, and a first target value is determined from the multiple first values.

[0158] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0159] If the second target value is less than the first target value, when the load of the slave node corresponding to the second target value is updated to the load after the second data flow, the master process of the master node determines the third slave node whose load meets the target load condition from the slave nodes in the target structure tree, until the Nth target value is determined; wherein, the first data flow to be scheduled also includes the second data flow;

[0160] The slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value are designated as the first target slave node.

[0161] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0162] Obtain the first sample load of the first sample slave node in the sample slave node of the sample structure tree; wherein, the first sample slave node represents any sample slave node in all sample slave nodes of the sample structure tree;

[0163] Assign the sample data stream to the first sample slave node, and obtain the second sample load after the first sample slave node carries the sample data stream;

[0164] Based on the first sample load and the second sample load, determine the sample stream load of the sample data stream;

[0165] The initial load prediction model is trained based on the sample stream load to obtain the target load prediction model.

[0166] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0167] After the first data stream to be scheduled is allocated to the first target slave node through the master process of the master node, the second target parameters are determined based on the load of the slave nodes in the target structure tree.

[0168] If the second target parameter is greater than the preset threshold, the master process of the master node obtains the second scheduled data stream from the second scheduled slave node determined based on the load of the slave node, and determines the second target slave node based on the estimated load of the slave node after carrying the second scheduled data stream; wherein, the scheduled data stream also includes the second scheduled data stream; the target slave node also includes the second target slave node.

[0169] In other embodiments of this application, the acquisition unit 32 is further configured to perform the following steps:

[0170] If a third slave node to be scheduled is deleted in the target structure tree, the target slave node is determined from the other slave nodes based on the load of the slave nodes other than the third slave node to be scheduled in the target structure tree, through the master process of the master node.

[0171] It should be noted that a detailed explanation of the steps performed by each unit can be found in [reference needed]. Figure 1 , 3 The data processing methods provided in the embodiments corresponding to ~4 will not be described again here.

[0172] The data processing apparatus provided in the embodiments of this application can, after the slave nodes in the target structure tree are updated, obtain the data stream to be scheduled and the target slave node to which the data stream to be scheduled needs to be scheduled based on the load of each updated slave node through the master process of the master node, and then schedule the data stream to be scheduled to the target slave node through the master process of the master node. That is, in the data stream scheduling process after the slave nodes in the target structure tree are updated, the load of each slave node is fully considered, ensuring that the load among the slave nodes after scheduling is balanced, and solving the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies.

[0173] Based on the foregoing embodiments, embodiments of this application provide a data processing device that can be applied to... Figure 1 , 3 In the data processing method provided in the embodiments corresponding to ~4, refer to Figure 6 As shown, the data processing device 4 may include: a processor 41, a memory 42, and a communication bus 43, wherein:

[0174] Communication bus 43 is used to realize the communication connection between processor 41 and memory 42;

[0175] The processor 41 is used to execute the data processing program in the memory 42 to perform the following steps:

[0176] Call the main node of the target structure tree; where the target structure tree includes the main node corresponding to the main process and the slave nodes corresponding to the slave processes;

[0177] If the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled from the slave nodes of the target structure tree, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree.

[0178] The master process of the master node distributes the data stream to be scheduled to the target slave node, and the slave process of the target slave node processes the data stream to be scheduled.

[0179] In other embodiments of this application, the processor 41 is used to execute a data processing program in the memory 42. If the slave nodes of the target structure tree are updated, the processor 41 obtains the data stream to be scheduled from the slave nodes of the target structure tree through the master process of the master node, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree, in order to implement the following steps:

[0180] If a new slave node is added to the target structure tree, and the first target parameter is greater than a preset threshold, the master process of the master node obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream; wherein, the scheduled data stream includes the first scheduled data stream; the target slave node includes the first target slave node.

[0181] In other embodiments of this application, the processor 41 is configured to execute a data processing program in the memory 42 to obtain a data stream to be scheduled from a slave node to be scheduled based on the load of the slave node, and to determine a target slave node based on the estimated load of the slave node after it carries the data stream to be scheduled, in order to implement the following steps:

[0182] The master process of the master node determines the first slave node whose load meets the target load condition from the slave nodes in the target structure tree, and uses the target load prediction model to determine the first data stream from the first slave node;

[0183] Based on the estimated first load of each first other slave node after carrying the first data stream, a first target value is determined; wherein, the first other slave node is the slave node other than the first slave node in the target structure tree;

[0184] When the load of the slave node corresponding to the first target value is updated to the load after the first data flow, the master process of the master node determines the second slave node whose load meets the target load condition from the slave nodes in the target structure tree, and the target load prediction model is used to determine the second data flow from the second slave node.

[0185] Based on the estimated second load of each second other slave node after it carries the second data stream, the second target value is determined; wherein, the second other slave node is the slave node other than the second slave node.

[0186] If the second target value is greater than or equal to the first target value, the slave node corresponding to the first target value is determined as the first target slave node; wherein, the first data stream to be scheduled includes the first data stream.

[0187] In other embodiments of this application, processor 41 is configured to execute a data processing program in memory 42 to determine a first target value based on an estimated first load of each first other slave node after it has carried the first data stream, in order to perform the following steps:

[0188] Estimate the first load of each slave node after it has carried the first data stream;

[0189] For each estimation operation, multiple first values ​​are calculated for multiple first loads, and a first target value is determined from the multiple first values.

[0190] In other embodiments of this application, the processor 41 is further configured to execute a data processing program in the memory 42 to perform the following steps:

[0191] If the second target value is less than the first target value, when the load of the slave node corresponding to the second target value is updated to the load after the second data flow, the master process of the master node determines the third slave node whose load meets the target load condition from the slave nodes of the target structure tree, until the Nth target value is determined; wherein, the first data flow to be scheduled also includes the second data flow.

[0192] The slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value are designated as the first target slave node.

[0193] In other embodiments of this application, the processor 41 is further configured to execute a data processing program in the memory 42 to perform the following steps:

[0194] Retrieve the first sample load from the first sample node in the sample structure tree;

[0195] Distribute the sample data stream to the first sample slave node and obtain the second sample load from the first sample slave node;

[0196] Based on the first sample load and the second sample load, determine the sample stream load of the sample data stream;

[0197] The initial load prediction model is trained based on the sample stream load to obtain the target load prediction model.

[0198] In other embodiments of this application, the processor 41 is further configured to execute a data processing program in the memory 42 to perform the following steps:

[0199] After the first data stream to be scheduled is allocated to the first target slave node through the master process of the master node, the second target parameters are determined based on the load of the slave nodes in the target structure tree.

[0200] If the second target parameter is greater than the preset threshold, the master process of the master node obtains the second scheduled data stream from the second scheduled slave node determined by the load of the slave node based on the target structure tree, and determines the second target slave node based on the estimated load of the slave node after carrying the second scheduled data stream; wherein, the scheduled data stream also includes the second scheduled data stream; the target slave node also includes the second target slave node.

[0201] In other embodiments of this application, the processor 41 is further configured to execute a data processing program in the memory 42 to perform the following steps:

[0202] If a third slave node to be scheduled is deleted in the target structure tree, the target slave node is determined from the other slave nodes based on the load of the slave nodes other than the third slave node to be scheduled in the target structure tree, through the master process of the master node.

[0203] It should be noted that a detailed description of the steps performed by the processor can be found in [reference needed]. Figure 1 , 3 The data processing methods provided in the embodiments corresponding to ~4 will not be described again here.

[0204] The data processing device provided in the embodiments of this application can, after the slave nodes in the target structure tree are updated, obtain the data stream to be scheduled and the target slave node to which the data stream to be scheduled needs to be scheduled through the master process of the master node based on the load of each updated slave node, and then schedule the data stream to be scheduled to the target slave node through the master process of the master node. That is, in the data stream scheduling process after the slave nodes in the target structure tree are updated, the load of each slave node is fully considered, ensuring that the load among the slave nodes after scheduling is balanced, thus solving the problem of unbalanced load among slave nodes when scheduling data streams after adding or deleting slave nodes in related technologies.

[0205] Based on the foregoing embodiments, embodiments of this application provide a computer-readable storage medium storing one or more programs, which can be executed by one or more processors to achieve... Figure 1 , 3 The steps of the data processing method provided in the embodiments corresponding to ~4.

[0206] Based on the foregoing embodiments, embodiments of this application provide a computer program product, which includes a computer program that, when executed by a processor, implements... Figure 1 ,3 The steps of the data processing method provided in the embodiments corresponding to ~4.

[0207] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, this application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0208] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0209] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0210] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0211] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A data processing method, characterized in that, The method includes: Invoke the main node of the target structure tree; wherein, the target structure tree includes the main node corresponding to the main process and the slave nodes corresponding to the slave processes; If the slave nodes of the target structure tree are updated, the master process of the master node obtains the data stream to be scheduled from the slave nodes of the target structure tree, and determines the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree; The master process of the master node distributes the data stream to be scheduled to the target slave node, and the slave process of the target slave node processes the data stream to be scheduled.

2. The method according to claim 1, characterized in that, If a slave node in the target structure tree is updated, the master process of the master node retrieves the data stream to be scheduled from the slave nodes in the target structure tree, and determines the target slave node from the slave nodes in the target structure tree based on the load of the slave nodes in the target structure tree, including: If a new slave node is added to the target structure tree, and the first target parameter determined based on the load of the slave node is greater than a preset threshold, the master process of the master node obtains the first scheduled data stream from the first scheduled slave node determined based on the load of the slave node, and determines the first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream; wherein, the scheduled data stream includes the first scheduled data stream; and the target slave node includes the first target slave node.

3. The method according to claim 2, characterized in that, The step of obtaining a first scheduled data stream from a first scheduled slave node determined based on the load of the slave node through the master process of the master node, and determining a first target slave node based on the estimated load of the slave node after carrying the first scheduled data stream, includes: The master process of the master node determines the first slave node whose load meets the target load condition from the slave nodes of the target structure tree, and uses the target load prediction model to determine the first data stream from the first slave node; Based on the estimated first load of the slave node after each first other slave node carries the first data stream, a first target value is determined; wherein, the first other slave node is the slave node other than the first slave node in the slave node of the target structure tree; When the load of the slave node corresponding to the first target value is updated to the load after having the first data flow, the master process of the master node determines the second slave node whose load meets the target load condition from the slave nodes of the target structure tree, and uses the target load prediction model to determine the second data flow from the second slave node; A second target value is determined based on the estimated second load of the slave node after each of the second other slave nodes is updated to the second data stream; wherein, the second other slave node is the slave node other than the second slave node among the slave nodes; If the second target value is greater than or equal to the first target value, the slave node corresponding to the first target value is determined as the first target slave node; wherein, the first data stream to be scheduled includes the first data stream.

4. The method according to claim 3, characterized in that, The determination of the first target value based on the estimated first load of each other slave node after it carries the first data stream includes: Estimate the first load of each of the first other slave nodes after it has carried the first data stream; For each estimation operation, multiple first values ​​are calculated for multiple first loads, and the first target value is determined from the multiple first values.

5. The method according to claim 3 or 4, characterized in that, The method further includes: If the second target value is less than the first target value, when the load of the slave node corresponding to the second target value is updated to the load after having the second data stream, the master process of the master node determines the third slave node whose load meets the target load condition from the slave nodes of the target structure tree, until the Nth target value is determined; wherein, the first data stream to be scheduled also includes the second data stream; The slave node corresponding to the first target value, the slave node corresponding to the second target value, and the slave node corresponding to the Nth target value are designated as the first target slave node.

6. The method according to claim 3, characterized in that, The method further includes: Retrieve the first sample load from the first sample node in the sample structure tree; Assign a sample data stream to the first sample slave node, and obtain the second sample load after the first sample slave node carries the sample data stream; Based on the first sample load and the second sample load, the sample stream load of the sample data stream is determined; The initial load prediction model is trained based on the sample flow load to obtain the target load prediction model.

7. The method according to claim 2, characterized in that, The method further includes: After the first data stream to be scheduled is allocated to the first target slave node through the master process of the master node, the second target parameter is determined based on the load of the slave nodes in the target structure tree. If the second target parameter is greater than the preset threshold, the master process of the master node obtains the second scheduled data stream from the second scheduled slave node determined by the load of the slave node based on the target structure tree, and determines the second target slave node based on the estimated load of the slave node after carrying the second scheduled data stream; wherein, the scheduled data stream also includes the second scheduled data stream; the target slave node also includes the second target slave node.

8. The method according to claim 1, characterized in that, The method further includes: If a third slave node to be scheduled is deleted in the target structure tree, the target slave node is determined from the other slave nodes in the target structure tree based on their loads, through the master process of the master node.

9. A data processing apparatus, characterized in that, The device includes: The invocation unit is used to invoke the main node of the target structure tree; wherein, the target structure tree includes the main node corresponding to the main process and the slave nodes corresponding to the slave processes; The acquisition unit is configured to, if the slave nodes of the target structure tree are updated, acquire the data stream to be scheduled from the slave nodes of the target structure tree through the main process of the master node, and determine the target slave node from the slave nodes of the target structure tree based on the load of the slave nodes of the target structure tree; The processing unit is configured to allocate the data stream to be scheduled to the target slave node through the master process of the master node, and to process the data stream to be scheduled through the slave process of the target slave node.

10. A data processing device, characterized in that, The device includes: a processor, a memory, and a communication bus; The communication bus is used to realize the communication connection between the processor and the memory; The processor is used to execute a data processing program in the memory to implement the steps of the data processing method as described in any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores one or more programs, which can be executed by one or more processors to implement the steps of the data processing method as described in any one of claims 1 to 8.

12. A computer program product, the computer program product comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data processing method according to any one of claims 1-8.

Images