A parallel data processing method based on multiple devices

Abstract

The present invention relates to the field of computer technology, and in particular to a parallel data processing method based on multiple devices. The content includes: calculating the performance matrix of the computing device and the complexity matrix of the task, and combining them to generate a dynamic task mapping matrix; based on the dynamic task mapping matrix, performing dynamic task allocation, and then calculating the data dependency between the device and the task, and performing data synchronization between the devices to obtain the data synchronization rate between the devices; generating the global optimal task allocation through distributed adaptive iterative optimization. It solves the problem that the traditional method does not fully consider the differences in performance between devices, network delays and storage capabilities, resulting in waste of computing resources and low overall system efficiency; ignores the changes in task complexity over time and the impact of fluctuations in device performance, resulting in extended task processing time; fails to reasonably adjust the data synchronization rate, which easily leads to data transmission bottlenecks and network congestion, seriously affecting the parallel processing performance of the system.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a parallel data processing method based on multiple devices. Background Art

[0002] In today's technological context, with the rapid development of technologies such as big data, artificial intelligence, and the Internet of Things, the demand for data processing has exploded. A large number of computing tasks require higher processing speeds and lower latency, especially in complex parallel computing environments. How to efficiently use multiple devices and multiple equipment to work together has become a key challenge to improving the overall performance of the system. Multi-device parallel data processing has become one of the important research directions in various technical fields and is widely used in cloud computing, data centers, large-scale machine learning training, real-time data analysis and other scenarios. These scenarios often involve the processing and analysis of massive amounts of data, and the processing capacity of a single device can no longer meet such huge data needs.

[0003] In order to improve the processing power of the system, it has become a mainstream trend to use multiple devices to coordinate and process tasks in parallel. However, the complexity of the multi-device parallel processing environment has also increased accordingly. The processing power, network bandwidth, storage capacity and real-time load status of different devices are different. How to reasonably dynamically allocate tasks among multiple heterogeneous devices has become an important issue in parallel computing. In addition, in these multi-device environments, the computing tasks between different devices may have dependencies, and data transmission and synchronization between devices have also become crucial. Ensuring data consistency, timeliness and efficient transmission is the key to achieving parallel processing. At the same time, as the task processing proceeds, the load and performance of the device will change dynamically, which requires a more flexible and adaptive mechanism to adjust task allocation and load balancing to ensure that the entire system can continue to operate in an optimal state.

[0004] In summary, traditional data processing methods have the following technical problems: low efficiency in task allocation, failure to fully consider the differences in performance between devices, network delays and storage capabilities, resulting in long-term concentration of tasks on some high-performance devices, which easily causes these devices to be overloaded, resulting in waste of computing resources and low overall system efficiency; ignoring the impact of changes in task complexity over time and fluctuations in device performance, resulting in prolonged task processing time and reduced overall computing resource utilization efficiency; failing to reasonably adjust the data synchronization rate based on the bandwidth between devices, network delays and task dependencies, easily leading to data transmission bottlenecks and network congestion, seriously affecting the parallel processing performance of the system. Summary of the invention

[0005] The present invention provides a parallel data processing method based on multiple devices to solve the problems in traditional data processing methods, such as low efficiency of task allocation, failure to fully consider the differences in performance between devices, network delays and storage capabilities, resulting in long-term concentration of tasks on some high-performance devices, which easily causes these devices to be overloaded, resulting in waste of computing resources and low overall system efficiency; ignoring the changes in task complexity over time and the impact of device performance fluctuations, resulting in prolonged task processing time and reduced overall computing resource utilization efficiency; failing to reasonably adjust the data synchronization rate according to the bandwidth between devices, network delays and task dependencies, which easily leads to data transmission bottlenecks and network congestion, seriously affecting the parallel processing performance of the system.

[0006] A parallel data processing method based on multiple devices of the present invention specifically includes the following technical solutions:

[0007] A parallel data processing method based on multiple devices comprises the following steps:

[0008] S1. Calculate the performance matrix of the device and the complexity matrix of the task, combine the obtained task complexity matrix with the performance matrix of the device, and generate a dynamic task mapping matrix; perform dynamic task allocation based on the dynamic task mapping matrix;

[0009] S2. After the task allocation is completed, the data dependency between the device and the task is calculated; based on the data dependency between the device and the task, the data between the devices is synchronized to obtain the data synchronization rate between the devices; based on the dynamic task mapping matrix and the data synchronization rate between the devices, the global optimal task allocation is generated through distributed adaptive iterative optimization.

[0010] Preferably, the S1 specifically includes:

[0011] The processor performance of the device, the current load of the device, the storage capacity of the device, and the network delay parameters of the device are introduced to calculate the processing capacity of the device at different times and obtain the performance matrix of the device.

[0012] Preferably, the S1 specifically includes:

[0013] The historical execution load of the task and the dependency parameters of the task are introduced to quantify the complexity of the task and obtain the complexity matrix of the task.

[0014] Preferably, the S1 specifically includes:

[0015] Based on the complexity matrix of the task and the performance matrix of the device, a dynamic task mapping matrix is ​​generated. The specific formula is as follows:

[0016] ,

[0017] in, Indicates The tasks are The time is assigned to Dynamic task mapping matrix for each device; It is Energy consumption coefficient of each device; is Moment Performance matrix of each device; Indicates The tasks are The complexity matrix of the moment.

[0018] Preferably, the S2 specifically includes:

[0019] The data dependency between devices and tasks is calculated through the data dependency model. The specific formula is as follows:

[0020] ,

[0021] in, Indicates Tasks and Data dependencies of individual devices; Indicates The storage requirements of each task; Indicates Storage capacity of each device; Represents the data dependency adjustment parameters of devices and tasks; Indicates the data transfer speed.

[0022] Preferably, the S2 specifically includes:

[0023] Based on the data dependency between devices and tasks, the data synchronization rate between devices is obtained through the data synchronization function. The network delay of the device and the bandwidth of the device are introduced into the data synchronization function to automatically adjust the data synchronization rate between devices.

[0024] Preferably, the S2 specifically includes:

[0025] In the implementation process of distributed adaptive iterative optimization, based on the dynamic task mapping matrix and the data synchronization rate between devices, the task weights of the devices are introduced, the target loss function is calculated, and the rationality of the current task allocation is evaluated.

[0026] Preferably, the S2 specifically includes:

[0027] Through the gradient descent method, the task weight of each device is dynamically adjusted to optimize the target loss function; the next batch of tasks is assigned to different devices according to the updated task weight of each device and the dynamic task mapping matrix.

[0028] The beneficial effects of the technical solution of the present invention are:

[0029] 1. By calculating the performance matrix of the device and the complexity matrix of the task, tasks are reasonably allocated to multiple heterogeneous devices, realizing dynamic task allocation; the sine function is introduced to simulate the time periodic fluctuation of device performance; through the introduction of the dynamic task mapping matrix, tasks can be flexibly allocated according to the performance of the device and the complexity of the task, ensuring that each device can dynamically adjust the priority of processing tasks in different time periods, avoiding the problem of long-term concentration of tasks on certain devices, balancing the load of the equipment, and effectively improving the efficiency of task allocation in a multi-device environment.

[0030] 2. By calculating the actual task load and complexity of the device in detail, dynamically adjusting the device's task weight and task allocation strategy, complex tasks can be allocated to devices with stronger performance for processing, thereby maximizing the use of computing resources; in addition, the decay mechanism of task complexity over time allows some processed tasks to be gradually allocated to devices with lower performance, further optimizing the overall utilization of computing resources.

[0031] 3. By introducing the data dependency model and data synchronization rate function, the data consistency between devices can be effectively maintained; the data dependency model can dynamically identify the data transmission requirements between devices and tasks, and the data synchronization rate between devices is automatically adjusted according to the actual conditions such as the device bandwidth and network delay, avoiding network congestion and data conflicts; at the same time, through the effective control of network delays between devices, efficient data synchronization is maintained under high data dependency conditions, ensuring timely and accurate data sharing between parallel tasks.

[0032] 4. A distributed adaptive iterative optimization method is proposed. The rationality of the current task allocation is evaluated by calculating the target loss function. The task weight of the device is dynamically adjusted through the gradient descent method. The optimization is performed according to the actual task load of the device, the data synchronization rate, and the load adjustment rate of the device. This ensures that each device is load-balanced when processing tasks. The task allocation is continuously iterated and optimized according to the actual operating conditions to ensure that the task load can be adaptively adjusted when the load is uneven or the device status changes. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 The present invention is a flowchart of a parallel data processing method based on multiple devices. DETAILED DESCRIPTION

[0034] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the technical scheme in the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiment of the present invention. Obviously, the described embodiment is only a part of the embodiment of the present invention, not all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0035] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0036] A specific scheme of a parallel data processing method based on multiple devices provided by the present invention is described in detail below with reference to the accompanying drawings.

[0037] See attached Figure 1 , which shows a flow chart of a parallel data processing method based on multiple devices provided by an embodiment of the present invention, the method comprising the following steps:

[0038] S1. Calculate the performance matrix of the device and the complexity matrix of the task, combine the obtained task complexity matrix with the performance matrix of the device, and generate a dynamic task mapping matrix; perform dynamic task allocation based on the dynamic task mapping matrix;

[0039] In the parallel data processing solution based on multiple devices, the core lies in effectively allocating tasks, synchronizing data, and optimizing the use of computing resources. In order to ensure the parallel data processing in multiple heterogeneous devices (such as multi-core processors, GPUs, etc.), the present invention provides algorithm steps to adaptively adjust the load distribution, data synchronization and dynamic optimization between devices to ensure that the parallel computing process is efficient, stable and fault-tolerant.

[0040] The entire data processing process starts with the allocation of tasks. In a multi-device environment, task allocation is crucial because different devices have different processing performance, network bandwidth, and storage capacity. In order to reasonably allocate tasks to each device, the performance matrix of the device is first calculated, that is, the processing capacity of the device at different times, introducing parameters such as the device's processor performance, the device's current load, the device's storage capacity, and the device's network delay. The performance matrix calculation formula is as follows:

[0041] ,

[0042] in, is Moment The performance matrix of the device is Devices in Comprehensive performance at all times; It is Processor performance of each device; It is The current load of each device; It is Storage capacity of each device; , are weight coefficients, which are used to adjust the weights of the device's processor performance and storage capacity respectively; It is the frequency parameter of time, which determines the period of fluctuation of equipment performance; is a time variable; It is Network latency for each device; is an exponential decay parameter used to control the impact of the current load of the device on the performance of the device. In actual data processing, the performance of the device is not static, and the current load of the device will fluctuate with the changes in the processing tasks. The performance matrix calculation formula reflects the current load of the device in an exponential form, ensuring that as the load increases, the performance of the device is dynamically reduced to avoid overloaded devices from continuing to accept tasks; on the other hand, by introducing the sine function , so that the performance of the device changes periodically over time, simulating the fluctuations in actual task execution.

[0043] Not only do we need to know the performance of each device, we also need to quantify the complexity of the current task; calculate the complexity matrix of the task to ensure that the task can be matched with the appropriate device, and define the degree of correlation between the complexity of the task and its historical execution load and the dependencies within the task:

[0044] ,

[0045] in, Indicates The tasks are The complexity matrix of the moment; Represents the task complexity adjustment coefficient, which is used to adjust the impact of task complexity; Indicates The historical execution load of tasks, The load accumulation of the task in the past is The computing resources consumed during the historical execution of each task are recorded and counted; Indicates The dependency coefficient of each task indicates the degree of dependency between tasks. The greater the dependency, the higher the task complexity. Represents the time decay parameter. The calculation of task complexity ensures that complex tasks can be assigned to more powerful devices by considering the historical execution load of the task; at the same time, the time decay parameter is introduced , indicating that as time goes by, the complexity of some tasks may decrease (such as tasks that have been partially processed), and devices with stronger performance should give priority to processing tasks with higher complexity, thereby maximizing the use of computing resources.

[0046] The calculated task complexity matrix is ​​combined with the device performance matrix to generate a dynamic task mapping matrix. The dynamic task mapping matrix determines how tasks are allocated to each device. The formula is as follows:

[0047] ,

[0048] in, is the mapping task amount, indicating the The tasks are The time is assigned to Dynamic task mapping matrix for each device; It is The energy consumption coefficient of each device is used to adjust the balance between the energy consumption of the device and the task allocation. The performance matrix of the device and the complexity matrix of the task directly determine the priority of task allocation. The higher the complexity of the task, the larger the value of the complexity matrix, and the higher the performance matrix value of the allocated device. By adding Periodic function, the priority of task allocation changes over time, avoiding long-term overload of some devices. Dynamic task allocation is achieved through dynamic task mapping matrix, ensuring that tasks are reasonably allocated to devices and optimizing the overall parallel processing efficiency.

[0049] S2. After the task allocation is completed, the data dependency between the device and the task is calculated; based on the data dependency between the device and the task, the data between the devices is synchronized to obtain the data synchronization rate between the devices; based on the dynamic task mapping matrix and the data synchronization rate between the devices, the global optimal task allocation is generated through distributed adaptive iterative optimization.

[0050] After the task allocation is completed, data synchronization between devices is crucial. In the scenario of parallel processing of multiple devices, data consistency is an important factor to ensure the stability of the entire processing process. In order to effectively synchronize data between different devices, the dependency between devices and tasks is calculated through the data dependency model. The dependency reflects whether the device depends on the intermediate results of other devices when processing a task:

[0051] ,

[0052] in, Indicates Tasks and Data dependencies of individual devices; Indicates The storage requirements of each task; Indicates Storage capacity of each device; A regulation parameter representing the data dependency between the device and the task, used to control the rate of change of the data dependency between the device and the task; Indicates the data transmission speed, which is used to control the transmission rate between devices and tasks. If the difference between the storage capacity of a device and the storage requirement of a task is greater than the preset threshold, the device cannot efficiently process the task; the threshold is set according to the specific situation and is not limited here. By calculating the data dependency between devices and tasks, identify which devices need to synchronize data with other devices more frequently. The stronger the data synchronization requirement, the more frequent the data communication between devices.

[0053] Furthermore, the data synchronization rate between devices is adjusted through the data synchronization function to ensure data consistency. The formula of the data synchronization function is as follows:

[0054] ,

[0055] in, Indicates Devices in The data synchronization rate at the moment; is the data synchronization coefficient, which is used to control the overall priority of data synchronization; Indicates The bandwidth of each device. The higher the bandwidth between devices, the faster the data transmission; the stronger the data dependency between devices and tasks, the higher the data synchronization requirement between devices. The square term of network delay is added to the formula to ensure that when the network delay exceeds the preset delay limit, the synchronization rate is automatically reduced to prevent network congestion and ensure timely and conflict-free data sharing between parallel processing tasks; the delay limit is set according to the specific situation and is not limited here.

[0056] After task allocation and data synchronization, it is necessary to continuously optimize and adjust tasks. As tasks progress, the load of the device and the priority of the tasks will change dynamically; therefore, distributed adaptive iterative optimization is used to generate the global optimal task allocation. An objective loss function is defined to measure the effectiveness of the current task allocation:

[0057] ,

[0058] in, represents the target loss function, which indicates the effectiveness of task allocation. The smaller the loss, the more reasonable the task allocation. Indicates the total number of tasks; Indicates the total number of devices; is the actual task load of the device, indicating Devices in Time processing The amount of work for each task; It is The task weight of a device indicates The task load weight that each device can bear; Indicates The load adjustment rate of the device is used to control the The task weight of each device is adjusted over time. By calculating the actual task load of the device and mapping task volume The difference between the target loss function and the task allocation is used to measure whether the current task allocation is reasonable. The smaller the target loss function is, the more reasonable the task allocation is and the more uniform the load distribution of the equipment is.

[0059] The objective loss function is optimized by the gradient descent method, and the task weight of each device is dynamically adjusted. The update formula is:

[0060] ,

[0061] in, represents the updated task weight; represents the task weight before updating; is the learning rate, which is used to control the step size of the update; Represents the weight coefficient, which is used to adjust the influence of the data synchronization rate on the task weight update of each device. Through multiple rounds of iterations, the task allocation is continuously optimized so that the target loss function gradually converges to the minimum value, thereby achieving the global optimal task allocation state. The gradient descent method plays a dynamic adjustment role here, and can adaptively adjust the task load according to the actual task load of the device, so that the computing resources of all devices can be fully utilized.

[0062] The next batch of tasks is assigned to different devices based on the updated task weights of each device and the dynamic task mapping matrix, ensuring that tasks are reasonably allocated based on the latest calculation and data synchronization results.

[0063] In summary, a parallel data processing method based on multiple devices is completed.

[0064] The order of the embodiments of the invention is for description only and does not represent the advantages and disadvantages of the embodiments. The processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0065] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referenced to each other, and each embodiment focuses on the differences from other embodiments.

[0066] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that the technical solutions described in the above embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention.

Claims

1. A parallel data processing method based on multiple devices, characterized in that: The following steps are involved: S1. Calculate the performance matrix of the device and the complexity matrix of the task, combine the obtained task complexity matrix with the performance matrix of the device, and generate a dynamic task mapping matrix. The specific formula is as follows: , in, Indicates The tasks are The time is assigned to Dynamic task mapping matrix for each device; It is Energy consumption coefficient of each device; is Moment Performance matrix of each device; Indicates The tasks are The complexity matrix of the moment; It is a periodic function, indicating that the priority of task allocation changes over time; Dynamic task allocation based on dynamic task mapping matrix; S2. After the task allocation is completed, the data dependency between the device and the task is calculated; based on the data dependency between the device and the task, the data between the devices is synchronized to obtain the data synchronization rate between the devices; based on the dynamic task mapping matrix and the data synchronization rate between the devices, the target loss function is defined through distributed adaptive iterative optimization to evaluate the rationality of the current task allocation; the definition formula of the target loss function is as follows: , in, represents the target loss function; Indicates the total number of tasks; Indicates the total number of devices; is the actual task load of the device, indicating Devices in Time processing The amount of work for each task; It is The task weight of each device; Indicates Load adjustment rate of each device; Indicates Devices in The data synchronization rate at the moment; Through the gradient descent method, the task weight of each device is dynamically adjusted to optimize the target loss function; the specific formula is: , in, represents the updated task weight; represents the task weight before updating; is the learning rate; represents the weight coefficient; The next batch of tasks is assigned to different devices according to the updated task weights of each device and the dynamic task mapping matrix, and finally the global optimal task allocation is generated.

2. A method for parallel data processing based on multiple devices according to claim 1, characterized in that: The S1 specifically includes: The processor performance of the device, the current load of the device, the storage capacity of the device, and the network delay parameters of the device are introduced to calculate the processing capacity of the device at different times and obtain the performance matrix of the device.

3. A method for parallel data processing based on multiple devices according to claim 1, characterized in that: In the S1, it also includes: The historical execution load of the task and the dependency parameters of the task are introduced to quantify the complexity of the task and obtain the complexity matrix of the task.

4. A method for parallel data processing based on multiple devices according to claim 2, characterized in that: The S2 specifically includes: The data dependency between devices and tasks is calculated through the data dependency model. The specific formula is as follows: , in, Indicates Tasks and Data dependencies of individual devices; Indicates The storage requirements of each task; Indicates Storage capacity of each device; Tuning parameters representing data dependencies between devices and tasks; Indicates the data transfer speed.

5. A method for parallel data processing based on multiple devices according to claim 4, characterized in that: In the S2, it also includes: Based on the data dependency between devices and tasks, the data synchronization rate between devices is obtained through the data synchronization function. The network delay of the device and the bandwidth of the device are introduced into the data synchronization function to automatically adjust the data synchronization rate between devices.

Images