Data processing method, device, apparatus, and computer storage medium

Abstract

The embodiment of the present application relates to the technical field of computer data processing, and discloses a data processing method, which comprises the following steps: obtaining a computing environment parameter; the computing environment parameter comprises original computing data; determining the number of random data, the total number of target computing devices and the amount of to-be-allocated encoding tasks corresponding to each target computing device according to the computing environment parameter; performing calculation on the original computing data according to the number of random data, the total number of devices and the amount of to-be-allocated encoding tasks, so as to obtain device encoding data corresponding to each target computing device; performing calculation on the device encoding data and user input data through each target computing device, so as to obtain a plurality of intermediate results; and determining processed data corresponding to the user input data according to all the intermediate results. In the above manner, the embodiment of the present application improves the efficiency of data processing in an edge computing environment.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of computer data processing, and in particular to a data processing method, device, equipment and computer storage medium. BACKGROUND

[0002] In the existing data processing method based on edge computing environment, task allocation is usually to evenly distribute computing tasks to each edge computing device.

[0003] The inventor found in the implementation of the present application that the existing data processing method based on edge computing environment has the problem of low efficiency. SUMMARY

[0004] In view of the above problems, embodiments of the present application provide a data processing method, device, equipment and computer storage medium to solve the problem of low efficiency of data processing in the prior art.

[0005] According to an aspect of an embodiment of the present application, a data processing method is provided, which comprises:

[0006] obtaining a computing environment parameter; the computing environment parameter comprises original computing data;

[0007] determining a random data quantity, a total number of target computing devices and a quantity of to-be-allocated encoding tasks corresponding to each target computing device according to the computing environment parameter;

[0008] computing the original computing data according to the random data quantity, the total number of devices and the quantity of to-be-allocated encoding tasks to obtain device encoding data corresponding to each target computing device;

[0009] computing a plurality of intermediate results by each target computing device according to user input data and the device encoding data;

[0010] determining processed data corresponding to the user input data according to all the intermediate results.

[0011] In an optional manner, the computing environment parameter comprises the quantity of the original computing data, the size of the computing task, the quantity of optional computing devices, the quantity of resource limits of each optional computing device, and the unit cost of each optional computing device; and the target computing device is at least one of the optional computing devices.

[0012] In an optional manner, the method further comprises:

[0013] determining {1, …, (W) max} and Is the intersection of the sets empty? Where {1, ..., Wmax} is the set of all natural numbers in [1, Wmax]; W is the set of all resource-limited numbers of the available computing devices; (W) max The largest element in W; r∈{1,…,(W) max};w j The resource limit number of optional computing devices j; k is the number of optional computing devices; m is the size of the computing task; r is the amount of random data;

[0014] If the intersection is non-empty, then the value of r is in {1, ..., (W)}. max The process involves iterating through the data to determine the values ​​corresponding to each r value.

[0015] According to the respective r values From {1, ..., (W)} max Determine the minimum value of r in} min The maximum value of r max ;

[0016] With r min ≤r≤r max As a constraint, the total cost of resource consumption, c sum (r) is the optimization objective. The optimal solution for r is obtained, and this optimal solution is used to determine the number of random data points. θ r c is the number of the target computing devices; j Let c be the unit cost of the optional computing device j, and satisfy c. j ≥c j-1 ;

[0017] The total number of target computing devices and the amount of coding tasks to be assigned are determined based on the amount of random data.

[0018] In an alternative approach, r is a natural number; the method further includes: sequentially calculating the value of r increasing from 1 to (W). max Time corresponding The first one will be satisfied. The value of r is determined as r min ;

[0019] Calculate the value of r sequentially from (W) max When it decreases to 1, the corresponding The first one will be satisfied. The value of r is determined as r max .

[0020] In an alternative approach, the method further includes:

[0021] the number of target computing devices θ r is determined as

[0022] the to-be-allocated encoding task amount V(B j ) corresponding to the target computing device j is determined as:

[0023]

[0024] In an optional manner, the method further comprises:

[0025] generating random data; the number of the random data is the random data number;

[0026] determining an encoding coefficient matrix corresponding to each of the target computing devices according to the total number of devices and the to-be-allocated encoding task amount;

[0027] encoding the random data and the original computing data according to the encoding coefficient matrix to obtain the device encoding data corresponding to each of the target computing devices.

[0028] In an optional manner, the method further comprises:

[0029] constructing an (m+r) x (m+r) dimensional encoding coefficient matrix B as follows:

[0030]

[0031] the encoding coefficient matrix B j of the target computing device j is determined as:

[0032]

[0033] wherein, E t is a t x t dimensional unit matrix; O r，m is a r x m dimensional zero matrix; is a matrix composed of row vectors with indices from a to b in the matrix; r is the random data number; m is the computing task scale; z ∈ {1,..., θ r -1}; V(B z ) = r; V(B j ) is the to-be-allocated encoding task amount corresponding to the target computing device j; θ r is the number of target computing devices.

[0034] According to another aspect of the embodiments of the present application, a data processing apparatus is provided, comprising:

[0035] obtaining a computing environment parameter; the computing environment parameter comprises original computing data;

[0036] determining a random data quantity, a total number of target computing devices and a to-be-allocated encoding task quantity corresponding to each target computing device according to the computing environment parameter;

[0037] computing the original computing data according to the random data quantity, the total number of target computing devices and the to-be-allocated encoding task quantity to obtain device encoding data corresponding to each target computing device;

[0038] computing a plurality of intermediate results according to user input data and the device encoding data by each target computing device;

[0039] determining processed data corresponding to the user input data according to all the intermediate results.

[0040] According to another aspect of the embodiment of the present application, a data processing device is provided, comprising a processor, a memory, a communication interface and a communication bus, the processor, the memory and the communication interface complete communication with each other through the communication bus; the memory is used to store at least one executable instruction, and the executable instruction makes the processor execute the operation of the data processing method.

[0041] According to still another aspect of the embodiment of the present application, a computer readable storage medium is provided, and the storage medium stores at least one executable instruction, and the executable instruction makes the data processing device execute the operation of the data processing method.

[0042] The embodiment of the present application obtains a computing environment parameter; wherein the computing environment parameter comprises original computing data; a random data quantity, a total number of target computing devices and a to-be-allocated encoding task quantity corresponding to each target computing device are determined according to the computing environment parameter; the original computing data is computed according to the random data quantity, the total number of target computing devices and the to-be-allocated encoding task quantity to obtain device encoding data corresponding to each target computing device; a plurality of intermediate results are obtained according to user input data and the device encoding data by each target computing device; and processed data corresponding to the user input data is determined according to all the intermediate results.

[0043] Different from the prior art that evenly distributes the computing task among all target computing devices, the present embodiment does not consider that the target computing devices are heterogeneous and have inconsistent processing capabilities, thus the problem of large total cost of data processing exists in the even distribution, the present embodiment determines the optimal random data quantity, the total number of target computing devices and the respective target computing device corresponding to the to-be-distributed encoding task quantity according to the computing environment parameters, so as to adaptively distribute the computing task among the target computing devices according to the performance of each computing device, so that the total cost is minimized, thereby overcoming the problem of large total cost of data processing based on edge computing in the prior art.

[0044] The above description is only a summary of the technical solutions of the embodiments of the present application, in order to more clearly understand the technical means of the embodiments of the present application, the contents of the specification can be implemented, and in order to make the above and other purposes, features and advantages of the embodiments of the present application more obvious and easy to understand, the specific embodiments of the present application are described below. BRIEF DESCRIPTION OF DRAWINGS

[0045] The accompanying drawings are included to provide a further understanding of the application and are incorporated herein and constitute a part of the detailed description. In the drawings:

[0046] Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown;

[0047] Figure 2 A structural schematic diagram of a data processing device provided by an embodiment of the present application is shown;

[0048] Figure 3 A structural schematic diagram of a data processing device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0049] Exemplary embodiments of the present application will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments described herein.

[0050] Figure 1 A flowchart of a data processing method provided by an embodiment of the present application is shown, which is executed by a computer processing device. The computer processing device can include a mobile phone, a notebook computer, etc. As shown in the figure, Figure 1 The method includes the following steps:

[0051] Step 10: Obtain the computing environment parameters; the computing environment parameters include the original computing data.

[0052] In one embodiment of the present application, the computing environment parameters can be obtained by the center server, and the original computing data is the data required to complete the calculation of the user input data. The computing environment parameters include the number of original computing data, the size of the computing task, the number of optional computing devices, the number of resource limits of each optional computing device, and the unit cost of each optional computing device; and the target computing device is at least one of the optional computing devices.

[0053] Wherein, the optional computing device can be different types of edge computing terminals such as mobile phones, computers, and routers, and each optional computing device is connected with the center server. The number of resource limits refers to the maximum number of resources that each optional computing device can use for calculation, and the unit cost refers to the device resources consumed by each optional computing device to complete one unit of computing task.

[0054] Step 20: determining the number of random data, the total number of target computing devices, and the amount of to-be-assigned encoding tasks corresponding to each target computing device according to the computing environment parameters.

[0055] In one embodiment of the present application, on the one hand, in order to enable each target computing device to complete the corresponding amount of to-be-assigned encoding tasks, the amount of to-be-assigned encoding tasks should not exceed the resource limit data of the corresponding target computing device. On the other hand, in order to improve the security of data processing, considering that the amount of random data assigned to each target computing device should not exceed the total amount of random data when ensuring the information theory security requirement, the optimal total amount of random data can be determined.

[0056] Under the premise of ensuring data security and being able to complete the computing task, in order to reduce the total cost of completing the computing task, after determining the number of random data, the computing task can be assigned in combination with the resource limit data and the unit cost of each target computing device, so as to minimize the total cost of all target computing devices.

[0057] In one embodiment of the present application, it is first needed to determine whether data security and the ability to complete the computing task can be ensured, that is, whether there is a solution for the number of random data according to the number of resource limits of all target computing devices and the size of the computing task. And in the case of having a solution, the value interval of the optimal solution of the number of random data is determined according to the number of resource limits of all target computing devices and the size of the computing task, and then the value of the number of random data is traversed in the value interval, the corresponding total cost under each value is calculated, and the minimum total cost is taken as the optimization goal to obtain the optimal solution of the number of random data.

[0058] After obtaining the optimal solution of the number of random data, the minimum number of devices that can meet the sum of the optimal number of random data and the size of the computing task is determined as the number of target computing devices (denoted as θr Then, the first θ is determined based on the minimum value between the resource constraint data and the amount of random data for each target computing device. r The amount of coding tasks to be allocated to each of the -1 target computing devices is then determined, and the remaining computing tasks are allocated to the θ-th device. r One target computing device.

[0059] Therefore, in another embodiment of the present invention, step 20 further includes:

[0060] Step 201: Determine {1, ..., (W)} max}and Is the intersection of {1, ..., (W) empty? max} is [1, (W)] max W is the set of all natural numbers in the set; W is the set of resource limits for all the said optional computing devices; (W) max The largest element in W; r∈{1,…,(W) max};w j The resource limit number of optional computing devices j; k is the number of optional computing devices; m is the size of the computing task; and r is the amount of random data.

[0061] In one embodiment of the present invention, when the quantity of random data is r, the computing device s, while ensuring the information theory security of the computational data, j The encoded block V(B) stored on the upper j ) does not exceed r, that is, V(B) j V(B) ≤ r. Due to resource limitations of edge devices, V(B) ≤ r. j )≤W j We obtain V(B) j )≤min(w j Therefore, let The range of the number r of random data is:

[0062] In yet another embodiment of the invention, when {1, ..., (W)} max}and When the intersection of the two sets is empty, i.e., r is calculated sequentially from 1 to (W). max hour The value when When r has no solution, task allocation ends.

[0063] Step 202: If the intersection is non-empty, set the value of r to {1, ..., (W)}. max The process involves iterating through the data to determine the values ​​corresponding to each r value.

[0064] In one embodiment of the present application, if the intersection is not empty, it indicates that there exists r solution, therefore, the value of r is calculated from 1 to (W) max in turn.

[0065] Step 203: determining the minimum value r and the maximum value r max of r from {1, …, (W) min} respectively according to the corresponding max .

[0066] In one embodiment of the present application, considering that the elements in {1, …, (W) max} are increasing, therefore, when determining the value range of the optimal solution of r, the value can be traversed from 1 to (W) max in turn, and the first value that satisfies is determined as the minimum value r min of r, and correspondingly, the value is traversed from (W) max to 1 in turn, and the first value that satisfies is determined as the maximum value r max of r, thereby obtaining the value range of the optimal solution of r [r min , r max ].

[0067] Therefore, in another embodiment of the present application, r is a natural number, and step 203 further includes:

[0068] Step 2031: calculating the value of r from 1 to (W) max in turn, and determining the first value that satisfies as the value r . min

[0069] Step 2032: calculating the value of r from (W) max to 1 in turn, and determining the first value that satisfies as the value r . max

[0070] Step 204: taking r min ≤ r ≤ r max as a constraint condition, taking the minimum total cost c sum (r) as an optimization target, obtaining the optimal solution of r, and determining the optimal solution as the random data quantity; wherein, θ r is the number of target computing devices; c j ​​​Let c be the unit cost of the optional computing device j, and satisfy c. j ≥c j-1 .

[0071] In one embodiment of the present invention, r min ≤r≤r max That is, the total cost of resource consumption, c. sum When minimizing (r) is the optimization objective, the optimal solution for r (denoted as r) * The range of values ​​for ).

[0072] In one embodiment of the present invention, let c = INT_MAX, where INT_MAX is a sufficiently large value, and let r * In [r min r max Iterate through the data and calculate c separately. sum (r * When c sum (r * When ) < c, record the value of c at this time. sum (r * ) and r * c sum (r * ) = c, r = r * The traversal is complete, and the recorded value of r is now the optimal solution.

[0073] Step 205: Determine the total number of target computing devices and the amount of coding tasks to be assigned based on the amount of random data.

[0074] In one embodiment of the present invention, after determining the optimal solution for the number of random data points, the minimum number of devices that can satisfy the sum of the optimal number of random data points and the computational task size is determined as the target number of computing devices (denoted as θ). r Then, the first θ is determined based on the minimum value between the resource constraint data and the amount of random data for each target computing device. r The amount of coding tasks to be allocated to each of the -1 target computing devices is then determined, and the remaining computing tasks are allocated to the θ-th device. r One target computing device.

[0075] Therefore, in another embodiment of the present invention, step 205 further includes:

[0076] Step 2051: Calculate the number θ of the target computing devices. r Determined as

[0077] Step 2052: Assign the amount of coding task V(B) to be allocated to the target computing device j. j It is determined to be:

[0078]

[0079] In one embodiment of the present invention, k > θ r The amount of coding tasks to be assigned to the optional computing device k is determined to be 0, that is, no computing tasks are assigned to optional computing devices other than the target computing device.

[0080] Step 30: Calculate the original computing data based on the amount of random data, the total number of devices, and the amount of coding tasks to be assigned, to obtain the device coding data corresponding to each of the target computing devices.

[0081] In one embodiment of the present invention, a corresponding number of random data blocks are generated based on the amount of random data. An encoding coefficient matrix is ​​designed for each target computing device based on the total number of devices and the amount of encoding tasks to be assigned. Each target computing device encodes the random data blocks and the original computational data according to the encoding coefficient matrix, thereby obtaining device-encoded data corresponding to each target computing device.

[0082] In yet another embodiment of the present invention, step 30 further includes:

[0083] Step 301: Generate random data; the number of random data is the number of random data items.

[0084] In one embodiment of the present invention, r random data blocks are generated to obtain a random data set R = {R1, ..., R2}. r}, where r is the number of random data points.

[0085] Step 302: Determine the coding coefficient matrix corresponding to each of the target computing devices based on the total number of devices and the amount of coding tasks to be allocated.

[0086] In one embodiment of the present invention, it can be determined from the aforementioned task allocation algorithm that there exists an edge device S. z z∈{1,...,θ r -1},V(B Z Since ) = r, the target computing devices with the amount of coding tasks to be allocated as the random data quantity are used as the separators. The coding coefficient matrix corresponding to each target computing device is designed to achieve the optimal allocation amount of coding tasks to be allocated to each target computing device as determined by the aforementioned task allocation steps, and to make the decoding complexity of the coding coefficient matrix low.

[0087] Therefore, in another embodiment of the present invention, step 302 further includes:

[0088] Step 3021: Construct an encoding coefficient matrix B of (m+r) x (m+r) dimensions as follows:

[0089]

[0090] In an embodiment of the present application, E t is a unit matrix of t x t dimensions; O r，m is a zero matrix of r x m dimensions; is a matrix composed of row vectors with indices from a to b in the matrix; r is the number of random data; m is the scale of a computing task; z e {1,..., θ r -1}; V(B z ) = r; V(B j ) is the amount of encoding tasks corresponding to the target computing device j; θ r is the number of target computing devices.

[0091] Step 3022: Encode the encoding coefficient matrix B j of the target computing device j as follows:

[0092]

[0093] where θ r is the number of target computing devices.

[0094] Step 303: Encode the random data and the original computing data according to the encoding coefficient matrix to obtain the device encoding data corresponding to each target computing device respectively.

[0095] In an embodiment of the present application, let where A i is the i-th original data block, A1-A m composes the original computing data. Linearly encode T to generate the total encoding data BT.

[0096] Encode T according to the encoding coefficient matrix B j corresponding to each target computing device respectively to obtain the encoding data B j T, and distribute B j T to the corresponding target computing device S j , wherein the optional computing devices {S θ+1 ,..., S k} do not store the encoding coefficient matrix.

[0097] Step 40: Perform computing by each target computing device according to the user input data and the device encoding data to obtain a plurality of intermediate results.

[0098] In one embodiment of the present application, the user input data x is sent to each target computing device S by a user device S0 j After that, S j The encoded data B j T is multiplied by x to obtain an intermediate result B j Tx, and then each S j The corresponding B j Tx is returned to S0.

[0099] Step 50: determining the processed data corresponding to the user input data according to all the intermediate results.

[0100] In one embodiment of the present application, all the intermediate results are decoded to obtain the processed data Ax according to the following formula:

[0101]

[0102] In one embodiment of the present application, Ax can be calculated by the user device S0 that sends the user input data, and the user terminal device only needs to perform m times of subtraction operation on the received m+r intermediate results to obtain the required result Ax, and the calculation complexity is much lower than that of calculating the matrix multiplication on the mxl-dimensional data matrix A and the lxl-dimensional input vector x, thereby reducing the decoding complexity and improving the data processing efficiency.

[0103] The data processing method provided in the embodiments of the present application obtains the computing environment parameters, wherein the computing environment parameters include original computing data; determines the number of random data, the total number of target computing devices and the amount of encoding tasks to be allocated to each target computing device according to the computing environment parameters; performs calculation on the original computing data according to the number of random data, the total number of devices and the amount of encoding tasks to be allocated, to obtain device encoding data corresponding to each target computing device; performs calculation on the user input data and the device encoding data by each target computing device to obtain a plurality of intermediate results; and determines the processed data corresponding to the user input data according to all the intermediate results.

[0104] Unlike the prior art that evenly allocates the computing tasks among all target computing devices without considering that the target computing devices have inconsistent processing capabilities in a heterogeneous case, thereby causing the problem of large total cost of data processing in the case of even allocation, the data processing method provided in the embodiments of the present application determines the optimal number of random data, the total number of target computing devices and the amount of encoding tasks to be allocated to each target computing device according to the computing environment parameters, so that the computing tasks can be adaptively allocated among the target computing devices according to the performance of each computing device, so that the total cost is minimized, thereby overcoming the problem of large total cost of data processing based on edge computing in the prior art.

[0105] Figure 2 A schematic diagram of the structure of a data processing apparatus provided in an embodiment of the present invention is shown. Figure 2 As shown, the device 60 includes: an acquisition module 601, an allocation module 602, an encoding module 603, a calculation module 604, and a determination module 605.

[0106] The acquisition module 601 is used to acquire computing environment parameters, including raw computing data.

[0107] The allocation module 602 is used to determine the amount of random data, the total number of target computing devices, and the amount of coding tasks to be allocated for each target computing device according to the computing environment parameters.

[0108] The encoding module 603 is used to calculate the original computing data based on the amount of random data, the total number of devices, and the amount of encoding tasks to be assigned, so as to obtain the device encoding data corresponding to each of the target computing devices.

[0109] The calculation module 604 is used to perform calculations on each of the target computing devices based on user input data and device encoded data to obtain multiple intermediate results;

[0110] The determining module 605 is used to determine the processed data corresponding to the user input data based on all the intermediate results.

[0111] In one alternative approach, the computing environment parameters include the amount of raw computing data, the scale of the computing task, the number of optional computing devices, the resource limits of each optional computing device, and the unit cost of each optional computing device; the target computing device is at least one of the optional computing devices.

[0112] In an alternative embodiment, the allocation module 602 is also used for:

[0113] Determine {1, ..., (W)} max}and Is the intersection of {1, ..., (W) empty? max} is [1, (W)] max W is the set of all natural numbers in the set; W is the set of resource limits for all the said optional computing devices; (W) max The largest element in W; r∈{1,…,(W) max};w j The resource limit number of optional computing devices j; k is the number of optional computing devices; m is the size of the computing task; r is the amount of random data;

[0114] If the intersection is non-empty, the value of r is traversed in {1, …, (W) max} to determine the respective

[0115] According to the respective , the minimum value r max and the maximum value r min of r are determined from {1, …, (W) max ;

[0116] The optimal solution of r is obtained as a constraint condition of r min ≤ r ≤ r max , and the minimum total cost c sum (r) is the optimization objective, and the optimal solution is determined as the random data quantity; wherein, θ r is the number of target computing devices; c j is the unit cost of the optional computing device j, and satisfies c j ≥ c j-1 ;

[0117] According to the random data quantity, the total number of devices of the target computing device and the amount of to-be-assigned encoding tasks are determined.

[0118] In an optional manner, r is a natural number; the allocation module 602 is further configured to:

[0119] The value of r is sequentially calculated from 1 to (W) max , and the corresponding is calculated. The first r value that satisfies min is determined as r ;

[0120] The value of r is sequentially calculated from (W) max to 1, and the corresponding is calculated. The first r value that satisfies max is determined as r .

[0121] In an optional manner, the allocation module 602 is further configured to:

[0122] The number of target computing devices θ r is determined as

[0123] The amount of to-be-assigned encoding tasks V(B j ) corresponding to the target computing device j is determined as:

[0124]

[0125] In an optional mode, the encoding module 603 is further configured to:

[0126] generate random data; the number of the random data is the random data number;

[0127] determine an encoding coefficient matrix corresponding to each of the target computing devices according to the total number of the devices and the to-be-allocated encoding task amount;

[0128] encode the random data and the original computing data according to the encoding coefficient matrix, to obtain the device encoding data corresponding to each of the target computing devices.

[0129] In an optional mode, the encoding module 603 is further configured to:

[0130] construct an (m+r) x (m+r) dimensional encoding coefficient matrix B as follows:

[0131]

[0132] wherein, E t is a t x t dimensional unit matrix; O r，m is a r x m dimensional zero matrix; is a matrix composed of row vectors with indices from a to b in the matrix; r is the random data number; m is the computing task scale; z ∈ {1,..., θ r -1}, V(B z ) = r; V(B j ) is the to-be-allocated encoding task amount corresponding to the target computing device j; θ r is the number of the target computing devices.

[0133] the encoding coefficient matrix B j of the target computing device j is determined as:

[0134]

[0135] wherein, θ r is the number of the target computing devices.

[0136] The data processing apparatus provided by the embodiment of the present application obtains a computing environment parameter, wherein the computing environment parameter comprises original computing data; determines a random data quantity, a total number of target computing devices and a to-be-allocated encoding task quantity corresponding to each target computing device according to the computing environment parameter; performs computation on the original computing data according to the random data quantity, the total number of target computing devices and the to-be-allocated encoding task quantity, to obtain device encoding data corresponding to each target computing device; performs computation on the user input data and the device encoding data according to each target computing device, to obtain a plurality of intermediate results; and determines processed data corresponding to the user input data according to all the intermediate results.

[0137] Unlike the prior art, which evenly allocates computing tasks among all target computing devices without considering that the target computing devices are heterogeneous and have inconsistent processing capabilities, thus resulting in a large total cost of data processing, the data processing apparatus provided by the embodiment of the present application determines the optimal random data quantity, the total number of target computing devices and the to-be-allocated encoding task quantity corresponding to each target computing device according to the computing environment parameter, so that the computing tasks can be adaptively allocated among the target computing devices according to the performance of each computing device, so as to minimize the total cost, thereby overcoming the problem of a large total cost of data processing based on edge computing in the prior art.

[0138] Figure 3 The structure of the data processing apparatus provided by the embodiment of the present application is shown, and the embodiment of the present application does not limit the specific implementation of the data processing apparatus.

[0139] As shown in Figure 3 , the data processing apparatus can include a processor 702, a communications interface 704, a memory 706 and a communications bus 708.

[0140] The processor 702, the communications interface 704 and the memory 706 can communicate with each other through the communications bus 708. The communications interface 704 is configured to communicate with network elements such as clients or other servers. The processor 702 is configured to execute the program 710, and specifically can execute the related steps in the above-mentioned data processing method embodiment.

[0141] Specifically, the program 710 can include program code, and the program code includes computer executable instructions.

[0142] The processor 702 can be a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to perform the operations of embodiments of the application. The data processing device can include one or more processors of the same type or different types, such as one or more CPUs and one or more ASICs.

[0143] The memory 706 stores the program 710. The memory 706 can include a high-speed RAM memory and can also include a non-volatile memory, such as at least one disk memory.

[0144] The program 710 can be specifically invoked by the processor 702 to cause the data processing device to perform the following operations:

[0145] Obtaining a computing environment parameter; the computing environment parameter includes original computing data;

[0146] Determining a random data quantity, a total number of devices of target computing devices, and an amount of to-be-assigned encoding tasks corresponding to each of the target computing devices according to the computing environment parameter;

[0147] Computing the original computing data according to the random data quantity, the total number of devices, and the amount of to-be-assigned encoding tasks to obtain device encoding data corresponding to each of the target computing devices;

[0148] Obtaining a plurality of intermediate results by computing, according to user input data and the device encoding data, through each of the target computing devices;

[0149] Determining processed data corresponding to the user input data according to all the intermediate results.

[0150] In an optional manner, the computing environment parameter includes a quantity of the original computing data, a computing task size, a quantity of optional computing devices, a quantity of resource limits of each of the optional computing devices, and a unit cost of each of the optional computing devices; and the target computing device is at least one of the optional computing devices.

[0151] In an optional manner, the program 710 is invoked by the processor 702 to cause the data processing device to perform the following operations:

[0152] Determining {1, …, (W) max} and whether the intersection is empty; wherein {1, …, Wmax} is a set of all natural numbers in [1, Wmax]; W is a set of resource limit numbers of all the optional computing devices; (W) max is the largest element in W; r ∈ {1, …, (W) max}; w j is the resource limit number of the optional computing device j; k is the number of the optional computing devices; m is the computing task scale; r is the random data number;

[0153] If the intersection is non-empty, the value of r is traversed in {1, …, (W) max to determine the corresponding

[0154] According to the corresponding of each r value, the minimum value r max and the maximum value r min of r are determined from {1, …, (W) max ;

[0155] The optimal solution of r is obtained by taking r min ≤ r ≤ r max as a constraint condition and taking the minimum total resource consumption cost c sum (r) as an optimization objective, and the optimal solution is determined as the random data number; wherein θ r is the number of the target computing devices; c j is the unit cost of the optional computing device j, and satisfies c j ≥ c j-1 ;

[0156] The total number of devices of the target computing devices and the amount of the to-be-allocated encoding tasks are determined according to the random data number.

[0157] In an optional mode, r is a natural number; the program 710 is called by the processor 702 to make the data processing device perform the following operations:

[0158] The value of r is sequentially calculated from 1 to (W) max , and the corresponding is calculated. The value of r that first satisfies is determined as r min ;

[0159] The value of r is sequentially calculated from (W) max to 1, and the corresponding is calculated. The value of r that first satisfies is determined as r max .

[0160] In an alternative manner, the program 710 is invoked by the processor 702 to cause the data processing device to perform the following operations:

[0161] The number θ of the target computing devices r Determined as

[0162] The amount of coding task to be assigned corresponding to the target computing device j, V(B) j It is determined to be:

[0163]

[0164] In an alternative manner, the program 710 is invoked by the processor 702 to cause the data processing device to perform the following operations:

[0165] Generate random data; the number of random data is the same as the number of random data items.

[0166] The coding coefficient matrix corresponding to each of the target computing devices is determined based on the total number of devices and the amount of coding tasks to be assigned.

[0167] The random data and the original computational data are encoded according to the encoding coefficient matrix to obtain the device encoding data corresponding to each target computing device.

[0168] In an alternative manner, the program 710 is invoked by the processor 702 to cause the data processing device to perform the following operations:

[0169] The (m+r)×(m+r) dimensional coding coefficient matrix B is constructed as follows:

[0170]

[0171] in, E t It is a t×t dimensional identity matrix; O r，m It is an r×m dimensional zero matrix; Let r be the matrix consisting of row vectors with indices from a to b; r be the number of random data points; m be the computational task size; z ∈ {1, ..., θ} r -1};V(B z )=r;V(B j ) represents the amount of coding task to be assigned to the target computing device j;

[0172] The encoding coefficient matrix B of the target computing device j j Determined as:

[0173]

[0174] wherein θ r is the number of the target computing devices.

[0175] The data processing device provided by the embodiment of the present application obtains computing environment parameters, wherein the computing environment parameters include original computing data; determines a random data quantity, a total number of target computing devices, and a to-be-allocated encoding task quantity corresponding to each target computing device according to the computing environment parameters; performs computation on the original computing data according to the random data quantity, the total number of target computing devices, and the to-be-allocated encoding task quantity, to obtain device encoding data corresponding to each target computing device; performs computation on the user input data and the device encoding data according to each target computing device, to obtain a plurality of intermediate results; and determines processed data corresponding to the user input data according to all the intermediate results.

[0176] Unlike the prior art, which evenly allocates computing tasks among all target computing devices without considering that the target computing devices are heterogeneous and have inconsistent processing capabilities, thus resulting in a large total cost of data processing, the data processing device provided by the embodiment of the present application determines an optimal random data quantity, a total number of target computing devices, and a to-be-allocated encoding task quantity corresponding to each target computing device according to computing environment parameters, so that the computing tasks are adaptively allocated among the target computing devices according to the performance of each computing device, and the total cost is minimized, thus overcoming the problem of a large total cost of data processing based on edge computing in the prior art.

[0177] The embodiment of the present application provides a computer readable storage medium, the storage medium stores at least one executable instruction, the executable instruction runs on the data processing device, and makes the data processing device execute the data processing method in any method embodiment.

[0178] The executable instruction can be specifically used to make the data processing device perform the following operations:

[0179] Obtain computing environment parameters; the computing environment parameters include original computing data;

[0180] Determine a random data quantity, a total number of target computing devices, and a to-be-allocated encoding task quantity corresponding to each target computing device according to the computing environment parameters;

[0181] Perform computation on the original computing data according to the random data quantity, the total number of target computing devices, and the to-be-allocated encoding task quantity, to obtain device encoding data corresponding to each target computing device;

[0182] The target computing device calculates a plurality of intermediate results according to the user input data and the device encoding data according to the user input data and the device encoding data;

[0183] The processing data corresponding to the user input data is determined according to all the intermediate results.

[0184] In an optional manner, the computing environment parameters include the number of original computing data, the size of a computing task, the number of optional computing devices, the number of resource limits of each optional computing device, and the unit cost of each optional computing device; and the target computing device is at least one of the optional computing devices.

[0185] In an optional manner, the executable instructions cause the data processing device to perform the following operations:

[0186] Determine whether the intersection of {1, …, (W) max} and is empty; wherein {1, …, Wmax is a set of all natural numbers in [1, Wmax]; W is a set of the number of resource limits of all the optional computing devices; (W) max is the largest element in W; r∈{1, …, (W) max}; w j is the number of resource limits of the optional computing device j; k is the number of the optional computing devices; m is the size of the computing task; r is the number of random data;

[0187] If the intersection is non-empty, the value of r is traversed in {1, …, (W) max}, and

[0188] According to the corresponding of each r value, the minimum value r max and the maximum value r min of r are determined from {1, …, (W) max ;

[0189] The optimal solution of r is obtained under the constraint condition of r min ≤r≤r max and the optimization objective of the minimum total resource consumption cost c sum (r), and the optimal solution is determined as the number of random data; wherein θ r is the number of target computing devices; c j is the unit cost of the optional computing device j, and satisfies c j ≥c j-1 ;

[0190] determining a total number of devices of the target computing devices according to the random data quantity and a quantity of encoding tasks to be allocated.

[0191] In an optional mode, r is a natural number; the executable instructions cause the data processing device to perform the following operations:

[0192] sequentially calculating values of r from 1 to (W) max corresponding determining the first r value satisfying as r min ;

[0193] sequentially calculating values of r from (W) max decreasing to 1 corresponding determining the first r value satisfying as r max .

[0194] In an optional mode, the executable instructions cause the data processing device to perform the following operations:

[0195] determining the number θ r of the target computing devices as

[0196] determining the quantity of encoding tasks V(B j ) corresponding to the target computing device j as:

[0197]

[0198] In an optional mode, the executable instructions cause the data processing device to perform the following operations:

[0199] generating random data; a quantity of the random data is the random data quantity;

[0200] determining an encoding coefficient matrix corresponding to each of the target computing devices according to the total number of devices and the quantity of encoding tasks to be allocated;

[0201] encoding the random data and the original computing data according to the encoding coefficient matrix to obtain the device encoding data corresponding to each of the target computing devices.

[0202] In an optional mode, the executable instructions cause the data processing device to perform the following operations:

[0203] constructing an encoding coefficient matrix B of (m+r)×(m+r) dimensions as follows:

[0204]

[0205] wherein, E t is a unit matrix of dimension t x t; O r，m is a zero matrix of dimension r x m; is a matrix consisting of row vectors from a to b in matrix; r is the number of random data; m is the size of the computing task; z e {1,..., θ r -1}; V(B z ) = r; V(B j ) is the amount of the encoding task to be allocated corresponding to the target computing device j;

[0206] The encoding coefficient matrix B j of the target computing device j is determined as:

[0207]

[0208] wherein, θ r is the number of the target computing devices.

[0209] The computer storage medium of the embodiment of the application obtains computing environment parameters; wherein, the computing environment parameters include original computing data; determines the number of random data, the total number of devices of the target computing devices and the amount of the encoding task to be allocated corresponding to each target computing device according to the computing environment parameters; performs computation on the original computing data according to the number of random data, the total number of devices and the amount of the encoding task to be allocated, to obtain device encoding data corresponding to each target computing device; performs computation through each target computing device according to user input data and the device encoding data, to obtain multiple intermediate results; determines the processed data corresponding to the user input data according to all the intermediate results.

[0210] Different from the prior art that evenly allocates the computing task among all the target computing devices without considering that the target computing devices are heterogeneous and have inconsistent processing capabilities, thus the total cost of data processing is large, the computer storage medium provided by the embodiment of the application determines the optimal number of random data, the total number of devices of the target computing devices and the amount of the encoding task to be allocated corresponding to each target computing device according to the computing environment parameters, so that the computing task is adaptively allocated among the target computing devices according to the performance of each computing device, so that the total cost is minimized, thus the problem of large total cost of data processing based on edge computing in the prior art can be overcome.

[0211] The embodiment of the application provides a data processing device for executing the above-mentioned data processing method.

[0212] The embodiment of the present application provides a computer program, which can be invoked by a processor to enable a data processing device to perform the data processing method in any method embodiment.

[0213] The embodiment of the present application provides a computer program product, which comprises a computer program stored on a computer readable storage medium, and the computer program comprises program instructions, which, when executed on a computer, enable the computer to perform the data processing method in any method embodiment.

[0214] The algorithms and displays presented herein are not inherently related to any particular computer, virtual system, or other apparatus. Various general purpose systems can be used with programs in accordance with the teachings herein, or it can prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems will be apparent from the description above. In addition, the present embodiment is not intended to be limited to any particular programming language. It will be appreciated that a variety of programming languages can be used to implement the teachings of the present application as described herein, and any references below to specific languages are provided for disclosure of enablement only.

[0215] In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the application can be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been described in detail in order to avoid obscuring the understanding of this description.

[0216] Similarly, it is to be understood that the mechanical details of the application sometimes are grouped into single embodiments, figures, or descriptions for clarity sake, in order to not obscure the concepts of the various aspects of the application. However, this method of disclosure is not to be interpreted as reflecting an intention that the application requires more features than are explicitly recited in each claim.

[0217] Those skilled in the art can understand that the modules in the device in the embodiments can be adaptively changed and arranged in one or more devices different from the embodiments. The modules, units or components in the embodiments can be combined into one module, unit or component, and can be divided into multiple sub-modules or sub-units or sub-components. Except that at least some of the features and / or processes or units are mutually exclusive, all combinations of all features disclosed in the specification (including the accompanying claims, abstract and drawings) and all processes or units of any methods or apparatuses disclosed can be used. Unless explicitly stated otherwise, each feature disclosed in the specification (including the accompanying claims, abstract and drawings) can be replaced by an alternative feature providing the same, equivalent or similar purpose.

[0218] It should be noted that the above-mentioned embodiments illustrate rather than limit the application, and that those skilled in the art will be able to design many alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word 'comprising' does not exclude the presence of elements or steps other than those listed in a claim. The word 'a' or 'an' preceding an element does not exclude the presence of a plurality of such elements. The application can be implemented by means of both hardware and software, and any combination thereof. In a unitary claim, several devices, apparatuses or means can be listed, comprising means for carrying out a certain task. The use of the term'means' in a claim is intended to refer to a combination of devices, apparatuses or means for carrying out a task. The word 'first','second', 'third', etc. do not imply any order. The use of these terms is to be construed as an indication of particular embodiments. Steps in the above-described embodiments, unless otherwise specified, are not to be construed as necessarily limiting the order in which the steps are performed.

Claims

1. A data processing method, characterized in that, The method includes: Obtain computing environment parameters; the computing environment parameters include raw computing data; The amount of random data, the total number of target computing devices, and the amount of coding tasks to be assigned to each target computing device are determined based on the computing environment parameters. The original computational data is calculated based on the amount of random data, the total number of devices, and the amount of coding tasks to be assigned, to obtain the device coding data corresponding to each of the target computing devices. Multiple intermediate results are obtained by performing calculations on each of the target computing devices based on user input data and device-encoded data. Based on all the intermediate results, determine the processed data corresponding to the user input data; The computing environment parameters include the amount of raw computing data, the scale of the computing task, the number of optional computing devices, the resource limits of each optional computing device, and the unit cost of each optional computing device; the target computing device is at least one of the optional computing devices. The step of determining the amount of random data, the total number of target computing devices, and the amount of coding tasks to be allocated for each target computing device based on the computing environment parameters includes: Determine {1, ..., (W)} max }and Is the intersection of {1, ..., (W) empty? max } is [1, (W)] max W is the set of all natural numbers in the set; W is the set of resource limits for all the said optional computing devices; (W) max The largest element in W; r∈{1,···,(W) max };w j The resource limit number of optional computing devices j; k is the number of optional computing devices; m is the size of the computing task; r is the amount of random data; If the intersection is non-empty, then the value of r is in {1, ..., (W)}. max The process involves iterating through the data to determine the values ​​corresponding to each r value. According to the respective r values From {1,···,(W)} max Determine the minimum value of r in} min The maximum value of r max ; With r min ≤r≤r max As a constraint, the total cost of resource consumption, c sum (r) is the optimization objective. The optimal solution for r is obtained, and this optimal solution is used to determine the number of random data points. θ r c is the number of the target computing devices; j Let c be the unit cost of the optional computing device j, and satisfy c. j≥ c j-1 ; The total number of target computing devices and the amount of coding tasks to be assigned are determined based on the amount of random data.

2. The method according to claim 1, characterized in that, r is a natural number; the terms corresponding to each r value are as follows: From {1,···,(W)} max Determine the minimum value of r in} min The maximum value of r max ,include: Calculate the value of r sequentially from 1 to (W). max Time corresponding The first one will be satisfied. The value of r is determined as r min ; Calculate the value of r sequentially from (W) max When it decreases to 1, the corresponding The first one will be satisfied. The value of r is determined as r max .

3. The method according to claim 1, characterized in that, The step of determining the total number of target computing devices and the amount of coding tasks to be assigned based on the amount of random data includes: The number θ of the target computing devices r Determined as The amount of coding task to be assigned corresponding to the target computing device j, V(B) j It is determined to be:

4. The method according to claim 1, characterized in that, The step of calculating the original computational data based on the amount of random data, the total number of devices, and the amount of coding tasks to be assigned, to obtain the device coding data corresponding to each of the target computing devices, includes: Generate random data; the number of random data is the same as the number of random data items. The coding coefficient matrix corresponding to each of the target computing devices is determined based on the total number of devices and the amount of coding tasks to be assigned. The random data and the original computational data are encoded according to the encoding coefficient matrix to obtain the device encoding data corresponding to each target computing device.

5. The method according to claim 4, characterized in that, The coding coefficient matrix corresponding to each of the target computing devices is determined based on the total number of devices and the amount of coding tasks to be allocated, including: The (m+r)×(m+r) dimensional coding coefficient matrix B is constructed as follows: in, E t It is a t×t dimensional identity matrix; o r,m It is an r×m dimensional zero matrix; Let r be the matrix consisting of row vectors with indices from a to b; r be the number of random data points; m be the computational task size; z ∈ {1, ..., θ} r -1};V(B z )=r;V(B j ) represents the amount of coding task to be assigned to the target computing device j; The encoding coefficient matrix B of the target computing device j j Determined as: Where, θ r The number of the target computing devices.

6. A data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire computing environment parameters, including raw computing data. The allocation module is used to determine the amount of random data, the total number of target computing devices, and the amount of coding tasks to be allocated for each target computing device based on the computing environment parameters. The encoding module is used to calculate the original computing data based on the amount of random data, the total number of devices, and the amount of encoding tasks to be assigned, so as to obtain the device encoding data corresponding to each of the target computing devices. The calculation module is used to perform calculations on each of the target computing devices based on user input data and device encoded data to obtain multiple intermediate results; The determining module is used to determine the processed data corresponding to the user input data based on all the intermediate results; The computing environment parameters include the amount of raw computing data, the scale of the computing task, the number of optional computing devices, the resource limits of each optional computing device, and the unit cost of each optional computing device; the target computing device is at least one of the optional computing devices. The allocation module is further used for: Determine {1, ..., (W)} max }and Is the intersection of {1, ..., (W) empty? max } is [1, (W)] max W is the set of all natural numbers in the set; W is the set of resource limits for all the said optional computing devices; (W) max The largest element in W; r∈{1,···,(W) max };w j The resource limit number of optional computing devices j; k is the number of optional computing devices; m is the size of the computing task; r is the amount of random data; If the intersection is non-empty, then the value of r is in {1, ..., (W)}. max The process involves iterating through the data to determine the values ​​corresponding to each r value. According to the respective r values From {1,···,(W)} max Determine the minimum value of r in} min The maximum value of r max ; With r min ≤r≤r max As a constraint, the total cost of resource consumption, c sum (r) is the optimization objective. The optimal solution for r is obtained, and this optimal solution is used to determine the number of random data points. θ r c is the number of the target computing devices; j Let c be the unit cost of the optional computing device j, and satisfy c. j≥ c j-1 ; The total number of target computing devices and the amount of coding tasks to be assigned are determined based on the amount of random data.

7. A data processing device, characterized in that, include: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation of the data processing method as described in any one of claims 1-5.

8. A computer-readable storage medium, characterized in that, The storage medium stores at least one executable instruction, which, when executed on the data processing device, causes the data processing device to perform the operation of the data processing method as described in any one of claims 1-5.

Images