A method, system and device for efficient parallel data alignment

Abstract

The embodiment of the present invention provides an efficient parallel data alignment method, system and device, which relates to the field of data processing technology and is applied to a first server. The method includes: according to the data identifiers of each stored data, the stored data are respectively allocated to different preset first arrays; for each first array, the data contained in the first array and the data contained in the target second array in the second server are subjected to data alignment processing to obtain data alignment sub-results, the second server is preset with a second array corresponding to the first array one by one, and the target second array is: the second array corresponding to the first array; each data alignment sub-result is merged to obtain the data alignment result between the first server and the second server. The solution provided by the embodiment of the present invention can reduce the data transmission resources consumed in a single data alignment process.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an efficient parallel data alignment method, system and device. Background Art

[0002] An object often has multiple types of feature data. For example, when the object is a user, the user's feature data includes age, gender, occupation, education, and other types of feature data. In addition, the types of feature data of objects stored in the servers of different organizations are different. For example, the communication company's server stores user's call data, SMS data, and other types of feature data, while the shopping website's server stores user's purchase data, purchase time data, and other types of feature data.

[0003] In some application scenarios, it is necessary to jointly process the feature data of objects stored in servers of multiple organizations. However, because the feature data of the same object may not be stored in the servers of different organizations, it is necessary to determine the objects whose feature data are stored in different servers before data processing, that is, to perform data alignment on the objects corresponding to the feature data stored in different servers.

[0004] In the prior art, when servers of multiple organizations jointly perform data alignment processing, the servers can be divided into a sending end and a receiving end. The sending end needs to send all the data stored in itself to the receiving end before the receiving end can start the data alignment processing. A large amount of data needs to be transmitted between the sending end and the receiving end at one time, consuming a large amount of data transmission resources at one time. Summary of the invention

[0005] The purpose of the embodiments of the present invention is to provide an efficient parallel data alignment method, system and device method to reduce the data transmission resources consumed in a single data alignment process. The specific technical solution is as follows:

[0006] In a first aspect, an embodiment of the present invention provides an efficient parallel data alignment method, which is applied to a first server, and the method includes:

[0007] According to the data identifiers of the stored data, the stored data are respectively allocated to different preset first arrays;

[0008] For each first array, data alignment processing is performed on the data included in the first array and the data included in the target second array in the second server to obtain a data alignment sub-result, wherein the second server is: a server participating in the data alignment processing except the first server, a second array corresponding to the first array is preset in the second server, the target second array is: a second array corresponding to the first array, and the data in the target second array is: the second server allocates the data stored in itself to the target second array in the same way as the first server;

[0009] The data alignment sub-results are combined to obtain a data alignment result between the first server and the second server.

[0010] In one embodiment of the present invention, for each first array, performing data alignment processing on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result includes:

[0011] Determine a first array where data exists;

[0012] Sending the determined identifier of the first array to the second server, so that the second server determines a second array contained in the second server and corresponding to the first array containing the data;

[0013] For each first array containing data, data alignment processing is performed on the data contained in the first data and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0014] In one embodiment of the present invention, the step of allocating the stored data to different preset first arrays according to the data identifiers of the stored data includes:

[0015] Calculate the hash value of each data separately;

[0016] According to the corresponding relationship between the preset hash value and the first array, the stored data is respectively distributed to different preset first arrays.

[0017] In one embodiment of the present invention, for each first array, performing data alignment processing on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result includes:

[0018] For each first array, based on the oblivious transfer (OT) protocol, data alignment processing is performed on the data included in the first array and the data included in the target second array in the second server to obtain a data alignment sub-result.

[0019] In one embodiment of the present invention, the amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

[0020] In a second aspect, an embodiment of the present invention provides an efficient parallel data alignment system, the system comprising a first server and a second server;

[0021] The first server is used to distribute the stored data into different preset first arrays according to the data identifiers of the stored data;

[0022] The second server is used to allocate the stored data to different preset second arrays according to the data identifiers of the stored data in the same manner as the first server, wherein the second array is a preset array corresponding to the first array one by one;

[0023] The first server and the second server are further configured to perform data alignment processing on the data contained in each first array in the first server and the data contained in a target second array in the second server, wherein the target second array is: a second array corresponding to the first array;

[0024] The first server is further configured to obtain a data alignment sub-result for each first array, and merge the data alignment sub-results to obtain a data alignment result between the first server and the second server.

[0025] In one embodiment of the present invention,

[0026] The first server is further configured to determine a first array of data; and send an identifier of the determined first array to the second server;

[0027] The second server is further configured to determine, based on the received identifier of the first array, a second array contained in the second server corresponding to the first array containing the data;

[0028] The first server and the second server are specifically used to perform data alignment processing on the data contained in each first array containing data and the data contained in the target second array in the second array determined in the second server to obtain a data alignment sub-result.

[0029] In one embodiment of the present invention, the first server is specifically used to:

[0030] Calculate the hash value of each stored data respectively;

[0031] According to a first corresponding relationship between a preset hash value and the first array, the stored data is respectively allocated to different preset first arrays;

[0032] The second server is specifically used for:

[0033] Calculate the hash value of each stored data respectively;

[0034] According to a second correspondence between a preset hash value and a second array, the stored data is respectively distributed to different preset second arrays, wherein according to the first correspondence and the second correspondence, the first array and the second array corresponding to the same hash value correspond to each other.

[0035] In one embodiment of the present invention, the first server and the second server are specifically used to:

[0036] For each first array, based on the OT protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0037] In one embodiment of the present invention, the amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

[0038] In a third aspect, an embodiment of the present invention provides an efficient parallel data alignment device, which is applied to a first server, and the device includes:

[0039] A data allocation module, used for respectively allocating the stored data to different preset first arrays according to the data identifiers of the stored data;

[0040] a data alignment module, configured to perform data alignment processing on the data contained in the first array and the data contained in the target second array in the second server for each first array, and obtain a data alignment sub-result, wherein the second server is a server other than the first server that participates in the data alignment processing, the second server is preset with a second array corresponding to the first array in a one-to-one manner, the target second array is a second array corresponding to the first array, and the data in the target second array is the data stored in the second server allocated to the target second array by the second server in the same manner as the first server;

[0041] The result merging module is used to merge the various data alignment sub-results to obtain the data alignment result between the first server and the second server.

[0042] In one embodiment of the present invention, the data alignment module is specifically used to:

[0043] Determine a first array where data exists;

[0044] Sending the determined identifier of the first array to the second server, so that the second server determines a second array contained in the second server and corresponding to the first array containing the data;

[0045] For each first array containing data, data alignment processing is performed on the data contained in the first data and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0046] In one embodiment of the present invention, the data allocation module is specifically used to:

[0047] Calculate the hash value of each data separately;

[0048] According to the corresponding relationship between the preset hash value and the first array, the stored data is respectively distributed to different preset first arrays.

[0049] In one embodiment of the present invention, the data alignment module is specifically used to:

[0050] For each first array, based on the OT protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0051] In one embodiment of the present invention, the amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

[0052] In a fourth aspect, an embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0053] Memory, used to store computer programs;

[0054] The processor is used to implement any method step described in the first aspect when executing a program stored in the memory.

[0055] In a fifth aspect, an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, any method step described in the first aspect is implemented.

[0056] In a sixth aspect, an embodiment of the present invention further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the method steps described in the first aspect above.

[0057] Beneficial effects of the embodiments of the present invention:

[0058] The embodiment of the present invention provides an efficient parallel data alignment method, which is applied to a first server. The first server allocates the stored data to different preset first arrays according to the data identifiers of each stored data. For each first array, the data contained in the first array and the data contained in the target second array in the second server are subjected to data alignment processing to obtain a data alignment sub-result. The second server is provided with a second array corresponding to the first array one by one. The second server allocates its own stored data to different second arrays in the same manner as the first server. The target second array is the second array corresponding to the first array. The data alignment sub-results are merged to obtain a data alignment result between the first server and the second server.

[0059] As can be seen from the above, since the first server and the second server use the same method to allocate their respective stored data to the first array and the second array respectively, and since the allocation method used is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, by subsequently performing data alignment processing on the data in the corresponding first data and the second data, different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention, and a person skilled in the art can also obtain other drawings based on these drawings.

[0061] Figure 1 A schematic flow chart of a first efficient parallel data alignment method provided by an embodiment of the present invention;

[0062] Figure 2 A schematic flow chart of a second efficient parallel data alignment method provided by an embodiment of the present invention;

[0063] Figure 3 A schematic diagram of the structure of an efficient parallel data alignment system provided by an embodiment of the present invention;

[0064] Figure 4A schematic flow chart of a third efficient parallel data alignment method provided by an embodiment of the present invention;

[0065] Figure 5 A schematic flow chart of a fourth efficient parallel data alignment method provided by an embodiment of the present invention;

[0066] Figure 6 A schematic diagram of the structure of an efficient parallel data alignment device provided by an embodiment of the present invention;

[0067] Figure 7 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field based on this application belong to the scope of protection of the present invention.

[0069] Since different servers in the prior art need to transmit a large amount of data at one time when performing data alignment processing, a single data alignment consumes a large amount of data transmission resources. In order to solve the above problem, an embodiment of the present invention provides an efficient parallel data alignment method, system and device.

[0070] An embodiment of the present invention provides an efficient parallel data alignment method, which is applied to a first server. The method includes:

[0071] According to the data identifiers of the stored data, the stored data are respectively allocated to different preset first arrays;

[0072] For each first array, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result, wherein the second server is: a server participating in the data alignment processing except the first server, the second server is preset with a second array corresponding to the first array one by one, the target second array is: a second array corresponding to the first array, and the data in the target second array is: the second server allocates the data stored in itself to the target second array in the same way as the first server;

[0073] The data alignment sub-results are combined to obtain a data alignment result between the first server and the second server.

[0074] As can be seen from the above, since the first server and the second server use the same method to allocate their respective stored data to the first array and the second array respectively, and since the allocation method used is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, by subsequently performing data alignment processing on the data in the corresponding first data and the second data, different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0075] See also Figure 1 , is a flow chart of a first efficient parallel data alignment method provided by an embodiment of the present invention, which is applied to a first server, and the first server can be any one of the servers participating in the data alignment process. For example, if a user-side server with less stored data and a cloud server with more stored data perform data alignment together, the first server can be either a user-side server or a cloud server.

[0076] Specifically, the above method includes S101-S103.

[0077] S101: According to data identifiers of respective stored data, the stored data are respectively allocated to different preset first arrays.

[0078] Specifically, the first server participating in data alignment and the second server other than the first server need to use the same method to allocate data, and the second server can be one or more. The second server is preset with a second array corresponding to the first array one by one, and the number of the first array and the second array is the same. The first server allocates the data stored in itself to the first array in the above manner, and the second server allocates the data stored in itself to the second array in the above manner. Since the first array corresponds to the second array one by one, and the first server and the second server use the same data allocation method, for the same data in the first server and the second server, the first array allocated by the first server corresponds to the second array allocated by the second server.

[0079] In order to ensure that the first server and the second server can use the same method to allocate data, and to ensure that the first array and the second array correspond one-to-one, the user can pre-set the same data allocation method and the one-to-one corresponding first array and second array in the first server and the second server respectively.

[0080] In addition, the first server and the second server may also unify the data allocation method and the number of arrays before performing data allocation. Specifically, the first server may send the identifier of the data allocation method and the number of arrays to the second server to unify the data allocation method and the number of arrays, and then start to execute step S101. Or after receiving the identifier of the data allocation method and the number of arrays sent by the second server, the first server executes step S101 according to the data allocation method indicated by the second server, and allocates the data to the received number of first arrays.

[0081] Among them, different data have different data identifiers, and the same data has the same corresponding data identifier in the first server and the second server. For example, when the above data is user data, the above data identifier can be the user's ID number, telephone number, etc.

[0082] In one embodiment of the present invention, the value ranges of data identifiers corresponding to different arrays are different, and different data can be allocated to different arrays according to the values ​​of the data identifiers. Taking the data identifier as an ID number as an example, the value ranges of ID numbers corresponding to different first arrays can be preset, and the value ranges of the ID numbers of the corresponding first array and second array are the same, and the first array and the second array can allocate their respective stored data to different arrays according to the values ​​of the ID number.

[0083] In another embodiment of the present invention, the first server may also distribute the stored data into different first arrays according to the following steps A-B, and the second server may also distribute the stored data into different first arrays according to the following steps A-B.

[0084] Step A: Calculate the hash value of each data separately.

[0085] Specifically, a uniform hash function can be used to calculate the hash value of each data. The hash values ​​of different data calculated using the uniform hash function are often different. In this case, the number of the set first array can be greater than the minimum number, that is, the number of the second array is also greater than the minimum number. The above minimum number is the minimum value of the number of data stored in the first server and the number of data stored in the second server. Since the number of the first array and the second array is greater than the minimum number, for a server that stores fewer data, the data recorded in the server can be roughly evenly distributed to different arrays. Each array may contain only one piece of data, and there must be an array that does not contain data after the data is distributed.

[0086] In addition, a non-uniform hash function may also be used to calculate the hash value of each data.

[0087] Furthermore, the hash value calculation function adopted by the second server when distributing data is the same as that adopted by the first server.

[0088] Step B: According to the correspondence between the preset hash values ​​and the first arrays, the stored data are respectively allocated to different preset first arrays.

[0089] Specifically, different hash values ​​calculated for different data are different, and each hash value can be divided by a preset dividend to obtain a remainder, and the value of the preset dividend is the same as the number of the first array. For example, if the number of the first array is 10, the preset dividend is 10. Different first arrays can correspond to different remainders, so that the stored data can be allocated to the first array corresponding to the calculated remainder.

[0090] In addition, the corresponding remainders of the first array and the second array are the same.

[0091] S102: For each first array, perform data alignment processing on the data included in the first array and the data included in the target second array in the second server to obtain a data alignment sub-result.

[0092] The target second array is: a second array corresponding to the first array.

[0093] Specifically, the first server and the second server may perform data alignment processing on the data contained in different first arrays and the target second arrays corresponding to the first arrays in sequence, or may perform data alignment processing on the data contained in multiple first arrays and the target second arrays corresponding to the first arrays synchronously and in parallel, thereby realizing a parallel data alignment processing process.

[0094] In addition, the obtained data alignment sub-result includes: data contained in both the first array and the target second array. Since the first server and the second server use the same method to distribute data, the same data stored in the first server and the second server will be distributed to the corresponding first array and the second array respectively. Therefore, by performing data alignment processing on the data contained in the corresponding first array and the second array respectively, the data contained in both the first server and the second server can be determined without omission.

[0095] In one embodiment of the present invention, the first server and the second server may use the data alignment method in the prior art to perform data alignment processing on the data contained in the first array and the target second array. For the data contained in each first array, the data alignment processing may be performed in the same manner or in different manners. For example, the data alignment method may be data alignment based on homomorphic encryption, etc., which is not limited in the embodiment of the present invention.

[0096] In another embodiment of the present invention, the following step C may be used to implement the above step S102.

[0097] Step C: for each first array, based on the Oblivious Transfer (OT) protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0098] Among them, there are multiple data alignment methods based on the OT protocol in the prior art, and one of the methods is briefly described here.

[0099] Specifically, the first server can map each piece of data recorded by itself to a preset numerical range based on a preset key to generate a first numerical matrix. The first server also randomly generates a random matrix composed of numerical values ​​0 and 1, and the number of rows and columns of the random matrix are the same as those of the first numerical matrix. For each element in the first numerical matrix, the first server performs an XOR calculation on the element and the element at the target position in the random matrix to obtain a calculation result, and the target position is: the row and column position of the element in the first numerical matrix, and the calculation result is used as the element at the target position in the second numerical matrix to obtain a second numerical matrix. The number of rows and columns in the second numerical matrix is ​​also the same as that of the first numerical matrix.

[0100] The second server generates an OT key based on the OT protocol. The number of bits contained in the OT key is the same as the number of columns contained in the above numerical matrix, and each bit corresponds to a column in the second numerical matrix. The first server and the second server exchange data based on the OT protocol, and the second server generates an auxiliary matrix.

[0101] Specifically, a bit value of 1 in the above-mentioned OT key indicates that the second server obtains the element value contained in the first target column corresponding to the bit in the second numerical matrix, as the value of each element in the column of the auxiliary matrix generated by itself, whose column number is the same as the column number of the first target column; a bit value of 0 indicates that the second server obtains the element value contained in the second target column corresponding to the bit in the random matrix, as the element value of each element in the column of the auxiliary matrix generated by itself, whose column number is the same as the column number of the second target column.

[0102] It can be seen from the above that since the values ​​of each element contained in the auxiliary matrix generated by the second server are all from the second numerical matrix and the random matrix generated by the first server, the first server needs to send the data stored in itself to the second server during the process of the second server generating the auxiliary matrix. The more data contained in the first server, the larger the generated second data matrix and the random matrix, and the more data the first server needs to transmit to the second server during the data alignment process. Therefore, the server with the least amount of stored data can be selected from the servers participating in the data alignment process as the first server, that is, the amount of data stored by the first server is less than or equal to the amount of data stored by the second server, thereby reducing the amount of data transmitted from the first server to the second server during the data alignment process based on the OT protocol, thereby reducing the consumption of data transmission resources.

[0103] For example, if the servers participating in the data alignment process are respectively a user-end server storing less data and a cloud server storing more data, the user-end server may be selected as the first server.

[0104] Furthermore, the first server can send the preset key used when generating the first numerical matrix to the second server, and the second server also maps each data stored in itself to the preset numerical range based on the preset key to generate a third numerical matrix. The second server further performs privacy intersection processing on the third numerical matrix and the auxiliary matrix to obtain a first intersection matrix, and sends the first intersection matrix to the first server.

[0105] In addition, the first server performs the same privacy intersection processing on the first numerical matrix and the random matrix to obtain a second intersection matrix. The first server compares the first intersection matrix with the second intersection matrix to obtain a data alignment processing sub-result.

[0106] It should be noted that, since the above-mentioned data alignment processing method based on the OT protocol belongs to the prior art, the embodiment of the present invention only briefly describes it.

[0107] S103: Merge each data alignment sub-result to obtain a data alignment result between the first server and the second server.

[0108] Specifically, different data alignment sub-results respectively include data included in different first arrays and corresponding second arrays, and a union of the data alignment sub-results can be obtained as a data alignment result between the first server and the second server.

[0109] As can be seen from the above, since the first server and the second server use the same method to allocate their respective stored data to the first array and the second array respectively, and since the allocation method used is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, by subsequently performing data alignment processing on the data in the corresponding first data and the second data, different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0110] See also Figure 2 , is a flow chart of a second efficient parallel data alignment processing method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 1 Compared with the illustrated embodiment, the above step S102 can be implemented by the following steps S102A-S102C.

[0111] S102A: Determine the first array in which data exists.

[0112] Specifically, since the first arrays are preset, after the first server allocates the data to different first arrays, some of the first arrays may not contain data. In particular, if the number of allocated first arrays is greater than the number of data stored in the first server, there will inevitably be first arrays that do not contain data.

[0113] The data included in the second array corresponding to the first array without data will not be stored in the first server, so the first array without data may not participate in the subsequent data alignment process. On the contrary, the embodiment of the present invention may determine the first array with data, and only continue the subsequent data alignment process for the data included in the first array with data.

[0114] S102B: Sending the determined identifier of the first array to the second server, so that the second server determines the second array contained in the second server and corresponding to the first array containing the data.

[0115] The identifier of the first array may be the serial number of the first array. The second server may record the correspondence between the first array and the second array. After receiving the identifier of the first array, the second server determines the second array corresponding to the first array containing data based on the correspondence. Thus, it can be determined that the data contained in the other second arrays except the determined second array does not participate in the subsequent data alignment process.

[0116] In particular, if the amount of data contained in the first server is much smaller than the amount of data contained in the second server, the second array corresponding to the first array that does not contain data often contains a larger amount of data. The data in the above-mentioned second array does not participate in the subsequent data alignment processing process, which can greatly reduce the computing resources required for the subsequent data alignment processing process.

[0117] For example, if 100 first arrays are set in the first server and 100 second arrays are set in the second server, in this case, if 10 pieces of data are stored in the first server, even if the 10 pieces of data are respectively allocated to 10 different first arrays, there are still 90 first arrays that do not contain data. If 10 million pieces of data are stored in the second server, taking the example of 10 million pieces of data being evenly allocated to 100 different second arrays, each second array contains 100,000 pieces of data after allocation. Since the 90 first arrays do not contain data, they do not need to participate in the subsequent data alignment process, and the data contained in the 90 second arrays corresponding to the first arrays that do not contain data will not participate in the subsequent data alignment process, that is, the 9 million pieces of data stored in the second server will not participate in the subsequent data alignment process, and the computing resources required in the data alignment process are greatly reduced.

[0118] S102C: For each first array containing data, perform data alignment processing on the data contained in the first data and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0119] Specifically, step S102C is similar to the aforementioned step S102, and will not be described in detail in this embodiment of the present invention.

[0120] As can be seen from the above, in the embodiment of the present invention, only the first array containing data is subjected to the subsequent data alignment processing. Therefore, the data contained in the second array corresponding to the first array not containing data in the second server will not participate in the subsequent data alignment processing, thereby reducing the total number of data participating in the data alignment processing in the second server, thereby improving the overall efficiency of the data alignment processing process.

[0121] Corresponding to the aforementioned efficient parallel data alignment method applied to the first server, an embodiment of the present invention provides an efficient parallel data alignment system.

[0122] See also Figure 3, is a schematic diagram of the structure of an efficient parallel data alignment system provided by an embodiment of the present invention, the system includes a first server 301 and a second server 302. The first server 301 may be any server participating in the data alignment process, the second server 302 may be other servers participating in the data alignment process except the first server 301, and the number of the second servers 302 may be one or more.

[0123] See also Figure 4 , is a flow chart of a third efficient parallel data alignment method provided in an embodiment of the present invention, the method includes the following steps S401-S404.

[0124] S401: The first server 301 allocates the stored data to different preset first arrays according to the data identifiers of the stored data.

[0125] S402: The second server 302 allocates the stored data to different preset second arrays according to the data identifiers of the stored data in the same manner as the first server.

[0126] Among them, the above-mentioned second array is: a preset array that corresponds one-to-one to the above-mentioned first array.

[0127] S403 : The first server 301 and the second server 302 perform data alignment processing on the data included in each first array in the first server 301 and the data included in the target second array in the second server 302 .

[0128] The target second array is: a second array corresponding to the first array.

[0129] S404 : the first server 301 obtains the data alignment sub-result for each first array, and merges the data alignment sub-results to obtain the data alignment result between the first server 301 and the second server 302 .

[0130] As can be seen from the above, since the first server and the second server use the same method to allocate their respective stored data to the first array and the second array respectively, and since the allocation method used is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, by subsequently performing data alignment processing on the data in the corresponding first data and the second data, different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0131] See also Figure 5 , is a flow chart of a fourth efficient parallel data alignment method provided by an embodiment of the present invention, which is similar to the aforementioned Figure 4 Compared with the embodiment shown, the above method further includes the following steps S405-S407, and the above step S403 can be implemented by the following step S403A.

[0132] S405: The first server 301 determines that a first array of data exists.

[0133] S406: The first server 301 sends the determined identifier of the first array to the second server 302.

[0134] S407: The second server 302 determines the second array corresponding to the first array containing data based on the received identifier of the first array.

[0135] S403A: The first server 301 and the second server 302 perform data alignment processing on the data contained in each first array containing data and the data contained in the target second array in the second array determined in the second server 302 to obtain a data alignment sub-result.

[0136] As can be seen from the above, in the embodiment of the present invention, only the first array containing data is subjected to the subsequent data alignment processing. Therefore, the data contained in the second array corresponding to the first array not containing data in the second server will not participate in the subsequent data alignment processing, thereby reducing the total number of data participating in the data alignment processing in the second server, thereby improving the overall efficiency of the data alignment processing process.

[0137] In one embodiment of the present invention, the first server 301 is specifically used for:

[0138] Calculate the hash value of each stored data respectively;

[0139] According to a first corresponding relationship between a preset hash value and the first array, the stored data is respectively allocated to different preset first arrays;

[0140] The second server 302 is specifically used for:

[0141] Calculate the hash value of each stored data respectively;

[0142] According to a second correspondence between a preset hash value and a second array, the stored data is respectively distributed to different preset second arrays, wherein according to the first correspondence and the second correspondence, the first array and the second array corresponding to the same hash value correspond to each other.

[0143] In one embodiment of the present invention, the first server 301 and the second server 302 are specifically used for:

[0144] For each first array, based on the OT protocol, data alignment processing is performed on the data included in the first array and the data included in the target second array in the second server 302 to obtain a data alignment sub-result.

[0145] In one embodiment of the present invention, the amount of data stored in the first server 301 is less than or equal to the amount of data stored in the second server 302 .

[0146] Corresponding to the aforementioned efficient parallel data alignment method applied to the first server, an embodiment of the present invention further provides an efficient parallel data alignment device applied to the first server.

[0147] See also Figure 6 , is a schematic diagram of the structure of an efficient parallel data alignment device provided by an embodiment of the present invention, which is applied to a first server, and the device includes:

[0148] The data allocation module 601 is used to allocate the stored data to different preset first arrays according to the data identifiers of the stored data;

[0149] The data alignment module 602 is used to perform data alignment processing on the data contained in the first array and the data contained in the target second array in the second server for each first array, and obtain a data alignment sub-result, wherein the second server is a server other than the first server that participates in the data alignment processing, the second server is preset with a second array corresponding to the first array in a one-to-one manner, the target second array is a second array corresponding to the first array, and the data in the target second array is: the second server allocates the data stored in itself to the target second array in the same manner as the first server;

[0150] The result merging module 603 is used to merge the various data alignment sub-results to obtain the data alignment result between the first server and the second server.

[0151] As can be seen from the above, since the first server and the second server use the same method to allocate their respective stored data to the first array and the second array respectively, and since the allocation method used is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, by subsequently performing data alignment processing on the data in the corresponding first data and the second data, different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0152] In one embodiment of the present invention, the data alignment module 602 is specifically used to:

[0153] Determine a first array where data exists;

[0154] Sending the determined identifier of the first array to the second server, so that the second server determines a second array contained in the second server and corresponding to the first array containing the data;

[0155] For each first array containing data, data alignment processing is performed on the data contained in the first data and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0156] As can be seen from the above, in the embodiment of the present invention, only the first array containing data is subjected to the subsequent data alignment processing. Therefore, the data contained in the second array corresponding to the first array not containing data in the second server will not participate in the subsequent data alignment processing, thereby reducing the total number of data participating in the data alignment processing in the second server, thereby improving the overall efficiency of the data alignment processing process.

[0157] In one embodiment of the present invention, the data allocation module 601 is specifically used to:

[0158] Calculate the hash value of each data separately;

[0159] According to the corresponding relationship between the preset hash value and the first array, the stored data is respectively distributed to different preset first arrays.

[0160] In one embodiment of the present invention, the data alignment module 602 is specifically used to:

[0161] For each first array, based on the OT protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

[0162] In one embodiment of the present invention, the amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

[0163] The embodiment of the present invention further provides an electronic device, such as Figure 7 As shown, it includes a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704.

[0164] Memory 703, used for storing computer programs;

[0165] The processor 701 is configured to implement a high-efficiency parallel data alignment method applied to the first server when executing a program stored in the memory 703 .

[0166] When the electronic device provided by the embodiment of the present invention is used as the first server for data alignment, since the first server and the second server respectively allocate the data stored in each to the first array and the second array in the same manner, since the allocation method adopted is the same, if the same data exists, the same data will be allocated to the corresponding first array and the second array. Therefore, the data alignment processing is subsequently performed on the data in the corresponding first data and the second data, respectively, and different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0167] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0168] The communication interface is used for communication between the above electronic device and other devices.

[0169] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0170] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0171] In another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned efficient parallel data alignment methods are implemented.

[0172] When executing the computer program stored in the computer-readable storage medium applied to the data processing end provided by the embodiment of the present invention to perform data alignment, since the first server and the second server respectively allocate the data stored in each of them to the first array and the second array in the same manner, since the allocation method adopted is the same, if there is the same data, the same data will be allocated to the corresponding first array and the second array. Therefore, the data alignment processing is subsequently performed on the data in the corresponding first data and the second data, respectively, and different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0173] In another embodiment provided by the present invention, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the efficient parallel data alignment methods in the above embodiments.

[0174] When executing the computer program stored in the computer-readable storage medium applied to the data provider provided by the embodiment of the present invention to perform data alignment, since the first server and the second server respectively allocate the data stored in each of them to the first array and the second array in the same way, since the allocation method adopted is the same, if there is the same data, the same data will be allocated to the corresponding first array and the second array. Therefore, the data alignment processing is subsequently performed on the data in the corresponding first data and the second data, respectively, and different data alignment sub-results can be obtained respectively, and then the different data alignment sub-results are merged to obtain the data alignment result. Since in the embodiment of the present invention, only the data in one array needs to be aligned at one time, the data transmission resources consumed in a single data alignment process can be reduced. In addition, the data alignment processing of different arrays can be performed in parallel, and the efficiency of data alignment is also high.

[0175] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented by software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state hard disk Solid State Disk (SSD)), etc.

[0176] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0177] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for systems, devices, electronic devices, computer-readable storage media, and computer program products, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0178] The above description is only a preferred embodiment of the present invention and is not intended to limit the protection scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.

Claims

1. An efficient parallel data alignment method, characterized in that: Applied to a first server, the method comprises: According to the data identifiers of the stored data, the stored data are respectively allocated to different preset first arrays; Determine a first array where data exists; send an identifier of the determined first array to a second server, so that the second server determines a second array contained in itself and corresponding to the first array where data exists; for each first array where data exists, perform data alignment processing on the data contained in the first array and the data contained in a target second array in the second server to obtain a data alignment sub-result, wherein the second server is: a server participating in the data alignment processing except the first server, a second array corresponding to the first array is preset in the second server, the target second array is: a second array corresponding to the first array, and the data in the target second array is: the second server allocates the data stored in itself to the target second array in the same manner as the first server; The data alignment sub-results are combined to obtain a data alignment result between the first server and the second server.

2. The method according to claim 1, characterized in that The step of allocating the stored data to different preset first arrays according to the data identifiers of the stored data includes: Calculate the hash value of each data separately; According to the corresponding relationship between the preset hash value and the first array, the stored data is respectively distributed to different preset first arrays.

3. The method according to claim 1, characterized in that For each first array containing data, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result, including: For each first array containing data, based on the oblivious transfer (OT) protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

4. The method according to any one of claims 1 to 3, characterized in that The amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

5. An efficient parallel data alignment system, characterized in that: The system comprises a first server and a second server; The first server is used to distribute the stored data into different preset first arrays according to the data identifiers of the stored data; The second server is used to allocate the stored data to different preset second arrays according to the data identifiers of the stored data in the same manner as the first server, wherein the second array is a preset array corresponding to the first array one by one; The first server is further configured to determine a first array of data; and send an identifier of the determined first array to the second server; The second server is further configured to determine, based on the received identifier of the first array, a second array contained in the second server and corresponding to the first array containing the data; The first server and the second server are specifically configured to perform data alignment processing on the data contained in each first array having data and the data contained in a target second array in the second array determined in the second server, to obtain a data alignment sub-result, wherein the target second array is: a second array corresponding to the first array; The first server is further configured to obtain a data alignment sub-result for each first array, and merge the data alignment sub-results to obtain a data alignment result between the first server and the second server.

6. The system according to claim 5, characterized in that The first server is specifically used for: Calculate the hash value of each stored data respectively; According to a first corresponding relationship between a preset hash value and the first array, the stored data is respectively allocated to different preset first arrays; The second server is specifically used for: Calculate the hash value of each stored data respectively; According to a second correspondence between a preset hash value and a second array, the stored data is respectively distributed to different preset second arrays, wherein according to the first correspondence and the second correspondence, the first array and the second array corresponding to the same hash value correspond to each other.

7. The system according to claim 5, characterized in that The first server and the second server are specifically used for: For each first array containing data, based on the OT protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

8. The system according to any one of claims 5 to 7, characterized in that: The amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

9. An efficient parallel data alignment device, characterized in that: Applied to a first server, the device comprises: A data allocation module, used for respectively allocating the stored data to different preset first arrays according to the data identifiers of the stored data; A data alignment module is used to determine a first array where data exists; send an identifier of the determined first array to a second server, so that the second server determines a second array contained in itself and corresponding to the first array where data exists; for each first array where data exists, perform data alignment processing on the data contained in the first array and the data contained in a target second array in the second server to obtain a data alignment sub-result, wherein the second server is: a server participating in the data alignment processing except the first server, a second array corresponding to the first array is preset in the second server, the target second array is: a second array corresponding to the first array, and the data in the target second array is: the second server allocates the data stored in itself to the target second array in the same way as the first server; The result merging module is used to merge the various data alignment sub-results to obtain the data alignment result between the first server and the second server.

10. The device according to claim 9, characterized in that The data allocation module is specifically used for: Calculate the hash value of each data separately; According to the corresponding relationship between the preset hash value and the first array, the stored data is respectively distributed to different preset first arrays.

11. The device according to claim 9, characterized in that The data alignment module is specifically used for: Determine a first array where data exists; send an identifier of the determined first array to the second server, so that the second server determines a second array contained in itself and corresponding to the first array where data exists; For each first array containing data, based on the OT protocol, data alignment processing is performed on the data contained in the first array and the data contained in the target second array in the second server to obtain a data alignment sub-result.

12. The device according to any one of claims 9 to 11, characterized in that The amount of data stored in the first server is less than or equal to the amount of data stored in the second server.

13. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 4 when executing a program stored in a memory.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 4 are implemented.

Images