A data processing method and device

Abstract

The application provides a data processing method and device, the method comprises the following steps: receiving the storage location information of a cloud service platform sent by a data provider, and pulling encrypted data according to the storage location information; transmitting the encrypted data to a distributed cluster for storage after the encrypted data is decrypted; generating a cluster data calculation task request according to the path information used for storage of the distributed cluster and a calculation task, and submitting the cluster data calculation task request to the distributed cluster; receiving a data settlement result returned by the distributed cluster after the data obtained from the path information according to the calculation task is calculated, transmitting the data settlement result to the cloud service platform after the data settlement result is encrypted, and notifying the data provider to pull the data settlement result. The security of data can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data, in particular to a data processing method and device. BACKGROUND

[0002] Under the Internet environment, massive data is generated from different sources and systems, and is usually stored and processed by different data suppliers. For specific business application scenarios, data transmission, circulation, processing, storage and auditing are usually required. For example, for the business application scenario of information delivery effect tracking, since the data volume of delivery effect tracking is very large, the data provider needs to transmit data from different sources to a data processing platform with large computing capacity. The data processing platform analyzes and processes the received data for data post-processing, thereby obtaining data of information conversion effect and returning it to the data provider.

[0003] Currently, in the transmission process of transmitting data to the data processing platform and transmitting the data of information conversion effect back to the data provider, the security of data transmission can be ensured by encrypting the transmitted data. However, in the data transmission process, the data will be leaked by cracking the encrypted data or by the data processing platform decrypting and storing the data. Therefore, how to improve data security is a technical problem to be solved. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a data processing method and device to improve data security.

[0005] In a first aspect, the present application provides a data processing method, comprising:

[0006] receiving the storage location information of the cloud service platform sent by the data provider, and pulling encrypted data according to the storage location information;

[0007] transmitting the decrypted data to a distributed cluster for storage;

[0008] generating a cluster data computing task request according to the path information for storage and the computing task of the distributed cluster, and submitting it to the distributed cluster;

[0009] receiving the data settlement result returned by the distributed cluster according to the data obtained from the path information according to the computing task, and transmitting it to the cloud service platform after encryption processing, and notifying the data provider to pull the data settlement result.

[0010] With reference to the first aspect, in a first possible implementation of the first aspect, the method further comprises, after receiving the storage location information of the cloud service platform sent by the data providing party, before pulling the encrypted data according to the storage location information, the method further comprises:

[0011] The data processing party sets an independent processing unit for processing the compressed data according to the received storage location information of the cloud service platform, so that the independent processing unit performs the pulling of the compressed data according to the storage location information from the cloud service platform.

[0012] With reference to the first aspect, in a second possible implementation of the first aspect, the method further comprises, before receiving the storage location information of the cloud service platform sent by the data providing party, the method further comprises:

[0013] The data to be transmitted is salted to obtain salted data;

[0014] The salted data is encrypted according to a pre-set encryption algorithm to obtain encrypted data, and the encrypted data is transmitted to the cloud service platform for storage.

[0015] With reference to the second possible implementation of the first aspect, in a third possible implementation of the first aspect, the encryption algorithm comprises an information digest algorithm, a fifth version information digest algorithm, and an asymmetric algorithm.

[0016] With reference to the second possible implementation of the first aspect, in a fourth possible implementation of the first aspect, the cloud service platform is an Azure platform, and the method further comprises:

[0017] The encrypted data is pushed to the Azure platform in a web push manner by using a serial SCSI connection access manner provided by the Azure platform.

[0018] With reference to the first aspect, or any one of the first to fourth possible implementations of the first aspect, in a fifth possible implementation of the first aspect, the data settlement result returned by the distributed cluster according to the data obtained from the path information according to the computing task comprises:

[0019] The distributed cluster analyzes the cluster data computing task, obtains the stored path information and the computing task;

[0020] The salted and encrypted data is obtained from the stored path information;

[0021] According to the data provider, the distributed cluster is notified in advance of the data set obtained by the distributed cluster, and each data in the data set is salted and encrypted in the same salted encryption manner as the to-be-transmitted data, to obtain salted and encrypted pre-stored data;

[0022] For each salted and encrypted data, the same salted and encrypted pre-stored data as the salted and encrypted data is obtained, and the data corresponding to the salted and encrypted pre-stored data is extracted from the data set to obtain a data calculation set.

[0023] The distributed cluster calculates the data calculation set according to the cluster data calculation task, obtains a data settlement result, and transmits the data calculation result to the independent processing unit.

[0024] In combination with any one of the first aspect, the first possible implementation manner to the fourth possible implementation manner of the first aspect, the embodiment of the application provides a sixth possible implementation manner of the first aspect, and the method further includes:

[0025] Recording a unit operation log of operating the to-be-transmitted data;

[0026] Receiving a distributed operation log recorded by the distributed cluster and operating the cluster data calculation task;

[0027] Splicing the unit operation log and the distributed operation log, and storing the spliced log.

[0028] In the second aspect, the embodiment of the application further provides a data processing device, which includes:

[0029] The encryption data acquisition module is configured to receive storage location information of the cloud service platform sent by the data provider, and pull encrypted data according to the storage location information;

[0030] The data transmission module is configured to transmit the encrypted data to the distributed cluster for storage after decryption of the encrypted data;

[0031] The task submission module is configured to generate a cluster data calculation task request according to path information used for storage of the distributed cluster and a calculation task, and submit to the distributed cluster;

[0032] The settlement result acquisition module is configured to receive a data settlement result returned by the distributed cluster according to the calculation task and data obtained from the path information, transmit the data settlement result to the cloud service platform after encryption processing, and notify the data provider to pull the data settlement result.

[0033] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, and the processor implements the steps of the method when running the computer program.

[0034] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the computer program runs on a processor to perform the steps of the method.

[0035] The data processing method and device provided by the embodiment of the present application can ensure the transmission security of data in the public network and the security of data processing, and can improve the security of data.

[0036] In order to make the above objectives, features and advantages of the present application more apparent, the following will specifically describe a preferred embodiment in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments, and it should be understood that the following drawings only show some of the embodiments of the present application, and therefore should not be regarded as a limitation to the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0038] Figure 1 Fig. 1 shows a flowchart of the data processing method provided by the embodiment of the present application;

[0039] Figure 2 Fig. 2 shows a structure diagram of the data processing device provided by the embodiment of the present application;

[0040] Figure 3 Fig. 2 shows a structure diagram of the data processing device provided by the embodiment of the present application;

[0041] Figure 4A structural schematic diagram of a computer device 400 is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0042] To make the objects, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part but not all of the embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.

[0043] In the embodiments of the present application, data security is processed from aspects of data transmission, flow, processing, storage, auditing and the like, and a data secure transmission and processing method is provided, which can be applied to data processing under a large data level and business application scenarios such as distributed computing depending on a Hadoop bottom layer, and can meet comprehensive requirements in a data security specification.

[0044] The present application provides a data processing method and device, which will be described below by embodiments.

[0045] Figure 1 A data processing method flowchart provided by the embodiments of the present application is shown. As shown in Figure 1 The method comprises the following steps.

[0046] In step 101, the storage location information of a cloud service platform sent by a data provider is received, and encrypted data is pulled according to the storage location information.

[0047] In the embodiments of the present application, as an optional embodiment, before the storage location information of the cloud service platform sent by the data provider is received, the method further comprises the following steps.

[0048] The data provider performs salt encryption on the data to be transmitted, performs encryption compression on the salt-encrypted data, obtains compressed data, and transmits the compressed data to the cloud service platform for storage.

[0049] In the embodiment of the present application, the data provider performs salted encryption on the data to be transmitted, and performs encryption compression on the salted encrypted data in an encrypted compression manner to obtain encrypted data, which is transmitted to a cloud service platform, such as an Azure platform. After the Azure platform stores the encrypted data, the data provider is notified of the storage location information of the cloud service platform, and the data provider sends the storage location information of the cloud service platform to the data processing party. As an optional embodiment, the data to be transmitted is identity (ID) data and sensitive information data, wherein the sensitive information data includes but is not limited to marketing data, technical data, etc.

[0050] In the embodiment of the present application, as an optional embodiment, the data provider performs salted encryption on the data to be transmitted, including:

[0051] A11, performing salted processing on the data to be transmitted to obtain salted data;

[0052] In the embodiment of the present application, as an optional embodiment, the personal ID data and the sensitive information data are subjected to salted processing. The salted processing can be based on the agreement between the data provider and the data processing party, and a pre-set string is used for salted processing, and the string includes but is not limited to numbers, and / or letters, such as dataencrypted2021.

[0053] A12, performing encryption processing on the salted data according to a pre-set encryption algorithm to obtain salted encrypted data.

[0054] In the embodiment of the present application, the salted data after salted is encrypted, and as an optional embodiment, the encryption algorithm includes but is not limited to a message digest (MD) algorithm, a fifth version message digest (MD5) algorithm, and an asymmetric algorithm.

[0055] In the embodiment of the present application, as an optional embodiment, the data to be transmitted is shown in Table 1:

[0056] Table 1

[0057] Devices ID Timestamp 5D28291E47067A492CEB6691E43605CA 1636374544

[0058] After the string "dataencrypted2021" is used to perform salted processing on the data to be transmitted, the obtained salted data is:

[0059] 5D28291E47067A492CEB6691E43605CAdataencrypted2021.

[0060] The encryption algorithm uses the MD5 algorithm, and the salted data is encrypted by using the MD5 algorithm, and the salted encrypted data is:

[0061] MD5(‘5D28291E47067A492CEB6691E43605CAdataencrypted2021’).

[0062] The salted encrypted data is shown in Table 2:

[0063] Table 2

[0064] Devices ID Timestamp 7fb63455df6bfe50ba51d6d20047c8cf 1636374544

[0065] In the embodiment of the present application, the salted encrypted data is encrypted and compressed, so that the salted encrypted data (encrypted data) is transmitted in an encrypted and compressed manner. As an optional embodiment, the compression manner can be based on a prior agreement, including but not limited to zip, tar and other compression manners, and a compression password is set to ensure the security of data transmission. For example, the compression password is set to ‘zip2021’.

[0066] In the embodiment of the present application, after the salted encrypted data is encrypted and compressed, the encrypted and compressed data is transmitted to a cloud service platform for storage, for example, an Azure platform, so that the data processing party can pull the data to be processed from the cloud service platform according to the storage location information. Taking the Azure platform as an example, since the Azure platform uses a secure channel of https key encryption manner, the embodiment of the present application can use the secure channel of the platform for data transmission, without considering the security processing manner of the Azure platform itself. As an optional embodiment, the encrypted and compressed data (compressed data) is pushed to the Azure platform by using a web push manner by using a serial SCSI (SAS, Serial Attached SCSI) connection access manner provided by the Azure platform.

[0067] In the embodiment of the present application, as an optional embodiment, after receiving the storage location information of the cloud service platform sent by the data providing party, before pulling the encrypted data according to the storage location information, the method further comprises:

[0068] The data processing party sets an independent processing unit for processing the compressed data according to the received storage location information of the cloud service platform, so that the independent processing unit executes the pulling of the compressed data according to the storage location information from the cloud service platform.

[0069] In the embodiment of the present application, as an optional embodiment, the independent processing unit can be set as a virtual machine. By setting the virtual machine dedicated for the to-be-transmitted data, the virtual machine is enabled to perform operations of data pulling, data calculation task submission, data result processing and other services of the to-be-transmitted data, thereby realizing isolated processing with other data and improving the security of data processing. Further, the operations processed in the virtual machine can also be recorded in operation logs, so that the operations processed in the virtual machine are improved in security in the data processing process through detailed operation log recording.

[0070] In the embodiment of the present application, as an optional embodiment, the virtual machine can be deployed by scripting.

[0071] In step 102, after the encrypted data is decrypted, the decrypted data is transmitted to a distributed cluster for storage.

[0072] In the embodiment of the present application, as an optional embodiment, the independent processing unit decrypts and decompresses the compressed data (encrypted data) pulled from the cloud service platform, and then transmits the decrypted and decompressed data to a distributed cluster for storage. The storage path information is returned to the independent processing unit.

[0073] In the embodiment of the present application, the independent processing unit pulls compressed data from the Azure platform to the local end, performs local decryption and decompression, and then uploads the decrypted and decompressed data to a distributed cluster, such as a Hadoop cluster, for storage. As an optional embodiment, the decrypted and decompressed data is stored in a Hadoop Distributed FileSystem (HDFS) directory of the Hadoop cluster.

[0074] In the embodiment of the present application, the independent processing unit performs decompression operation by using a pre-agreed compression password. For example, if it is agreed to use zip compression and the compression password is zip2021, the corresponding decompression password zip2021 is input for decompression in the decompression process.

[0075] In the embodiment of the present application, the salted and encrypted data obtained after decompression is transmitted to a preset HDFS directory of the Hadoop cluster. As an optional embodiment, the preset HDFS directory (storage path) can be determined according to actual business, combined with business domain and processing time. For example, if the current time is 2021-11-09, the storage path information of the compressed data after decompression and decryption can be set as:

[0076] / user / security / IMP / 20211109 / original / xxx

[0077] Step 103, generating a cluster data computing task request according to the path information for storage of the distributed cluster and the computing task, and submitting to the distributed cluster;

[0078] In the embodiment of the application, the independent processing unit can generate a cluster data computing task request by combining the path information of the HDFS directory based on the business scenario to which the data belongs, and submit the request to the underlying Hadoop cluster to perform distributed computing by using the distributed resources of the Hadoop cluster.

[0079] In the embodiment of the application, the sensitive information data involved in the computing task processing logic, such as the ID field, is subjected to salted encryption processing. As an optional embodiment, the code segment corresponding to the computing task processing logic is as follows:

[0080] SELECT md5(upper(md5(upper(ifa)))+”dataencrypted2021”),md5(upper(him)+”dataencrypted2021”)

[0081] FROM table A

[0082] Step 104, receiving the data settlement result returned by the distributed cluster according to the computing task and the data obtained from the path information, performing encryption processing, and transmitting the result to the cloud service platform, and notifying the data provider to pull the data settlement result.

[0083] In the embodiment of the application, the independent processing unit receives the data settlement result.

[0084] In the embodiment of the application, as an optional embodiment, the data settlement result returned by the distributed cluster according to the computing task and the data obtained from the path information includes:

[0085] B11, the distributed cluster parses the cluster data computing task, obtains the stored path information and the computing task;

[0086] B12, obtaining the salted and encrypted data from the stored path information;

[0087] In the embodiment of the application, taking the encryption algorithm MD5 as an example, the salted and encrypted data is a digest value. Since the salted and encrypted data cannot be obtained by inverse deduction, even if it is intercepted during transmission, the sensitive information data cannot be obtained by cracking.

[0088] B13, according to the data provider, in the data set obtained by the distributed cluster, each data in the data set is salted and encrypted in the same salted encryption manner as the to-be-transmitted data, to obtain salted and encrypted pre-stored data;

[0089] In the embodiment of the application, the data provider pre-notifies the distributed cluster to capture data corresponding to the computing task according to the computing task, for example, the computing task is to count the data received and sent by an interface, and the distributed cluster captures the data received and sent by the interface according to the computing task. Wherein, the to-be-transmitted data is part of the data captured by the distributed cluster.

[0090] B14, for each salted and encrypted data, obtain the same salted and encrypted pre-stored data as the salted and encrypted data, extract the data corresponding to the salted and encrypted pre-stored data from the data set, and obtain a data calculation set;

[0091] In the embodiment of the application, the original data corresponding to the salted and encrypted data is obtained by salted encryption of the data corresponding to the computing task captured by the distributed cluster.

[0092] B15, the distributed cluster performs calculation on the data calculation set according to the cluster data calculation task, obtains a data settlement result, and transmits the data calculation result to the independent processing unit.

[0093] In the embodiment of the application, as an optional embodiment, the data calculation result obtained based on the cluster data calculation task processing is stored in HDFS path / result / xxx, and the obtained data calculation result is returned to the local virtual machine (independent processing unit), and the virtual machine performs encryption compression processing on the received data calculation result. Wherein, the compression adopts: zip, tar mode, and the encryption password is zip2021.

[0094] In the embodiment of the application, the virtual machine transmits the encryption compression package obtained by the encryption compression processing to the Azure platform, and notifies the data provider to pull the encryption compression package from the path corresponding to the cloud service platform.

[0095] In the embodiment of the application, as an optional embodiment, the method further comprises:

[0096] The independent processing unit records a unit operation log of operation on the to-be-transmitted data;

[0097] The distributed cluster records a distributed operation log of operation on the cluster data calculation task, and returns the distributed operation log to the independent processing unit;

[0098] The independent processing unit splices the unit operation log and the distributed operation log, and stores the spliced log.

[0099] In the embodiment of the present application, the operation performed on each service retains a corresponding operation log, which is used to verify the security of data processing.

[0100] The data processing method of the embodiment of the present application can ensure the security of data transmission in a public network and the security of data processing, can meet the security considerations of enterprises in data interaction and processing, and meets the requirements and purposes of security audit; further, by proposing a data encryption and file compression encryption processing method of a large data order of magnitude, a secure practice method of transmission and calculation of a local independent processing unit and a Hadoop cluster task is provided.

[0101] Figure 2 A schematic diagram of a data processing system structure provided by the embodiment of the present application is shown. Figure 2 As shown in the figure, the system comprises a data provider 201, a cloud service platform 202, a data processing party 203 and a distributed cluster 204, wherein,

[0102] The data provider 201 is configured to perform salt encryption on the to-be-transmitted data, perform encryption compression on the salt-encrypted data to obtain compressed data, transmit the compressed data to the cloud service platform 202 for storage, and notify the data processing party 203 of the storage location information of the cloud service platform 202.

[0103] In the embodiment of the present application, as an optional embodiment, the data provider performs salt encryption on the to-be-transmitted data, comprising:

[0104] The to-be-transmitted data is subjected to salt processing to obtain salted data.

[0105] The salted data is subjected to encryption processing according to a pre-set encryption algorithm to obtain salt-encrypted data.

[0106] In the embodiment of the present application, as an optional embodiment, the encryption algorithm comprises an information digest algorithm, a fifth version information digest algorithm and an asymmetric algorithm.

[0107] In the embodiment of the present application, as an optional embodiment, the cloud service platform is an Azure platform, and the transmission of the compressed data to the cloud service platform comprises:

[0108] The encrypted and compressed data is pushed to the Azure platform by using a web push mode through a serial SCSI connection access mode provided by the Azure platform.

[0109] The data processing party 203 is configured to set an independent processing unit for processing the compressed data according to the received storage location information of the cloud service platform 202, so that the independent processing unit pulls the compressed data from the cloud service platform 202.

[0110] The independent processing unit is configured to transmit the compressed data pulled from the cloud service platform 202 to the distributed cluster 204 for storage after decryption and decompression.

[0111] In the embodiment of the application, the encrypted and compressed password of the salted and encrypted data is decrypted.

[0112] The independent processing unit is configured to generate a cluster data calculation task request according to the path information for storage and the calculation task of the distributed cluster 204, and submit the cluster data calculation task request to the distributed cluster 204.

[0113] The independent processing unit is configured to receive the data settlement result returned by the distributed cluster 204 according to the cluster data calculation task, perform encrypted and compressed processing, and transmit the data settlement result to the cloud service platform 202, and notify the data provider 201 to pull the data settlement result from the cloud service platform 202.

[0114] In the embodiment of the application, as an optional embodiment, the data settlement result returned by the distributed cluster according to the cluster data calculation task includes:

[0115] The distributed cluster analyzes the cluster data calculation task to obtain the stored path information and the calculation task.

[0116] The salted and encrypted data is obtained from the stored path information.

[0117] According to the data set obtained by the distributed cluster according to the pre-notification of the data provider, each data in the data set is salted and encrypted in the same salted and encrypted manner as the to-be-transmitted data, to obtain the salted and encrypted pre-stored data.

[0118] For each salted and encrypted data, the salted and encrypted pre-stored data identical to the salted and encrypted data is obtained, and the data corresponding to the salted and encrypted pre-stored data is extracted from the data set, to obtain a data calculation set.

[0119] The distributed cluster calculates the data calculation set according to the cluster data calculation task, obtains a data calculation result, and transmits the data calculation result to the independent processing unit.

[0120] In an optional embodiment of the present application, the independent processing unit is further configured to record a unit operation log of operation on the data to be transmitted, the distributed cluster 204 is further configured to record a distribution operation log of operation on the cluster data calculation task and return the distribution operation log to the independent processing unit, and the independent processing unit is further configured to splice the unit operation log and the distribution operation log and store the spliced log.

[0121] Figure 3 A structure diagram of a data processing device provided by an embodiment of the present application is shown. Figure 3 As shown in the figure, the device comprises:

[0122] The encryption data acquisition module 301 is configured to receive storage location information of a cloud service platform sent by a data provider, and pull encryption data according to the storage location information.

[0123] In an optional embodiment of the present application, the encryption data acquisition module 301 is further configured to:

[0124] According to the received storage location information of the cloud service platform, an independent processing unit for processing the compressed data is set, so that the independent processing unit executes the operation of pulling the compressed data according to the storage location information from the cloud service platform.

[0125] The data transmission module 302 is configured to transmit the decrypted encryption data to a distributed cluster for storage.

[0126] In an optional embodiment of the present application, the decryption is decompression processing, the encryption data is decompressed by using a pre-agreed compression password, and the decompressed encryption data is transmitted to the distributed cluster for storage.

[0127] The task submission module 303 is configured to generate a cluster data calculation task request according to path information for storage of the distributed cluster and a calculation task, and submit the request to the distributed cluster.

[0128] In an embodiment of the present application, specific data calculation task processing logic is written based on a business scenario to which the data belongs, and a cluster data calculation task request is generated in combination with path information of an HDFS directory.

[0129] The settlement result acquisition module 304 is used to receive the data settlement result returned by the distributed cluster based on the data obtained from the path information according to the computing task, encrypt it and transmit it to the cloud service platform, and notify the data provider to pull the data settlement result.

[0130] In this embodiment of the invention, as an optional implementation, the distributed cluster parses the cluster data computation task request to obtain the stored path information and computation task; from the stored path information, it obtains salted and encrypted data; based on the data set obtained by the distributed cluster in advance notified by the data provider, it salts and encrypts each piece of data in the dataset using the same salting and encryption method as the data to be transmitted, to obtain salted and encrypted pre-stored data; for each piece of salted and encrypted data, it obtains the same salted and encrypted pre-stored data, extracts the data corresponding to the salted and encrypted pre-stored data from the dataset, to obtain the data computation set; and performs calculations on the data computation set to obtain the data settlement result.

[0131] In an optional embodiment of the present invention, the device further includes:

[0132] A log recording module (not shown in the figure) is used to record unit operation logs of operations performed on the data to be transmitted;

[0133] Receive the distributed operation logs recorded by the distributed cluster, which record operations performed on the cluster data computing tasks;

[0134] The unit operation log and the distribution operation log are spliced ​​together and stored.

[0135] like Figure 4 As shown, one embodiment of this application provides a computer device 400 for performing... Figure 1 The device includes a memory 401, a processor 402 connected to the memory 401 via a bus, and a computer program stored in the memory 401 and executable on the processor 402. The processor 402 executes the computer program to implement the steps of the data processing method.

[0136] Specifically, the memory 401 and processor 402 mentioned above can be general-purpose memory and processor, without any specific limitations. When the processor 402 runs the computer program stored in the memory 401, it can execute the above-mentioned data processing method.

[0137] Corresponding to Figure 1The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is run by a processor to execute the steps of the data processing method.

[0138] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk or the like, and the computer program stored on the storage medium can be run to execute the data processing method.

[0139] In the embodiments provided by the present application, it should be understood that the disclosed system and method can be implemented in other manners. The embodiments described above are only schematic and illustrative, and the division of the units is only a logical function division, and there can be another division manner in actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, and the indirect couplings or communication connections between the units can be in electrical, mechanical or other forms.

[0140] The units described as separated components can or can not be physically separated, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on a plurality of network units. In actual implementation, some or all of the units can be selected according to actual needs to achieve the purposes of the embodiments.

[0141] In addition, each functional unit in the embodiments provided by the present application can be integrated in a processing unit, or each unit can exist physically as a separate unit, or two or more units can be integrated in one unit.

[0142] The functions described above can be implemented in the form of software function units and sold or used as an independent product, which can be stored in a computer readable storage medium. Based on such an understanding, the technical solutions of the present application essentially or partially, or a part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various other media that can store program codes.

[0143] It should be noted that like reference numerals and letters refer to like items throughout the accompanying drawings, and once an item is defined in one drawing, it should not be further defined and explained in subsequent drawings, and further, the terms "first", "second", "third" and the like are used only to distinguish descriptions, and cannot be understood as indicating or implying relative importance.

[0144] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, and are used to illustrate the technical solutions of the present application, but are not limiting, and the protection scope of the present application is not limited thereto, although the present application is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some technical features, and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method of processing data, characterized by, The method comprises the following steps: receiving the storage location information of the cloud service platform sent by the data provider, and pulling the encrypted data according to the storage location information; after decrypting the encrypted data, transmitting the decrypted data to the distributed cluster for storage; generating a cluster data calculation task request according to the path information for storage and the calculation task of the distributed cluster, and submitting the cluster data calculation task request to the distributed cluster; receiving the data settlement result returned by the distributed cluster after performing calculation on the data obtained from the path information according to the calculation task, performing encryption processing on the data settlement result, and transmitting the data settlement result to the cloud service platform, and notifying the data provider to pull the data settlement result; the receiving the data settlement result returned by the distributed cluster after performing calculation on the data obtained from the path information according to the calculation task comprises: the distributed cluster analyzes the cluster data calculation task request, obtains the stored path information and the calculation task; from the stored path information, obtain the salted encrypted data; according to the data set obtained by the distributed cluster according to the pre-notification of the data provider, respectively salted and encrypted each data in the data set in the same salted and encrypted manner as the to-be-transmitted data, to obtain the salted and encrypted pre-stored data; for each salted and encrypted data, obtain the salted and encrypted pre-stored data same as the salted and encrypted data, extract the data corresponding to the salted and encrypted pre-stored data from the data set, to obtain a data calculation set; the distributed cluster performs calculation on the data calculation set according to the cluster data calculation task, and obtains the data settlement result.

2. The method of claim 1, wherein, After receiving the storage location information of the cloud service platform sent by the data provider, before pulling the encrypted data according to the storage location information, the method further comprises: the data processing party sets an independent processing unit for processing compressed data according to the received storage location information of the cloud service platform, so that the independent processing unit performs the step of pulling the compressed data from the cloud service platform according to the storage location information.

3. The method of claim 1, wherein, Before receiving the storage location information of the cloud service platform sent by the data provider, the method further comprises: salt processing the to-be-transmitted data to obtain salted data; encrypting the salted data according to a pre-set encryption algorithm to obtain encrypted data, and transmitting the encrypted data to the cloud service platform for storage.

4. The method of claim 3, wherein, The encryption algorithm comprises an information digest algorithm, a fifth version information digest algorithm, and an asymmetric algorithm.

5. The method of claim 3, wherein, The cloud service platform is an Azure platform, and the method further comprises: using the serial SCSI connection access mode provided by the Azure platform, and using the web push mode to push the encrypted data to the Azure platform.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: recording a unit operation log of the to-be-transmitted data; receiving a distributed operation log recorded by the distributed cluster for operating the cluster data calculation task; splicing the unit operation log and the distributed operation log, and storing the spliced log.

7. A data processing device, characterized by ​ The encrypted data acquisition module is configured to receive storage location information of a cloud service platform sent by a data provider, and pull encrypted data according to the storage location information. The data transmission module is configured to transmit the encrypted data to a distributed cluster for storage after decryption. The task submission module is configured to generate a cluster data calculation task request according to path information for storage and a calculation task of the distributed cluster, and submit the request to the distributed cluster. The settlement result acquisition module is configured to receive a data settlement result returned by the distributed cluster after calculation of data obtained from the path information according to the calculation task, transmit the result to the cloud service platform after encryption, and notify the data provider to pull the data settlement result. The receiving of the data settlement result returned by the distributed cluster after calculation of data obtained from the path information according to the calculation task includes: The distributed cluster analyzes the cluster data calculation task request, obtains the stored path information and the calculation task, obtains salted and encrypted data from the stored path information, and respectively salt encrypts each data in the data set according to the same salt encryption method as the data to be transmitted to obtain salted and encrypted pre-stored data. For each salted and encrypted data, the same salted and encrypted pre-stored data as the salted and encrypted data is obtained, the data corresponding to the salted and encrypted pre-stored data is extracted from the data set to obtain a data calculation set. The distributed cluster calculates the data calculation set according to the cluster data calculation task to obtain a data settlement result, and returns the data calculation result. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the data processing method according to any one of claims 1 to 6. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the data processing method according to any one of claims 1 to 6.

8. A computer device, comprising: ​ ​ 9. A computer-readable storage medium, characterized in that, ​

Images