Data processing system and method based on cloud edge collaboration and electronic equipment

Abstract

The embodiment of the invention provides a data processing system and method based on cloud edge collaboration and electronic equipment, the system comprises a cloud service center and a group of edge nodes, each edge node corresponds to a data node of a data provider, and data assets of each edge node provide a data source for the cloud service center in a racking mode; the cloud service center is used for notifying the edge node to which each piece of business data used by the computing business belongs to upload each piece of business data to the cloud service center; executing a calculation service by using a group of service data, and issuing a calculation result to the first edge node; the second edge node to which the target business data belongs is used for executing a target processing operation on the target business data and uploading the processed target business data to the cloud service center, and the target processing operation comprises at least one of a desensitization operation, an encryption operation and a feature extraction operation; and the first edge node is used for receiving the calculation result issued from the cloud service center.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of communications. Specifically, the embodiments of the present application relate to a data processing system, method, and electronic device based on cloud-edge collaboration. Background Art

[0002] Currently, most data application scenarios based on cloud-edge collaboration are data upload / download between edge nodes / devices within the same enterprise and the cloud data center. When cross-node applications are required, especially when the nodes belong to different enterprises, there are data security risks due to data transfer between different nodes.

[0003] It can be seen that in the data processing solution based on cloud-edge collaboration in the related art, there is a problem of poor data security caused by data transfer between different nodes. Summary of the Invention

[0004] The embodiments of the present application provide a data processing system, method, and electronic device based on cloud-edge collaboration to at least solve the technical problem of poor data security caused by data transfer between different nodes in the data processing solution based on cloud-edge collaboration in the related art.

[0005] According to one aspect of the embodiments of the present application, a data processing system based on cloud-edge collaboration is provided, including: a cloud service center and a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center in a warehousing manner. Among them, the cloud service center is configured to notify each edge node to which each service data in a group of service data used for a to-be-executed computing service belongs to upload each service data to the cloud service center; execute the computing service using the group of service data, and send the computing result of the computing service to a first edge node in the group of edge nodes, where the first edge node is a pre-specified edge node to which the computing result is sent; a second edge node to which target service data belongs is configured to perform a target processing operation on the target service data and upload the processed target service data to the cloud service center, where the target service data is any service data in the group of service data, and the target processing operation is a processing operation specified when the target service data is warehoused and executed when the target service data is out of the warehouse. The target processing operation includes at least one of the following: desensitization operation, encryption operation, and feature extraction operation; the first edge node is configured to receive the computing result sent from the cloud service center.

[0006] According to another aspect of the embodiments of the present application, there is also provided a data processing method based on cloud-edge collaboration, including: receiving a data upload notification sent by a cloud service center, and extracting target service data from the data assets of the current edge node, where the cloud service center belongs to a data processing system based on cloud-edge collaboration, the data processing system further includes a group of edge nodes, each edge node in the group of edge nodes corresponds to a data node of a data provider, the data assets of each edge node provide data sources for the cloud service center in a warehousing manner, the current edge node belongs to the group of edge nodes, and the target service data is one of a group of service data used for the computing services to be executed by the cloud service center; performing a target processing operation on the target service data, and uploading the processed target service data to the cloud service center, where the target processing operation is a processing operation specified when the target service data is warehoused and executed when the target service data is out of the warehouse, and the target processing operation includes at least one of the following: a desensitization operation, an encryption operation, and a feature extraction operation.

[0007] According to yet another aspect of the embodiments of the present application, there is also provided a data processing device based on cloud-edge collaboration, including: a first execution unit, configured to receive a data upload notification sent by a cloud service center, and extract target service data from the data assets of the current edge node, where the cloud service center belongs to a data processing system based on cloud-edge collaboration, the data processing system further includes a group of edge nodes, each edge node in the group of edge nodes corresponds to a data node of a data provider, the data assets of each edge node provide data sources for the cloud service center in a warehousing manner, the current edge node belongs to the group of edge nodes, and the target service data is one of a group of service data used for the computing services to be executed by the cloud service center; a second execution unit, configured to perform a target processing operation on the target service data, and upload the processed target service data to the cloud service center, where the target processing operation is a processing operation specified when the target service data is warehoused and executed when the target service data is out of the warehouse, and the target processing operation includes at least one of the following: a desensitization operation, an encryption operation, and a feature extraction operation.

[0008] According to yet another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, where the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0009] According to another aspect of the embodiments of the present application, there is provided a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the steps in any of the above method embodiments.

[0010] According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to execute the steps in any of the above method embodiments through the computer program.

[0011] Through the present application, a method is adopted in which different data providers provide data sources for the cloud service center by putting their data assets on the shelf, and the processing operations to be performed when the data assets are taken out of the warehouse are specified when the data assets are put on the shelf. The data processing system based on cloud-edge collaboration includes a cloud service center and a group of edge nodes. Each edge node corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center by being put on the shelf. Accordingly, the data circulation channels between the data nodes can be opened up; for the computing services to be executed, the cloud service center notifies the edge nodes to which the business data used in the computing service belongs to upload each business data to the cloud service center. For a set of uploaded business data, the cloud service center can use it to execute the computing service. In this way, the same computing service can use the business data provided by different data providers for computing. For each edge node to which the business data belongs, when the business data is taken out of the warehouse, the business data is first desensitized, encrypted, feature extracted, etc. based on the processing operations specified when the business data is put on the shelf, and then the processed business data is uploaded to the cloud service center. The computing results of the computing service can be sent to the designated edge node, and the designated edge node can be the edge node that needs to execute the computing service. Through the above method, the data of multiple data nodes can participate in data computing while ensuring data security, realizing data sharing and circulation, achieving the technical effect of improving the security of data circulation, and further solving the problem of poor data security in the data processing solution based on cloud-edge collaboration in the related art. Description of the Drawings

[0012] Figure 1 is a schematic structural diagram of an optional data processing system based on cloud-edge collaboration according to the embodiments of the present application;

[0013] Figure 2 is a schematic diagram of an optional data processing system based on cloud-edge collaboration according to the embodiments of the present application;

[0014] Figure 3 It is a data flow diagram of an optional data processing system based on cloud-edge collaboration according to an embodiment of the present application;

[0015] Figure 4 It is a schematic diagram of data flow in an optional data processing system based on cloud-edge collaboration according to an embodiment of the present application;

[0016] Figure 5 It is a schematic flow diagram of an optional data processing method based on cloud-edge collaboration according to an embodiment of the present application;

[0017] Figure 6 It is a flowchart of another optional data processing method based on cloud-edge collaboration according to an embodiment of the present application;

[0018] Figure 7 It is a structural block diagram of an optional data processing device based on cloud-edge collaboration according to an embodiment of the present application;

[0019] Figure 8 It is a computer system structural block diagram of an optional electronic device according to an embodiment of the present application. Detailed implementation manners

[0020] In order to enable those skilled in the art to better understand the solution of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present application.

[0021] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily need to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units does not necessarily need to be limited to those steps or units clearly listed, but may include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0022] According to one aspect of the embodiments of the present application, a data processing system based on cloud-edge collaboration is provided. Figure 1 It is a schematic structural diagram of an optional data processing system based on cloud-edge collaboration according to an embodiment of the present application, asFigure 1 As shown in Figure 1 , the above cloud-edge collaborative data processing system includes a cloud service center 101 and a group of edge nodes 102, where each edge node 102 in the group of edge nodes 102 corresponds to a data node of a data provider.

[0023] The cloud-edge collaborative data processing system in this embodiment can be applied to the communication field and is applicable to scenarios of data exchange and sharing between data nodes of different enterprises, providing a data processing solution for a data operation platform under the cloud-edge collaborative framework. Through the cloud-edge collaborative framework, data participation in data calculation by multiple data nodes (multiple different data providers) is realized while ensuring data security, achieving data circulation and sharing. The cloud-edge collaborative framework is an architecture that combines cloud computing and edge computing. In this architecture, edge computing devices (such as Internet of Things devices, mobile phones, intelligent sensors, etc.) are located closest to the data source or user and are responsible for processing data or tasks with high real-time requirements and sensitivity to latency. Correspondingly, edge nodes can be located at the network edge, close to the data source; while the cloud service center has powerful computing resources and storage capabilities and is responsible for processing tasks with high complexity, large data volume, and non-real-time nature. Thus, through the efficient collaborative work between cloud computing and edge computing, fast, secure processing and transmission of data can be achieved by reasonably allocating computing tasks. Currently, most cloud-edge collaborative data application scenarios are data upload / download between edge nodes / devices within the same enterprise and the cloud data center, and cross-node applications, especially scenarios where nodes belong to different enterprises are less common.

[0024] For the above cloud-edge collaborative framework, within an enterprise, cloud-edge collaborative data processing is usually carried out in a controllable network environment, and data security can be guaranteed by mechanisms such as enterprise-internal firewalls, security policies, and access control. However, when the application scenario expands to a cross-enterprise or cross-node environment, data needs to be transmitted and transferred between different networks and systems. On the one hand, data is easily intercepted or leaked during the transmission process. On the other hand, different edge nodes may adopt different data security measures, which may lead to inconsistencies and loopholes in data protection.

[0025] In the related technologies, the data asset sharing solutions are mainly based on data packages and API (Application Programming Interface) solutions. The main difficulty is that data sharing cannot be achieved without the original data being stored in the warehouse. In addition, the consideration of data security in cross-node applications is relatively insufficient. For example, most solutions do not provide dynamic protection mechanisms for data during transmission, such as real-time encryption, desensitization operations, etc., and do not consider the unified management and control of data when it flows between different nodes. There is a lack of a complete set of data protection processes, which leads to the fact that in actual application scenarios, especially in complex environments involving data nodes of multiple enterprises, data security is difficult to guarantee. For data providers other than enterprises (for example, schools, organizations, etc.), the above problems also exist when sharing data between data nodes of different data providers.

[0026] It can be seen that the data processing solutions based on cloud-edge collaboration in related technologies have the problem of poor data security due to the flow of data between different nodes. Therefore, a safer and more controllable data processing solution is needed to ensure the safe transmission and processing of data under the cloud-edge collaboration architecture.

[0027] In order to at least partially solve the above technical problems, in this embodiment, different data providers provide data sources for the cloud service center by putting their data assets on the shelves, and specify the processing operations to be performed when the data assets are out of the warehouse when the data assets are put on the shelves. The data processing system based on cloud-edge collaboration includes a cloud service center and a group of edge nodes, each edge node corresponds to a data node of a data provider, and the data assets of each edge node provide a data source for the cloud service center by putting them on the shelves. Accordingly, the channel for data flow between the data nodes can be opened; for the computing business to be executed, the cloud service center notifies the edge node to which each business data used by the computing business belongs to upload each business data to the cloud service center, and for the uploaded set of business data, the cloud service center can use it to execute the computing business, so that the same computing business can use the business data provided by different data providers for calculation, and for the edge node to which each business data belongs, when the business data is out of the warehouse, based on the processing operation specified when the business data is put on the shelves, the business data is first desensitized, encrypted, feature extracted, etc., through the above method, data of multiple data nodes can be realized to participate in data calculation while ensuring data security, and data sharing and circulation can be realized, thereby improving the security of data flow.

[0028] Taking the case where the data provider is an enterprise, based on the cloud-edge collaborative framework, the enterprise data nodes are used as edge nodes, and the data source is provided by listing the enterprise's internal data assets on the shelves. The data flow between the data nodes is opened up through the cloud service center, ensuring data security and realizing data sharing and computing.

[0029] The data processing system based on cloud-edge collaboration in this embodiment may include: a cloud service center and a group of edge nodes. The cloud service center may be a data computing platform, which can be used to extract the business data required to perform computing services from the edge nodes, and perform computing services based on the extracted business data to obtain the computing results of the computing services; each edge node corresponds to a data node of a data provider, which can be used to manage the data assets of the corresponding data provider, and based on the scheduling of the cloud service center, perform the specified processing operations on the business data to be scheduled, and then upload the processed business data. Here, the data provider may be the provider of the data source of the cloud service center, which may be an individual, an enterprise, a school or other organizational structure. In some examples of this embodiment, enterprises are used as examples for explanation.

[0030] Optionally, for each edge node, the data assets of each edge node provide a data source for the cloud service center by listing. When listing its data assets, it can specify the processing operations performed on the data assets when they are out of the warehouse. The specified processing operations may include but are not limited to at least one of the following: desensitization operation, encryption operation, and feature extraction operation. Among them, the desensitization operation refers to the processing of sensitive information so that it is not easy to be identified or associated with personal identity; the encryption operation refers to the conversion of raw data into a seemingly meaningless ciphertext through a certain algorithm, and only by mastering the corresponding key can it be decrypted and restored to the original data; the feature extraction operation refers to extracting representative features from the raw data for use in data analysis, machine learning and other tasks. The extracted features can represent the corresponding data, but cannot restore the corresponding data, and can also serve the purpose of data security.

[0031] Each edge node can perform a listing operation on its data assets. The listing operation can be performed by registering the description information of the data assets to the cloud service center to make it an available data source. In this way, the data assets of each edge node can become a schedulable data source for the cloud service center. When needed, these data assets can be uploaded to the cloud service center after corresponding conversion and protection according to the processing method set when listing, so as to provide the required data for computing services.

[0032] In this embodiment, when a computing service to be executed is involved, the cloud service center can notify each edge node to which each service data in a set of service data used by the computing service to be executed belongs to upload each service data to the cloud service center. The cloud service center can store computing service configuration information, which can be used to indicate the computing service to be executed and the edge node to which each piece of service data used by the computing service belongs. Then, through the communication connection established with the edge node, it can notify each edge node to which each service data belongs to upload each service data to the cloud service center, so as to ensure that the service data required for the computing service can be centralized in the cloud service center before the start of the computing task, preparing for the execution of the computing service.

[0033] Optionally, for each computing service, its data asset usage process can be defined. For example, the data required during the initialization of the application and the computing platform, as well as the computing results and nodes to be output, can be defined, and the data assets used and the output result format can be clarified. The definition of the data asset usage process can be completed by the administrator of the data processing system and configured or modified by the relevant personnel configuring the computing service as needed. This embodiment does not make any limitations in this regard.

[0034] For any piece of service data in a set of service data, that is, the target service data, the edge node to which it belongs is the second edge node. After receiving the data upload notification from the cloud service center, the second edge node can identify the service data required to be uploaded in the notification, that is, the target service data. The target service data can be any service data participating in the computing task, that is, any piece of service data in a set of service data. Here, the target service data is a certain data asset that has been put on the shelves by the second edge node.

[0035] The data assets put on the shelves by the second edge node can be stored in its local database, and the data assets stored in the local database are data sources. Before the target service data is taken out of the warehouse, the second edge node can perform a target processing operation on the target service data, that is, when the target service data is put on the shelves, the data provider designates a processing operation for the target service data based on considerations such as the characteristics of the data and the requirements of the computing service. When a computing service needs to use the target service data, the target service data can be extracted from the local database of the second edge node (that is, the data is taken out of the warehouse), and the second edge node can preprocess the target service data according to the predefined rules (that is, the target processing operation) when the target service data is put on the shelves. The target processing operation can include at least one of the following: desensitization operation, encryption operation, feature extraction operation. The desensitization operation, encryption operation, and feature extraction operation are similar to those in the foregoing embodiments and will not be elaborated here.

[0036] The cloud service center can obtain each piece of business data uploaded by the edge node to which each piece of business data belongs, so as to obtain a set of business data required for performing computing services. The cloud service center can use the set of business data to perform computing services, and the computing services can include, but are not limited to, at least one of the following: data analysis, model training, data prediction, data query, etc., and can also be other services related to data computing, which are not limited in this embodiment.

[0037] After the computing service is completed, the cloud service center can generate the computing result of the computing service. The generated computing result can be stored in the storage device where the execution is located. Optionally, the computing service can be configured by a certain edge node, and the obtained computing result can be sent to the edge node that configures the computing service, or to a specified one or some of the computing results. For the above scenarios, the cloud service center can send the computing result to the first edge node in a group of edge nodes, that is, the edge node that is pre-specified and to which the computing result needs to be sent, so as to ensure that the computing result can be accurately and securely transmitted to the desired recipient without being accessed by other unnecessary edge nodes. For the first edge node, it can receive the computing result sent from the cloud service center.

[0038] Optionally, the first edge node can be the edge node that uploads all or part of the business data in a set of business data, or can also be other edge nodes, and the number of the first edge nodes can be one or more. Taking the data provider as an enterprise as an example, in the application scenario of cross-enterprise data transmission, the cloud-edge collaborative data processing system can support cross-enterprise data collaboration as long as a data sharing agreement is established between two or more parties. Among them, the pre-specification of the edge node to which the computing result is sent can be based on the goal of the computing service, data requirements, or cooperation agreements between different enterprises, and can also be specified based on other requirements, which are not limited in this embodiment, as long as the directional transmission of the computing result can be ensured.

[0039] Through the data processing system based on cloud-edge collaboration provided by this application, the data processing system includes: a cloud service center and a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center by being put on the shelves. Among them, the cloud service center is used to notify each edge node to which each business data in a group of business data used for a to-be-executed computing service belongs to upload each business data to the cloud service center; execute the computing service using the group of business data, and send the computing result of the computing service to a first edge node in the group of edge nodes, where the first edge node is a pre-specified edge node to which the computing result is sent; the second edge node to which the target business data belongs is used to perform a target processing operation on the target business data and upload the processed target business data to the cloud service center, where the target business data is any one of the group of business data, and the target processing operation is a processing operation specified when the target business data is taken out of the warehouse and executed when the target business data is put on the shelves. The target processing operation includes at least one of the following: a desensitization operation, an encryption operation, and a feature extraction operation; the first edge node is used to receive the computing result sent from the cloud service center, which can solve the problem of poor data security caused by data transfer between different nodes in the related art's data processing solution based on cloud-edge collaboration, and improve the security of data transfer between different nodes.

[0040] In an exemplary embodiment, for each edge node, it can be used to put on the shelves the data assets it owns. The data assets of each edge node can include the field information of multiple data fields corresponding to the data assets of each edge node. The shelf information of the data assets of each edge node includes operation indication information for indicating the processing operation specified for the field information of each data field corresponding to the data assets of each edge node when the data assets of each edge node are taken out of the warehouse.

[0041] Each data asset can include the field information of one or more data fields. For example, taking the data provider as a hospital, its data assets can be the case information of patients, which can include the field information of multiple data fields such as "name", "blood pressure", "heart rate", "blood sugar", etc. Another example is that taking the data provider as an enterprise, its data assets can be the archived information of enterprise employees, which can include the field information of multiple data fields such as "name", "gender", "ID number", "mobile phone number", etc.

[0042] When uploading data assets, various fields of the data assets can be described. In addition to including conventional descriptions such as field names and data types, it can also include the processing actions when the data is exported. Correspondingly, the uploading information of the data assets of each edge node includes operation indication information for indicating the processing operations specified for the field information of each data field corresponding to the data assets of each edge node when the data assets of each edge node are exported. Based on this operation indication information, the processing operations that can be performed on the field information of all or part of the data fields can be carried out when the data is exported (that is, before the data is transmitted to the cloud service center for processing) to prevent the leakage of privacy and classified data involved in the data.

[0043] For example, as Figure 2 shown, the multiple data fields corresponding to a certain data asset can include Field A, Field B, Field C, and Field D. The specified processing operations can include de-identification operations (such as hashing operations, masking operations, etc.), encryption operations (such as homomorphic encryption, symmetric encryption, asymmetric encryption, state machine transformation, etc.), filtering / screening operations, etc. Among them, the processing operations pre-specified for each field can be respectively performed on the field information of each field, so as to respectively obtain the processed Field A', Field B', Field C', and Field D'.

[0044] It should be noted that for each uploaded data asset, the corresponding operation indication information can specify the processing operations to be performed on the field information of each data field of the data asset when the data asset is exported, or can indicate the processing operations to be performed on the field information of some data fields of the data asset when the data asset is exported. At this time, for non-critical data, no processing operations need to be specified to reduce the amount of operation indication information and improve the flexibility of data processing. In addition, specifying different processing operations for different fields can facilitate the data provider to customize the most suitable preprocessing strategy for different data fields according to the data characteristics and security requirements, as long as it can ensure that the key original data is not leaked after being exported.

[0045] For example, for a data asset involving identity information, which may include sensitive data (key data) such as name, ID number, mobile phone number, etc., corresponding processing operations can be specified. For example, a de-identification operation can be performed to remove sensitive information when the data is exported. For the field information of other data fields, only an encryption operation can be performed or no processing operation can be performed.

[0046] Through this embodiment, by specifying the processing operations to be performed on the field information in the data fields of the data asset when the data asset is exported during the uploading of the data asset, the flexibility of data transmission can be improved while ensuring that the key original data is not leaked after being exported.

[0047] In an exemplary embodiment, a second edge node is configured to perform a desensitization operation on the field information of a first field corresponding to target business data when the target processing operation includes a desensitization operation specified for the field information of the first field corresponding to the target business data; perform an encryption operation on the field information of a second field corresponding to the target business data when the target processing operation includes an encryption operation specified for the field information of the second field corresponding to the target business data; and perform a feature extraction operation on the field information of a third field corresponding to the target business data when the target processing operation includes a feature extraction operation specified for the field information of the third field corresponding to the target business data.

[0048] Similar to the foregoing embodiments, to ensure that critical raw data is not leaked after leaving the warehouse, when data assets are put on the shelves, each field of the data assets needs to be described, including conventional descriptions such as field names and data types, and should also include the processing actions when the data leaves the warehouse, which may include, but are not limited to: performing a desensitization operation on critical data, such as hashing, masking, partitioning, etc.; encryption operations, such as homomorphic encryption; data feature extraction, so that only the distribution characteristics of the data can be returned, etc.

[0049] Here, in a cloud-edge collaboration architecture, data faces security risks during transmission. In particular, sensitive information (such as personal identity information) may be leaked. By performing a desensitization operation on critical data, sensitive information can be deformed or replaced without changing the data's logical structure, making it impossible to be directly identified or associated with a specific individual, thereby achieving the purpose of protecting privacy.

[0050] Through the desensitization operation, unauthorized access and use of sensitive data can be prevented, thereby protecting data security. For target business data, when the target processing operation includes a desensitization operation specified for the field information of a first field corresponding to the target business data, the second edge node can perform a desensitization operation on the field information of the first field corresponding to the target business data. Optionally, the desensitization operation may include at least one of the following: a hashing operation, which may be to perform a hashing conversion on each value of the sensitive field to generate a fixed-length digest value to ensure that even if the data is intercepted, the original value cannot be reverse-inferred; a masking operation, which may be to hide sensitive information by replacing some or all of the characters in the sensitive field (for example, replacing with asterisks "*" or random characters); a partitioning operation, which may be to classify the values of the sensitive field according to a preset interval and only transmit the classified tags instead of the exact values to protect specific numerical information.

[0051] Through encryption operations, unauthorized access and use of critical data can be prevented, thus protecting data security. For the target business data, when the target processing operation includes an encryption operation specified for the field information of the second field corresponding to the target business data, the second edge node can perform the encryption operation on the field information of the second field corresponding to the target business data. Encryption operation is a means of protecting information security. It converts the original data (plaintext) into an encrypted form through a specific algorithm to protect the security of the data during transmission or storage.

[0052] Optionally, the encryption operation may include a homomorphic encryption operation. Homomorphic encryption is an encryption technology that allows computations to be performed on encrypted data without prior decryption. By performing a homomorphic encryption operation on the data, the cloud service center can still perform computations on the data, and the encrypted form of the computation result still maintains the property of homomorphic encryption, that is, after decrypting the computation result, the result can reflect the computations performed on the original data. Since both the encryption and decryption operations can be performed at the edge node, the cloud service center does not obtain, process, or store plaintext data, effectively protecting the security of the data and avoiding the plaintext state of the data when it is processed at the cloud service center, thus reducing the risk of data leakage. In addition, the encryption operation may also include symmetric encryption operations, asymmetric encryption operations, etc., and the type of encryption operation can be specified according to business requirements, which is not limited in this embodiment.

[0053] Here, before the data is exported, operations such as desensitization and encryption are performed, and the data provider can process it according to its own needs. For example, ID cards and mobile phone numbers are generally processed through hash to ensure that the data is within a safe and controllable range. Based on the edge data node as the data source, the edge data is incorporated into the data application / computation framework, broadening the forms and methods of data sharing and circulation.

[0054] Through the feature extraction operation, the distribution features of the data can be extracted. The extracted distribution features can characterize certain characteristics of the data, and can meet the business calculation requirements without disclosing the original data. The extracted distribution features can be the characteristics of the data value distribution. For example, the central tendency of the data (such as the mean, median, etc.), the degree of dispersion (such as the standard deviation, variance, etc.), the distribution form (such as the normal distribution, skewed distribution, etc.) or other features. For the target business data, in the case where the target processing operation includes a feature extraction operation specified for the field information of the third field corresponding to the target business data, the second edge node can perform the feature extraction operation on the field information of the third field corresponding to the target business data. Here, in the data processing scenario of cloud-edge collaboration, uploading all the original data to the cloud service center will not only consume a large amount of network resources, but also may upload unnecessary detailed information, increasing the burden of data cleaning and preprocessing. In addition, although some data does not contain sensitive information, its specific value may not be required for the calculation business. Therefore, extracting and uploading meaningful statistical features or distribution features can not only reduce the consumption of network resources, but also reduce the burden of data cleaning and preprocessing.

[0055] Through this embodiment, by performing the specified processing operation for the field information of each field of the data field corresponding to the business data, where the performed desensitization operation can be a hash operation, a masking operation, a partitioning operation, etc., the specified encryption operation can be a homomorphic encryption operation, and the performed feature extraction operation can be an operation for extracting the distribution features of the data, the security of data transmission can be improved and data privacy can be protected.

[0056] In an exemplary embodiment, each edge node can be deployed with a local database (data node database), and the data assets of each edge node can be stored in the local database of each edge node. For an edge node, each data asset can correspond to a data source stored in the local database. Among them, the local database can be an SQL (Structured Query Language) database, or other relational databases or other types of databases, which are not limited in this embodiment.

[0057] After receiving a call, the edge node can perform data extraction and data serialization / deserialization according to the data asset definition: when the database table is processed and uploaded to the data storage service, it is in the form of a data packet. The data asset can be stored in the local database in the form of a database table. After extracting the data asset from the local database, the extracted data asset is in the form of a database table. After performing the specified processing operation on the extracted data asset, in order to facilitate data transmission and the execution of computing services, the processed data asset can be converted into the form of a specified data object, and the converted data object can be serialized. The serialized data object can be uploaded to the cloud service center in the form of a data packet. Here, the data object can be related to the form of data objects that the cloud service center and the edge node can process. In this embodiment, a data frame can be used as the data object to facilitate conversion between multiple platforms.

[0058] Correspondingly, the second edge node is used to extract the target service data from the local database of the second edge node, where the extracted target service data is in the form of a database table; perform a target processing operation on the target service data, and convert the processed target service data into a data frame format; serialize the converted target service data, and upload the serialized target service data to the cloud service center in the form of a data packet.

[0059] In this embodiment, the target service data can be stored in the local database of the second edge node in the form of a database table. After extracting the target service data from the local database of the second edge node, the extracted target service data can be in the form of a database table. The second edge node can perform a target processing operation on the extracted target service data and convert the processed target service data into a data frame format.

[0060] It should be noted that when the cloud service center in the foregoing embodiment performs a computing service, it can convert the service data into a data object form that the computing platform can process. At this time, a set of standardized operations can be used to implement the conversion between the data file and the data object. In this embodiment, a data frame can be used as the data object to provide conversion methods for multiple platforms. The data is converted and processed when exported to obtain a unified data object for data applications to participate in data computing in the cloud service center. Correspondingly, before uploading to the cloud service center, the second edge node can convert the processed target service data into a data frame format, thereby ensuring data consistency and processability. In addition, splitting the database table and converting it into a data frame format can also improve the efficiency of data transmission.

[0061] The second edge node can also serialize the converted target service data to convert the data frame format of the data into a selected serialization format. Here, serialization is a process of converting a data structure or object into a sequence (e.g., a byte stream) that can be stored or transmitted. In the cloud-edge collaboration scenario, serializing the processed target service data can be converting the target service data converted into the data frame format into a linear sequence that can be transmitted across the network. That is, the structure and content in the data frame can be encoded so that when the data is needed, the serialized data can be reconverted back to the original data structure or object state.

[0062] Optionally, to improve the security and convenience of data transmission, the second edge node can encapsulate the serialized target service data in the form of a data packet. After the encapsulation is completed, the data packet is sent to the cloud service center for storage. When uploading the target service data in the form of a data packet, the second edge node can compress, encode, etc. the serialized target service data. The compression algorithm, encoding method, etc. used can be selected as needed, and this is not limited in this embodiment.

[0063] Optionally, after the service data is uploaded, the cloud service center can obtain the data packet uploaded by the edge node. After deserialization, it is converted into a data object form that the cloud service center can process, e.g., the data frame format. For the target service data, its corresponding data packet is the target data packet. The cloud service center can obtain the target data packet uploaded by the second edge node. After deserializing the target data packet, it is converted into the data frame format that the cloud service center can process, that is, the target service data converted into the data frame format. Correspondingly, the calculation result obtained by the specified calculation service can be in the data frame format. To facilitate the distribution of the calculation result, the cloud service center can serialize the calculation result in the data frame format and store the serialized calculation result (data file) in the form of a data packet.

[0064] Through this embodiment, by performing specified processing operations on the extracted service data, converting the processed service data into the data frame format, serializing the converted service data, and uploading the serialized service data to the cloud service center in the form of a data packet, the problem of data format incompatibility between different data nodes can be solved, the data consistency can be ensured, and the security and convenience of data transmission can also be improved.

[0065] In an exemplary embodiment, in order to facilitate the delivery of calculation results, the calculation results can be stored in the cloud service center in the form of data packets. The process of storing the calculation results in the cloud service center is similar to that in the aforementioned embodiment and is not described in detail here. Correspondingly, the calculation results can be delivered to the first edge node in the form of data packets. Transmitting data in the form of data packets can improve the security and convenience of data transmission. In addition to the calculation results, the data packet can also include other metadata, such as the result type, encryption method, etc., so that the receiving end that obtains the calculation results can quickly understand and process the calculation results.

[0066] For the first edge node, it can receive the calculation results stored in the form of data packets sent from the cloud service center; deserialize the calculation results, and the deserialized calculation results are in the data frame format. Here, the deserialization process can correspond to the serialization process of the above-mentioned data packets, that is, it is the reverse operation of the serialization process of the above-mentioned data packets, so that the calculation results in the form of data packets generated by the cloud service center through the serialization process can be restored to the calculation results in the data frame format; import the deserialized calculation results into the database table, and store the database table of the imported calculation results into the local database of the first edge node. By storing the database table of the imported calculation results in the local database, persistent storage of data can be achieved to ensure that the calculation results will not be lost.

[0067] Optionally, the local database of the first edge node may have a predefined database table structure to accommodate the calculation results, and the number, type and name of the fields in the database table may correspond to the data frame format of the calculation results to ensure that the data can be accurately imported. The edge node may be configured with a monitoring mechanism for data import, and in the event of errors in the import of calculation results, the calculation results may be re-imported, or the calculation results may be re-obtained from the cloud service center and re-imported according to steps similar to the above.

[0068] Through this embodiment, by deserializing the calculation results sent down, importing the calculation results in the data frame format obtained by deserialization into the database table and storing the database table after the calculation results are imported into the local database, the safe sending and storage of the calculation results can be ensured, thereby improving the security and reliability of data transmission.

[0069] In an exemplary embodiment, a cloud service center includes: a computing center, a scheduling center and a data storage system; wherein the scheduling center is used to notify the edge node to which each business data belongs to upload each business data to the data storage system; and when a group of business data has been uploaded, the computing center is notified to pull a group of business data; based on the notification of the completion of the calculation by the computing center, the calculation result stored in the data storage system is sent to the first edge node; the computing center is used to read a group of business data from the data storage system based on the notification of the completion of the data upload by the scheduling center; use a group of business data to perform computing business, store the calculation results in the data storage system, and notify the scheduling center to send the calculation results; the data storage system is used to store each business data uploaded by the edge node to which each business data belongs; and store the calculation results stored by the computing center.

[0070] As the core component of the data processing system based on cloud-edge collaboration, the cloud service platform is responsible for data processing, scheduling, and storage. In order to facilitate the execution of computing services and the storage of business data, the cloud service platform can be divided into different parts, namely, the scheduling center, the computing center, and the data storage system.

[0071] The dispatch center is responsible for importing and exporting data from data nodes during the computing process, and participates in the computing of the computing center to control the data flow process during the data computing process, ensuring that all data is available when the computing starts, and returns to the designated data node according to the established process. The aforementioned data flow between data nodes can be opened up by the dispatch center of the cloud service platform;

[0072] The computing center, as a data computing platform / center, is responsible for processing and computing the data exported by the data nodes, outputting the calculation results, and notifying the scheduler to push the data to the designated data nodes. The computing center can process the input and output data to ensure data security and privacy protection;

[0073] The data storage system (providing cloud storage services) is a storage component that stores data uploaded by data nodes and intermediate data of computing center results. The intermediate data can be processed accordingly (to ensure that the data is not cracked by a third party) and necessary data desensitization work can be done (to ensure data security in the cloud).

[0074] For the data export process, the scheduling center can, according to business needs, notify the data nodes participating in the calculation to export business data to the cloud storage platform according to business requirements, that is, upload it to the data storage system. During the data export process, the data is desensitized, encrypted, and serialized according to the definition of the data source by the data provider (i.e., the data supplier), and after compression, it is uploaded to the cloud storage service (data storage system). Among them, the data processing is a one-to-one conversion for data fields. For the above calculation services, the scheduling center can notify the edge node to which each business data belongs to upload each business data to the data storage system.

[0075] For the data calculation process, when the data nodes complete data upload, that is, after completing data preparation, the scheduling node can notify the calculation nodes to pull the data and participate in the calculation process of the calculation center; after the data is calculated, it is processed and compressed and then stored in the cloud storage service. For the above calculation services, when a set of business data has been uploaded, the scheduling center notifies the calculation center to pull a set of business data; the calculation center reads a set of business data from the data storage system based on the data upload notification from the scheduling center; uses a set of business data to execute the calculation service, and stores the calculation results in the data storage system.

[0076] For the data import process, after the calculation center is completed, it will notify the scheduling center to initiate the distribution of the calculation results, transmit the calculation results to the consumer data nodes (i.e., the data nodes that consume the calculation results), and after decompression, store them in the consumer data node database. For the above calculation services, the calculation center notifies the scheduling center to distribute the calculation results; the scheduling center, based on the notification of the completion of the calculation by the calculation center, distributes the calculation results stored in the data storage system to the first edge node.

[0077] Optionally, for different calculation services, the calculation center can pre-configure algorithms corresponding to different calculation services to ensure the correct execution of the corresponding calculation services. When distributing the calculation results, the scheduling center can directly control the data storage system to distribute the calculation results to the first edge node, or notify the first edge node to download the calculation results; the first edge node responds to the received notification to request the data storage system to distribute the calculation results, and the data storage system distributes the calculation results in response to the received request. The distribution method of the calculation results can be configured according to needs, and this is not limited in this embodiment.

[0078] For example, as Figure 3As shown, the scheduling center can notify the data nodes to upload data. Correspondingly, the data nodes extract the required data from the data sources of the local database and store it in the cloud storage service. After all the required data has been uploaded, the scheduling center can notify the computing center to extract the required data from the cloud storage service and execute the computing task. The computing results are serialized, compressed, and then stored in the cloud storage service. The cloud storage service can send the computing results to the specified data nodes and store them in the corresponding local databases. The way to send the computing results can be: synchronize the data packets to the edge data nodes through the cloud-edge collaboration framework, process the data according to the definition of the data results, and store it in the local database of the data nodes.

[0079] Through this embodiment, the cloud service center includes a scheduling center, a computing center, and a data storage system. Through the collaborative work of the scheduling center, the computing center, and the data storage system, the security and reliability of data transmission can be improved, and the convenience and efficiency of the execution of computing services can be enhanced.

[0080] In an exemplary embodiment, to improve the convenience of storing business data, the business data is uploaded in the form of data packets. Each uploaded business data is stored in the data storage system of the cloud service center in the form of data packets. Here, the data packet format can support compression and encryption, is suitable for network transmission, can reduce the bandwidth consumption of data transmission, and improve the security of data. The process of storing the above into the data storage system can be controlled and executed by the scheduling center, or the edge nodes can directly upload the business data to the data storage system. The data storage system can synchronize the upload progress data of the business data to the scheduling center to facilitate the scheduling center to determine the timing to notify the computing center that the business data upload is completed.

[0081] Correspondingly, the computing center is also used to deserialize each read business data to obtain each deserialized business data; use each deserialized business data to execute a computing service to obtain computing results; serialize the computing results and store the serialized computing results in the data storage system in the form of data packets.

[0082] Similar to the foregoing embodiment, when the data upload is completed, the application / computing center reads the uploaded data packets according to the configuration. After deserialization, it can be converted into a data object form that can be processed by the computing platform. At this time, a set of standardized operations can be adopted to realize the conversion between the data file and the data object. For each business data, the computing center can deserialize each read business data and convert it into a data frame format. Here, by uniformly converting the business data from different data sources into the data frame format, it is convenient for the computing center to uniformly process and analyze it, thereby realizing the convenience of data processing.

[0083] After deserializing each piece of business data, the computing center can use each piece of deserialized business data to perform computing services and obtain computing results. Correspondingly, the computing results are in the format of a data frame. To facilitate the storage and distribution of the computing results, the computing results can be serialized and the serialized computing results can be stored in the data storage system in the form of data packets, thereby reducing storage costs and bandwidth consumption during data transmission.

[0084] For example, for the data computing process, after the data is prepared, the scheduling center notifies the computing center to pull the data, decompress it, convert it into the data frame format, and participate in the computing of the computing center. After the data is computed, it is processed and compressed and then stored in the cloud storage service.

[0085] Through this embodiment, the computing center deserializes the business data to perform computing services using the business data in the data frame format, serializes the computing results, and stores them in the data storage system in the form of data packets. This can improve the efficiency and security of data transmission and reduce data transmission consumption while ensuring the correct execution of computing services.

[0086] In an exemplary embodiment, to facilitate data upload and storage, each edge node may be deployed with a data provider agent and a data consumer agent. Among them, the data provider agent of each edge node is used to perform the operations of listing (i.e., registering the data asset description of the edge node in the cloud service center), extracting, and uploading the data assets of each edge node, and the data consumer agent of each edge node is used to receive the data sent to each edge node.

[0087] In this embodiment, a data node agent can be deployed on the edge node, which is mainly responsible for operations such as registration, extraction, and writing of edge-side data. From the perspective of roles, it is divided into a data provider and a data consumer. The data provider agent provides the registration of data assets and securely exports the data to the cloud storage service; the data consumer agent is responsible for securely exporting the local data to the cloud storage service, and after the computing is completed by the computing center, it returns to the consumer agent (i.e., the data consumer agent) and writes it into the consumer database; the data can be desensitized and encrypted before data extraction and writing to ensure data security.

[0088] For the aforementioned computing service, the data provider agent of the second edge node can extract the target business data from the data assets of the second edge node, perform target processing operations on the extracted target business data, and upload the processed target business data to the cloud service center; while the data consumer agent of the first edge node can receive the computing results sent by the cloud service center and store the computing results in the local database of the first edge node.

[0089] For the scenario where the aforementioned cloud service center includes a computing center, a scheduling center, and a data storage system, the data supply - side agent of the second edge node can, according to the notice issued by the aforementioned scheduling center, extract the required target business data from the local database, perform target processing operations similar to those in the foregoing embodiments, such as desensitization operations, encryption operations, feature extraction operations, etc., and upload the processed target business data to the data storage system. The data consumption - side agent of the first edge node can be used to obtain the calculation result from the data storage system and store the calculation result in the local database of the first edge node.

[0090] Through this embodiment, by deploying the data supply - side agent and the data consumption - side agent, the processing process of data assets can be realized, the standardization of data transmission operations can be achieved, and the security and efficiency of data processing can be improved.

[0091] The following explains the cloud - edge collaborative data processing system in this embodiment with reference to optional examples. The application scenario of this optional example is data exchange, sharing, and calculation between different enterprise nodes, which can handle the problems of data - secure access and utilization (data security and generality) when sharing data between multiple enterprise data nodes. In this optional example, by incorporating enterprise data nodes as edge nodes into management, different data nodes manage the data of their own nodes, set the data protection methods involved in the export process when the data participates in calculations (data desensitization, data encryption processing, to solve data security problems), and when the data is exported, through conversion and processing, a unified data serialization method is used, and finally it is converted into a unified data object (data frame) for use by the computing framework, that is, for data applications to participate in data calculations in the computing center or edge nodes.

[0092] This optional example provides a set of general cross - data - node data extraction and standardization processes, directly using edge data as the data source for calculations; at the same time, it also provides a mechanism for fine - grained data processing, providing multiple data desensitization and encryption methods for data columns to achieve controllable data security. Here, the relevant personnel of the data provider can have data governance capabilities, describe the processing methods for each field of the data to be exported before the data is put on the shelves, and perform corresponding processing during export to prevent the leakage of privacy - related and confidential data in the data; at the same time, the data processing methods can cover common data desensitization and processing methods, reducing in - compatible scenarios.

[0093] Figure 4 is a schematic diagram of data flow in an optional cloud - edge collaborative data processing system according to an embodiment of the present application, as Figure 4As shown, the edge node can extract the required business data in the form of database tables from the local database. After data processing, the business data is converted into a data frame format, serialized and compressed into a data file and uploaded to the cloud service center; when the computing center of the cloud service center executes computing business, it can deserialize the data file into a data frame format, and execute business calculations through business processing logic; after the computing center outputs the calculation results, it can serialize and convert the calculation results into data packets, and send them to the edge node. The edge node deserializes the calculation results in the form of the sent data packets into a data frame format, imports the deserialized calculation results into the database table, and stores the database table of the imported calculation results in the local database.

[0094] like Figure 4 As shown, the required business data can be extracted from the local database of an edge node, the computing business can be executed and the computing result can be returned to the edge node, or the computing result can be returned to another edge node.

[0095] Through this optional example, the data assets registered with the enterprise data nodes are managed, each data field attribute of the assets is described, and the processing method of each field attribute is set to ensure that the outbound data assets are controlled in accordance with the specifications and requirements of the enterprise data management, thereby maximizing the control of data security during data outflow; at the same time, cloud-edge collaborative technology is used to open up communication between cloud and edge, and edge and edge, to achieve secure data transmission between cloud and edge.

[0096] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0097] According to another aspect of an embodiment of the present application, a data processing method based on cloud-edge collaboration is also provided. The data processing method based on cloud-edge collaboration can be applied to the above-mentioned system embodiments and optional implementation methods, and what has been explained will not be repeated here.

[0098] The data processing method based on cloud-edge collaboration in the embodiment of the present application can be applied to the edge node that provides business data in the data processing system based on cloud-edge collaboration in the aforementioned embodiment. Figure 5 is a flow chart of an optional data processing method based on cloud-edge collaboration according to an embodiment of the present application, such as Figure 5 As shown, the process of the above-mentioned data processing method based on cloud-edge collaboration may include the following steps:

[0099] Step S502: Receive the data upload notification sent by the cloud service center, and extract the target business data from the data assets of the current edge node. Here, the cloud service center belongs to a data processing system based on cloud-edge collaboration. The data processing system further includes a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center through the shelving method. The current edge node belongs to the group of edge nodes. The target business data is one of a group of business data used for the computing services to be executed by the cloud service center.

[0100] Step S504: Perform a target processing operation on the target business data, and upload the processed target business data to the cloud service center. Here, the target processing operation is the processing operation specified when the target business data is shelved and executed when the target business data is taken out of storage. The target processing operation includes at least one of the following: desensitization operation, encryption operation, and feature extraction operation.

[0101] In this embodiment, steps S502 to S504 may be executed by the second edge node in the foregoing embodiment. The manner in which the second edge node executes steps S502 to S504 is similar to that in the foregoing embodiment and has been described, so details are not repeated here.

[0102] Through the embodiments of the present application, by receiving the data upload notification sent by the cloud service center and extracting the target business data from the data assets of the current edge node. Here, the cloud service center belongs to a data processing system based on cloud-edge collaboration. The data processing system further includes a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center through the shelving method. The current edge node belongs to the group of edge nodes. The target business data is one of a group of business data used for the computing services to be executed by the cloud service center; performing a target processing operation on the target business data and uploading the processed target business data to the cloud service center. Here, the target processing operation is the processing operation specified when the target business data is shelved and executed when the target business data is taken out of storage. The target processing operation includes at least one of the following: desensitization operation, encryption operation, and feature extraction operation, the problem in the related art that the data processing solution based on cloud-edge collaboration has poor data security due to data transfer between different nodes is solved, and the security and reliability of data transmission between different nodes are improved.

[0103] In an exemplary embodiment, the above method further includes: shelving the data assets of the current edge node, where the data assets of the current edge node include field information of multiple data fields corresponding to the data assets of the current edge node, and the shelving information of the data assets of the current edge node includes operation indication information for indicating a processing operation specified for the field information of each data field corresponding to the data assets of the current edge node when the data assets of the current edge node are taken out of storage.

[0104] In an exemplary embodiment, performing a target processing operation on target service data includes: when the target processing operation includes a desensitization operation specified for the field information of a first field corresponding to the target service data, performing a desensitization operation on the field information of the first field corresponding to the target service data, where the desensitization operation includes at least one of the following: a hashing operation, a masking operation, and a partitioning operation; when the target processing operation includes an encryption operation specified for the field information of a second field corresponding to the target service data, performing an encryption operation on the field information of the second field corresponding to the target service data, where the encryption operation includes a homomorphic encryption operation; when the target processing operation includes a feature extraction operation specified for the field information of a third field corresponding to the target service data, performing a feature extraction operation on the field information of the third field corresponding to the target service data, where the feature extraction operation is used to extract the distribution features of the data.

[0105] In an exemplary embodiment, the data assets of the current edge node are stored in the local database of each edge node. Correspondingly, performing a target processing operation on target service data and uploading the processed target service data to the cloud service center includes: extracting the target service data from the local database of the current edge node, where the extracted target service data is in the form of a database table; performing a target processing operation on the target service data and converting the processed target service data into a data frame format; serializing the converted target service data and uploading the serialized target service data to the cloud service center in the form of a data packet.

[0106] In an exemplary embodiment, a data supply - side agent and a data consumption - side agent are deployed on the current edge node, where the data supply - side agent of the current edge node is used to perform the shelving operation, extraction operation, and upload operation of the data assets of the current edge node, and the data consumption - side agent of the current edge node is used to receive the data sent to the current edge node. Correspondingly, performing a target processing operation on target service data and uploading the processed target service data to the cloud service center includes: extracting the target service data from the data assets of the current edge node through the data supply - side agent of the current edge node, performing a target processing operation on the extracted target service data, and uploading the processed target service data to the cloud service center.

[0107] The data processing method based on cloud-edge collaboration in the embodiments of the present application will be explained below with reference to optional examples. In this optional example, a data processing method based on cloud-edge collaboration for data sharing between enterprises is provided. As Figure 6 shown, the process of this method may include the following steps:

[0108] Step 1, the edge data node completes data asset registration and annotates the processing method of the data asset table fields;

[0109] Step 2, the business defines the data asset usage process as needed, clarifying the data assets used and the output result format;

[0110] Step 3, the scheduling center notifies the edge data node to complete the extraction, serialization, compression of the data assets and upload them to the cloud storage service;

[0111] Step 4, the scheduling center notifies the execution of the computing task, and the task execution result is serialized, compressed and uploaded to the cloud storage service;

[0112] Step 5, the scheduling center notifies the edge data node to retrieve data from the cloud storage service and store it in the edge database.

[0113] Through this optional example, based on the cloud-edge collaboration framework, the enterprise data node is used as the edge node, and the data source is provided through the method of putting enterprise internal data assets on the shelves. The scheduling center is used to open up the data circulation between each data node. On the basis of ensuring data security, data sharing and computing are realized.

[0114] Through the description of the above embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software plus a necessary general hardware platform. Of course, it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, in essence, or the part that contributes to the prior art can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disc), and includes several instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in various embodiments of the present application.

[0115] According to another aspect of the embodiments of the present application, a data processing device based on cloud-edge collaboration is further provided. This device is used to implement the data processing method based on cloud-edge collaboration provided in the above embodiments, and those already described will not be repeated. As used hereinafter, the term "module" may be a combination of software and / or hardware that can achieve a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, implementation in hardware, or a combination of software and hardware is also possible and contemplated.

[0116] Figure 7 is a structural block diagram of an optional data processing device based on cloud-edge collaboration according to an embodiment of the present application. As Figure 7 shown, the device includes:

[0117] A first execution unit 702, configured to receive a data upload notification sent by a cloud service center, and extract target service data from the data assets of the current edge node. Among them, the cloud service center belongs to a data processing system based on cloud-edge collaboration. The data processing system further includes a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center in a shelving manner. The current edge node belongs to the group of edge nodes, and the target service data is one of a group of service data used for the computing services to be executed by the cloud service center;

[0118] A second execution unit 704, configured to perform a target processing operation on the target service data, and upload the processed target service data to the cloud service center. Among them, the target processing operation is a processing operation specified when the target service data is shelved and executed when the target service data is taken out of storage. The target processing operation includes at least one of the following: desensitization operation, encryption operation, and feature extraction operation.

[0119] It should be noted that the first execution unit 702 in this embodiment may be used to execute the above step S502, and the second execution unit 704 in this embodiment may be used to execute the above step S504.

[0120] Through the embodiments of the present application, a data upload notification sent by a cloud service center is received, and target service data is extracted from the data assets of the current edge node. Herein, the cloud service center belongs to a data processing system based on cloud-edge collaboration. The data processing system further includes a group of edge nodes. Each edge node in the group of edge nodes corresponds to a data node of a data provider. The data assets of each edge node provide data sources for the cloud service center in a shelving manner. The current edge node belongs to the group of edge nodes. The target service data is one of a group of service data used for the computing services to be executed by the cloud service center; a target processing operation is performed on the target service data, and the processed target service data is uploaded to the cloud service center. Herein, the target processing operation is a processing operation specified when the target service data is shelved and executed when the target service data is taken out of storage. The target processing operation includes at least one of the following: a desensitization operation, an encryption operation, and a feature extraction operation, which solves the problem of poor data security caused by data transfer between different nodes in the related art-based cloud-edge collaboration data processing solution, and improves the security and reliability of data transmission between different nodes.

[0121] In an exemplary embodiment, the above device further includes: a shelving unit configured to shelve the data assets of the current edge node. Herein, the data assets of the current edge node include field information of multiple data fields corresponding to the data assets of the current edge node. The shelving information of the data assets of the current edge node includes operation indication information for indicating, when the data assets of the current edge node are taken out of storage, the processing operations specified for the field information of each data field corresponding to the data assets of the current edge node.

[0122] In an exemplary embodiment, the second execution unit includes: a first execution module configured to perform a desensitization operation on the field information of the first field corresponding to the target service data when the target processing operation includes a desensitization operation specified for the field information of the first field corresponding to the target service data. Herein, the desensitization operation includes at least one of the following: a hashing operation, a masking operation, and a partitioning operation; a second execution module configured to perform an encryption operation on the field information of the second field corresponding to the target service data when the target processing operation includes an encryption operation specified for the field information of the second field corresponding to the target service data. Herein, the encryption operation includes a homomorphic encryption operation; a third execution module configured to perform a feature extraction operation on the field information of the third field corresponding to the target service data when the target processing operation includes a feature extraction operation specified for the field information of the third field corresponding to the target service data. Herein, the feature extraction operation is used to extract the distribution features of the data.

[0123] In an exemplary embodiment, the data assets of the current edge node are stored in the local database of each edge node. The second execution unit includes: an extraction module, configured to extract target service data from the local database of the current edge node, where the extracted target service data is in the form of a database table; a fourth execution module, configured to perform a target processing operation on the target service data and convert the processed target service data into a data frame format; and a fifth execution module, configured to serialize the converted target service data and upload the serialized target service data to the cloud service center in the form of a data packet.

[0124] In an exemplary embodiment, a data supply - side agent and a data consumption - side agent are deployed on the current edge node. Among them, the data supply - side agent of the current edge node is configured to perform the operations of putting on the shelf, extracting, and uploading the data assets of the current edge node, and the data consumption - side agent of the current edge node is configured to receive the data sent to the current edge node. The second execution unit includes: a sixth execution module, configured to extract target service data from the data assets of the current edge node through the data supply - side agent of the current edge node, perform a target processing operation on the extracted target service data, and upload the processed target service data to the cloud service center.

[0125] It should be noted that the above - mentioned various modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to this: all the above - mentioned modules are located in the same processor; or, the above - mentioned various modules are located in different processors in any combined form.

[0126] According to another aspect of the embodiments of the present application, there is provided a computer - readable storage medium. The computer - readable storage medium includes a stored program, where the program, when running, executes the steps in any one of the above - mentioned method embodiments.

[0127] In an exemplary embodiment, the above - mentioned computer - readable storage medium may include, but is not limited to: various media such as USB flash drives, ROMs, RAMs, mobile hard disks, magnetic disks, or optical discs that can store computer programs.

[0128] According to another aspect of the embodiments of the present application, there is provided an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. The processor is configured to execute the steps in any one of the above - mentioned method embodiments through the computer program. In an exemplary embodiment, the above - mentioned electronic device may further include a transmission device and an input / output device, where the transmission device is connected to the above - mentioned processor, and the input / output device is connected to the above - mentioned processor.

[0129] Specific examples in this embodiment may refer to the examples described in the above embodiments and exemplary embodiments, and will not be elaborated herein.

[0130] According to another aspect of the embodiments of the present application, a computer program product is further provided. The computer program product includes computer programs / instructions, and the computer programs / instructions contain program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit 801, various functions provided by the embodiments of the present application are executed. The serial numbers of the above embodiments of the present application are only for description and do not represent the advantages or disadvantages of the embodiments.

[0131] Figure 8 Schematically shows a block diagram of a computer system of an electronic device for implementing the embodiments of the present application. As Figure 8 shown, the computer system 800 includes a CPU 801, which can perform various appropriate actions and processes according to the programs stored in the ROM 802 or the programs loaded from the storage part 808 into the RAM 803. In the random access memory 803, various programs and data required for system operation are also stored. The central processing unit 801, the read-only memory 802, and the random access memory 803 are connected to each other through a bus 804. The I / O (Input / Output) interface 805 is also connected to the bus 804.

[0132] The following components are connected to the I / O interface 805: an input part 806 including a keyboard, a mouse, etc.; an output part 807 including such as a CRT (Cathode Ray Tube), an LCD (Liquid Crystal Display), etc. and speakers, etc.; a storage part 808 including a hard disk, etc.; and a communication part 809 including a network interface card such as a local area network card, a modem, etc. The communication part 809 performs communication processing via a network such as the Internet. The drive 810 is also connected to the input / output interface 805 as needed. A removable medium 811, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 810 as needed, so that the computer program read from it can be installed into the storage part 808 as needed.

[0133] In particular, according to the embodiments of the present application, the processes described in each method flowchart can be implemented as computer software programs. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through the communication part 809, and / or installed from the removable medium 811. When the computer program is executed by the central processing unit 801, various functions defined in the system of the present application are executed.

[0134] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example, and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0135] Obviously, those skilled in the art should understand that the above-mentioned modules or steps of the present application can be implemented by a general-purpose computing device. They can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices. They can be implemented by program codes executable by the computing device. Thus, they can be stored in a storage device and executed by the computing device. And in some cases, the steps shown or described can be executed in a different order from here, or they can be separately made into individual integrated circuit modules, or multiple modules or steps among them can be made into a single integrated circuit module to be implemented. In this way, the present application is not limited to any specific combination of hardware and software.

[0136] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, the present application can have various changes and modifications. Any modifications, equivalent replacements, improvements, etc. made within the principle of the present application should be included in the protection scope of the present application.

Claims

1. A data processing system based on cloud-edge collaboration, characterized in that: include: A cloud service center and a group of edge nodes, each edge node in the group of edge nodes corresponds to a data node of a data provider, and the data assets of each edge node provide a data source for the cloud service center by listing; wherein, The cloud service center is used to notify the edge node to which each business data in a set of business data used by the computing business to be executed belongs to upload each business data to the cloud service center; use the set of business data to execute the computing business, and send the calculation result of the computing business to the first edge node in the set of edge nodes, wherein the first edge node is a pre-specified edge node to which the calculation result is sent; The second edge node to which the target business data belongs is used to perform a target processing operation on the target business data, and upload the processed target business data to the cloud service center, wherein the target business data is any business data in the group of business data, and the target processing operation is a processing operation specified when the target business data is put on the shelf and performed when the target business data is out of the warehouse, and the target processing operation includes at least one of the following: desensitization operation, encryption operation, feature extraction operation; The first edge node is used to receive the calculation result sent from the cloud service center.

2. The data processing system according to claim 1, characterized in that: Each edge node is used to list the data assets of each edge node, wherein the data assets of each edge node include field information of multiple data fields corresponding to the data assets of each edge node, and the listing information of the data assets of each edge node includes operation instruction information for indicating the processing operation specified for the field information of each data field corresponding to the data assets of each edge node when the data assets of each edge node are shipped out.

3. The data processing system according to claim 2, characterized in that: The second edge node is used to perform a desensitizing operation on the field information of the first field corresponding to the target business data when the target processing operation includes a desensitizing operation specified for the field information of the first field corresponding to the target business data, wherein the desensitizing operation includes at least one of the following: a hash operation, a mask operation, and a partition operation; when the target processing operation includes an encryption operation specified for the field information of the second field corresponding to the target business data, perform the encryption operation on the field information of the second field corresponding to the target business data, wherein the encryption operation includes a homomorphic encryption operation; when the target processing operation includes a feature extraction operation specified for the field information of the third field corresponding to the target business data, perform the feature extraction operation on the field information of the third field corresponding to the target business data, wherein the feature extraction operation is used to extract distribution characteristics of data.

4. The data processing system according to claim 1, characterized in that: The data assets of each edge node are stored in a local database of each edge node; The second edge node is used to extract the target business data from the local database of the second edge node, wherein the extracted target business data is in the form of a database table; perform the target processing operation on the target business data, and convert the processed target business data into a data frame format; serialize the converted target business data, and upload the serialized target business data to the cloud service center in the form of a data packet.

5. The data processing system according to claim 4, characterized in that: The calculation results are stored in the cloud service center in the form of data packets; The first edge node is further used to deserialize the calculation result, wherein the deserialized calculation result is in a data frame format; import the deserialized calculation result into a database table, and store the database table into which the calculation result is imported into a local database of the first edge node.

6. The data processing system according to claim 1, characterized in that: The cloud service center includes: a computing center, a scheduling center and a data storage system; wherein, The scheduling center is used to notify the edge node to which each business data belongs to upload each business data to the data storage system; and when the group of business data has been uploaded, notify the computing center to pull the group of business data; based on the notification of the completion of the computing by the computing center, send the computing result stored in the data storage system to the first edge node; The computing center is used to read the set of business data from the data storage system based on the notification of the completion of the upload of the data of the scheduling center; perform the computing business using the set of business data, store the computing results in the data storage system, and notify the scheduling center to send the computing results; The data storage system is used to store each business data uploaded by the edge node to which each business data belongs; and store the calculation results stored by the computing center.

7. The data processing system according to claim 6, characterized in that: Each business data is uploaded to the data storage system in the form of a data packet; The computing center is also used to deserialize each of the business data read to obtain each of the business data after deserialization, wherein each of the business data after deserialization is in a data frame format; use each of the business data after deserialization to execute the computing business to obtain the computing result, wherein the computing result is in a data frame format; serialize the computing result, and store the serialized computing result in the form of a data packet in the data storage system.

8. The data processing system according to any one of claims 1 to 7, characterized in that: Each edge node is deployed with a data supply-side agent and a data consumption-side agent, wherein the data supply-side agent of each edge node is used to perform the operation of putting on the shelf, extracting and uploading the data assets of each edge node, and the data consumption-side agent of each edge node is used to receive the data sent to each edge node; The data supply-side agent of the second edge node is used to extract the target business data from the data assets of the second edge node, perform the target processing operation on the extracted target business data, and upload the processed target business data to the cloud service center; The data consumer agent of the first edge node is used to receive the calculation result sent by the cloud service center and store the calculation result in a local database of the first edge node.

9. A data processing method based on cloud-edge collaboration, characterized in that: include: Receive a data upload notification issued by a cloud service center, and extract target business data from the data assets of the current edge node, wherein the cloud service center belongs to a data processing system based on cloud-edge collaboration, and the data processing system also includes a group of edge nodes, each edge node in the group of edge nodes corresponds to a data node of a data provider, and the data assets of each edge node provide a data source for the cloud service center by listing, the current edge node belongs to the group of edge nodes, and the target business data is one of the business data in a group of business data used by the computing business to be executed by the cloud service center; Perform a target processing operation on the target business data, and upload the processed target business data to the cloud service center, wherein the target processing operation is a processing operation specified when the target business data is put on the shelf and performed when the target business data is shipped out of the warehouse, and the target processing operation includes at least one of the following: desensitizing operation, encryption operation, feature extraction operation.

10. The method according to claim 9, characterized in that The method further comprises: Listing the data assets of the current edge node, wherein the data assets of the current edge node include field information of multiple data fields corresponding to the data assets of the current edge node, and the listing information of the data assets of the current edge node includes operation instruction information for indicating the processing operation specified for the field information of each data field corresponding to the data assets of the current edge node when the data assets of the current edge node are shipped out.

11. The method according to claim 10, characterized in that The performing a target processing operation on the target business data includes: In a case where the target processing operation includes a desensitization operation specified for the field information of the first field corresponding to the target business data, performing the desensitization operation on the field information of the first field corresponding to the target business data, wherein the desensitization operation includes at least one of the following: a hash operation, a mask operation, and a partition operation; In a case where the target processing operation includes an encryption operation specified for field information of a second field corresponding to the target business data, performing the encryption operation on the field information of the second field corresponding to the target business data, wherein the encryption operation includes a homomorphic encryption operation; In a case where the target processing operation includes a feature extraction operation specified for field information of a third field corresponding to the target business data, the feature extraction operation is performed on the field information of the third field corresponding to the target business data, wherein the feature extraction operation is used to extract distribution characteristics of the data.

12. The method according to claim 9, characterized in that The data assets of the current edge node are stored in the local database of each edge node; The performing a target processing operation on the target business data and uploading the processed target business data to the cloud service center includes: Extracting the target service data from a local database of the current edge node, wherein the extracted target service data is in the form of a database table; Performing the target processing operation on the target service data, and converting the processed target service data into a data frame format; The converted target business data is serialized, and the serialized target business data is uploaded to the cloud service center in the form of a data packet.

13. The method according to any one of claims 9 to 12, characterized in that The current edge node is deployed with a data supply-side agent and a data consumption-side agent, wherein the data supply-side agent of the current edge node is used to perform the operation of putting on the shelf, extracting and uploading the data assets of the current edge node, and the data consumption-side agent of the current edge node is used to receive the data sent to the current edge node; The performing a target processing operation on the target business data and uploading the processed target business data to the cloud service center includes: The target business data is extracted from the data assets of the current edge node through the data supply-end agent of the current edge node, the target processing operation is performed on the extracted target business data, and the processed target business data is uploaded to the cloud service center.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the steps of the method described in any one of claims 9 to 13 when executed by a processor.

15. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method described in any one of claims 9 to 13 are implemented.

Images