Data management method and related device

Abstract

Disclosed in embodiments of the present application is a data management method. The method is applied to a data management system, and the data management system is located in an isolated operation environment, so that the use of third-party data can be effectively limited in the isolated operation environment, thereby preventing the leakage of the third-party data provided by a data provider. The third-party data is stored in a storage node of the data management system; when an execution node of the data management system runs a service program to execute a service of a data user, the execution node needs to send to the storage node a data reading request for the third-party data; and on the basis of a data reading strategy, the storage node can control reading of the third-party data by the service program, so that the use of the third-party data by the data user by means of the service program is monitored by the storage node, thereby ensuring that the third-party data is safely and appropriately used by the data user, and reducing the risk of data leakage.

Description

A data management method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office on January 11, 2024, with Chinese application number 202410044120.5 and invention name “A data management method, system, device and storage medium”, and claims priority to the Chinese patent application filed with the State Intellectual Property Office on April 29, 2024, with Chinese application number 202410532731.4 and invention name “A data management method and related devices”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to a data management method and related equipment. Background Art

[0003] With the continuous development of information technology, data is in high demand across various business fields, and data procurement is emerging in various application scenarios. For example, the performance improvement of artificial intelligence (AI) models relies heavily on high-quality training data. Currently, public and private data are no longer sufficient for AI model training. Therefore, AI developers (data users) need to purchase large amounts of high-quality data from third parties (data providers).

[0004] Currently, data providers usually clarify data usage rights through written agreements, etc., but the data users' use of data is usually not transparent enough to the data providers, which may lead to the abuse or even leakage of the data provided by the data providers.

[0005] Therefore, there is an urgent need for a method that can manage the use process of data provided by data providers to ensure that the data provided by data providers are used safely and reasonably by data users.

[0006] Summary of the Invention

[0007] The present application provides a data management method that can manage the use of data provided by a data provider to ensure that the data provided by the data provider is used safely and reasonably by the data user. The present application also provides corresponding devices, equipment, computer-readable storage media, and computer program products.

[0008] A first aspect of the present application provides a data management method, which is applied to a data management system, wherein the data management system is arranged in at least one data center, and the data management system is located in an isolated operating environment, and the isolated operating environment is connected to the data provider and the data user respectively through an application programming interface (API). The data management system includes an execution node and a storage node, and the storage node stores the third-party data of the data provider. The method includes: the execution node runs a business program, and the business program is used to execute the business of the data user; the execution node sends a data read request to the storage node, and the data read request indicates that the running business program requests to read first data from the storage node, and the first data belongs to third-party data; the storage node generates feedback information based on the data read request and the data reading policy for the third-party data and sends the feedback information to the execution node.

[0009] In the first aspect, the data management system is located in an isolated operating environment, which makes it difficult for data users to directly transfer third-party data provided by data providers outside the isolated operating environment, so that the use of third-party data is effectively restricted to the isolated operating environment, avoiding the leakage of third-party data provided by data providers.

[0010] In addition, the third-party data is stored in the storage node of the data management system. When the execution node of the data management system runs the business program to execute the business of the data user, when the third-party data needs to be used, the execution node needs to send a data read request to the storage node. In this way, the storage node can control the business program's reading of the third-party data based on the data reading policy, so that the data user's use of the third-party data through the business program is monitored by the storage node, and the storage node manages the use process of the third-party data provided by the data provider to ensure that the third-party data is used safely and reasonably by the data user, reducing the risk of data leakage.

[0011] In a possible implementation of the first aspect, the method further includes: monitoring data users' access behavior to the data management system through the API, and monitoring data output behavior of the data management system through the API.

[0012] In this possible implementation, APIs can be managed through an API gateway or other API management tools, allowing for monitoring of API-related data interactions. This allows monitoring of external nodes (data users) accessing internal nodes (data management systems) through APIs, as well as monitoring data output from data management systems through APIs.

[0013] Access behavior can be monitored by monitoring one or more of the following: access frequency, corresponding response size, response content, etc., thereby identifying abnormal access behavior. Monitoring data output behavior can also include detecting the similarity between the data to be output and third-party data provided by the data provider, thereby preventing third-party data leakage.

[0014] In a possible implementation of the first aspect, the data reading policy is determined by the data provider, and the data reading policy includes policy items and operations determined based on the policy items. The policy items include one or more of the following items: data policy items, usage policy items, quantity policy items, time policy items, and subject policy items; wherein, the data policy items are used to describe the data range that can be read in the third-party data; the usage policy items are used to describe the tasks that can use the third-party data in the business program; the quantity policy items are used to describe the usage threshold of the third-party data; the time policy items are used to describe the usage time range of the third-party data; and the subject policy items are used to describe the subject type to which the data reading policy can be applied.

[0015] In this possible implementation, the data reading strategy can be flexibly configured from one or more aspects through one or more strategy items. The configuration method is clear and easy to implement, which can improve configuration efficiency and facilitate understanding by data users and data providers, thereby facilitating negotiation and unification between data users and data providers.

[0016] In a possible implementation of the first aspect, the storage node includes one or more data reading policies regarding third-party data; the storage node generates feedback information and sends the feedback information to the execution node based on the data reading request and the data reading policies regarding the third-party data, including: the storage node matches the data reading request with a policy item of at least one data reading policy among the one or more data reading policies; if there is a data reading policy that matches the data reading request, the storage node generates feedback information and sends the feedback information to the execution node based on the operation in the data reading policy that matches the data reading request.

[0017] In this possible implementation, there may be one or more data read policies to be matched. When there are multiple data read policies to be matched, the multiple data read policies may be matched sequentially. Specifically, matching may be performed based on the policy items in the data read policies. The multiple data read policies are matched sequentially until a matching data read policy is found, and feedback information is generated based on the operation in the data read policy. Alternatively, until it is determined that all data read policies do not match, feedback information may be generated based on a default operation, or feedback information indicating that data read is denied may be generated.

[0018] In a possible implementation of the first aspect, the storage node also includes a data permission policy, which is used to determine the read and write permissions for the data based on the source of the data, wherein the data permission policy indicates that the data user has read permissions for third-party data from the data provider; before the storage node generates feedback information based on the first data and the data reading policy regarding the third-party data and sends the feedback information to the execution node, it also includes: the storage node determines, based on the data permission policy, that the business program has read permissions for the first data based on the fact that the first data belongs to third-party data.

[0019] In this possible implementation, the data permission policy can be a system-level policy of the data management system, thereby enabling basic management of the read and write permissions of data in the storage node through this system-level policy. The data permission policy can include information indicating that the data user has read permission for third-party data from the data provider. In this way, after determining that read permission is granted for the first data, the data can be read according to the data read policy.

[0020] It can be seen that in this possible implementation, the reading operation of data in the storage node can be managed more perfectly through at least two levels of strategies.

[0021] In a possible implementation of the first aspect, the data management system includes a management node; the method also includes: after the management node obtains the running instruction for the business program, it creates an access credential for the storage node; the management node instructs the execution node to run the business program and passes the access credential information to the execution node; the execution node sends a data read request to the storage node, including: the execution node sends the data read request to the storage node based on the access credential information.

[0022] In this possible implementation, the management node can manage the execution node's access credentials to the storage node. In other words, the management node can manage the execution node's permissions to interact with the storage node. This prevents the execution node from independently gaining access to the storage node, preventing the execution node from reading and using third-party data on the storage node without permission, thereby preventing the execution node from leaking and improperly using third-party data.

[0023] In a possible implementation manner of the first aspect, the method further includes: deleting the access credential after the execution node stops running the service program.

[0024] In this possible implementation, after the execution node stops running the business program, deleting the access credential can ensure the validity of the access credential.

[0025] In this way, each time an execution node starts a business program, it must re-acquire permission to interact with the storage node from the management node. This prevents execution nodes from independently obtaining access to storage nodes, nor can they abuse existing access rights. This effectively ensures controllability of each execution node's access to storage nodes, preventing execution nodes from reading and using third-party data on storage nodes without permission, thereby preventing the leakage and improper use of third-party data by execution nodes.

[0026] In a possible implementation of the first aspect, after the storage node generates feedback information based on the first data and the data reading policy regarding the third-party data and sends the feedback information to the execution node, it also includes: the storage node records data access behavior related to the data reading request.

[0027] In this possible implementation, after the storage node stores the information of the data access behavior, the data provider can conveniently and flexibly view the usage of the third-party data stored in the storage node.

[0028] In a possible implementation of the first aspect, the method further includes: the storage node obtains a query request input by a data provider, where the query request is used to request access records related to third-party data; and the storage node outputs an access record, where the access record includes data access behavior related to the data read request.

[0029] In this possible implementation, data providers can easily query the access and usage of third-party data by data users through storage nodes, making the use of third-party data transparent. Furthermore, data providers can promptly identify abnormal use of third-party data and adjust data access policies accordingly. Data providers can also initiate a policy negotiation process with data users through the data management system, using the data management system as an intermediary platform to negotiate new data access policies with data users to meet the requirements of both data providers and data users.

[0030] In a possible implementation of the first aspect, the method further includes: during the process of running the business program, the execution node sends the data to be stored to the storage node; after receiving the data to be stored, the storage node determines and stores a derivative link of the data to be stored based on third-party data read by the execution node from the storage node within a specified time period, the specified time period including the time period of the execution node running the business program this time, and the derivative link is used to indicate the data in the storage node used to generate the data to be stored.

[0031] In this possible implementation, when executing a business program, after reading third-party data from a storage node, an execution node may generate derivative data during the application of the third-party data. To ensure data security, this possible implementation can track and manage this derivative data through a derivative link to prevent leakage of the derived data.

[0032] In a possible implementation manner of the first aspect, the method further includes: after the execution node stops running the business program, deleting operation data related to the business program in the execution node.

[0033] In this possible implementation method, it can effectively ensure that the traces of third-party data used in the process of running business programs are deleted, avoiding the execution node from unreasonably retaining the third-party data used in the process of running business programs, thereby ensuring that the third-party data is not leaked and abused.

[0034] A second aspect of the present application provides a data management system that has the functionality to implement the method of the first aspect or any possible implementation of the first aspect. This functionality can be implemented in hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above-mentioned functionality, such as a management node, an execution node, and a storage node.

[0035] A third aspect of the present application provides a computing device cluster, which includes at least one computing device, and the at least one computing device includes a processor and a memory. The memory of at least one computing device stores computer-executable instructions that can be run on the processor. When the computer-executable instructions are executed by the processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0036] The fourth aspect of the present application provides a computer-readable storage medium storing one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0037] The fifth aspect of the present application provides a computer program product that stores one or more computer-executable instructions. When the computer-executable instructions are executed by a processor, the processor executes the method as described in the first aspect or any possible implementation of the first aspect.

[0038] A sixth aspect of the present application provides a chip system, which includes a processor for supporting the processor in implementing the functions involved in the first aspect or any possible implementation of the first aspect. In one possible design, the chip system may also include a memory for storing necessary program instructions and data. The chip system may be composed of a chip or may include a chip and other discrete devices.

[0039] Among them, the technical effects brought about by the second to sixth aspects or any possible implementation methods thereof can refer to the technical effects brought about by the first aspect or the relevant possible implementation methods of the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] FIG1 is a schematic diagram of an exemplary process provided in an embodiment of the present application;

[0041] FIG2 is a schematic diagram of an exemplary system framework of the AI ​​platform provided in an embodiment of the present application;

[0042] FIG3a is an exemplary schematic diagram of a development space in a logical multi-tenancy scenario provided by an embodiment of the present application;

[0043] FIG3 b is an exemplary schematic diagram of a development space in a physical multi-tenancy scenario provided by an embodiment of the present application;

[0044] FIG4 is an exemplary schematic diagram of an isolated operating environment provided in an embodiment of the present application;

[0045] FIG5 is an exemplary schematic diagram of a management node, an execution node, and a storage node in an AI model training scenario provided by an embodiment of the present application;

[0046] FIG6 is an exemplary schematic diagram of policy items of a data reading policy provided in an embodiment of the present application;

[0047] FIG7 is an exemplary schematic diagram of a data management method provided in an embodiment of the present application;

[0048] FIG8 is a schematic diagram of an exemplary process provided in an embodiment of the present application;

[0049] FIG9 is a schematic diagram of an exemplary process provided in an embodiment of the present application;

[0050] FIG10 is an exemplary schematic diagram of a derivative link provided in an embodiment of the present application;

[0051] FIG11 is an exemplary schematic diagram of a derivative link provided in an embodiment of the present application;

[0052] FIG12 is an exemplary schematic diagram of a data management system provided in an embodiment of the present application;

[0053] FIG13 is a schematic diagram of a structure of a computing device provided in an embodiment of the present application;

[0054] FIG14 is a schematic diagram of a structure of a computing device cluster provided in an embodiment of the present application;

[0055] FIG15 is a schematic diagram of the structure of a computing device cluster provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The following describes the embodiments of the present application in conjunction with the accompanying drawings. The terms used in the implementation methods of the present application are only used to explain the specific embodiments of the present application and are not intended to limit the present application.

[0057] Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.

[0058] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. The terms "first", "second", etc. in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchangeable where appropriate. This is merely a way of distinguishing objects with the same properties when describing them in the embodiments of this application. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent to such process, method, product, or apparatus.

[0059] In many application scenarios, the purchase of third-party data is involved.

[0060] For example, in the scenario of AI model training, AI developers (that is, data users) need to purchase a large amount of high-quality data from a third party (that is, data providers).

[0061] Currently, one way to constrain data users' rights to use purchased data is through a written contract that specifies their rights. However, due to the difficulty in overseeing data users' use of data, the data users' use of data is often not transparent to the data provider. This can lead to the potential misuse or even leakage of the data provided by the data provider (hereinafter referred to as third-party data for convenience).

[0062] Currently, a common approach to controlling data usage is the user-role-permission authorization model. Data providers assign minimum permissions to data users, limiting their data usage rights. However, this control approach is relatively simplistic. For example, it fails to control the number of times a data user uses third-party data, nor does it record their use of third-party data, rendering data providers unaware of third-party data usage. Furthermore, data users can still copy and export third-party data and its derivatives while using it through applications, potentially leading to data leaks.

[0063] Another method for controlling data usage is for the data provider to use a rights management service to attach usage permissions for third-party data to the third-party data and encrypt the data. For example, for third-party document data, the data provider can set usage permissions and encrypt the data. The data user can then access the encrypted third-party data and open it with an application that uses the document. The application can then decrypt the document and control the data user's access to it based on the permissions attached to the document.

[0064] However, in this data usage control method, data needs to be encrypted and decrypted, which takes a long time when there is a lot of data, and data users can obtain the plaintext data of third-party data through the application. Therefore, the third-party data can be easily copied and forwarded by data users through the application, resulting in leakage, and making it difficult for data providers to manage the use and deletion of third-party data and derivative data in the application.

[0065] It can be seen that traditional data usage control methods are difficult to ensure that the third-party data provided by data providers are used safely and reasonably by data users.

[0066] Based on this, an embodiment of the present application provides a data management method that can manage the use process of third-party data provided by a data provider to ensure that the third-party data provided by the data provider is used safely and reasonably by data users.

[0067] As shown in FIG1 , the data management method of the embodiment of the present application mainly involves one or more of the following aspects:

[0068] Build an isolated operating environment, configure policies, use policies to read data, manage derived data, and stop running business programs.

[0069] The following is an exemplary introduction to each aspect.

[0070] 1. Build an isolated operating environment.

[0071] The data management method can be applied to a data management system, which is located in an isolated operating environment.

[0072] The isolated execution environment may be implemented by a computing device cluster, wherein the computing device cluster may include one or more computing devices.

[0073] The type of any computing device is not limited herein. For example, any computing device may be a terminal device, or may be a server, server cluster, container, or virtual machine. When a computing device cluster includes multiple computing devices, the types of the different computing devices may be the same or different.

[0074] The one or more computing devices may be included in one or more data centers, and the data management system may be a system built based on resources in the data centers.

[0075] The specific forms of the isolated operating environment where the data management system is located can be various.

[0076] For example, the isolated operating environment can be located in a cloud platform or AI platform implemented through a cluster of computing devices, or it can be deployed on a server and provide cloud services and applications that provide data computing capabilities through the network or application programming interface (API), such as function cloud services, online document editors, network disks, etc.

[0077] The following is an exemplary introduction using the AI ​​platform as an example. It should be noted that the isolated operating environment is not limited to the AI ​​platform.

[0078] In the AI ​​model training scenario, the data management system located on the AI ​​platform can perform services including AI model training.

[0079] As shown in Figure 2, it is a schematic diagram of an exemplary system framework of an AI platform.

[0080] As shown in the example of FIG2 , the AI ​​platform can provide access interfaces to users such as data users and data providers.

[0081] For example, by providing an interface through the console, users can access the AI ​​platform's console through a browser, etc., thereby accessing the AI ​​platform; or, the application in the user's client device can access the API provided by the AI ​​platform through the API gateway, thereby accessing the AI ​​platform.

[0082] The AI ​​platform can also provide one or more of data management services, labeling services, training services, reasoning services, and storage services, and can provide access interfaces for users such as data users and data providers.

[0083] The labeling service, training service, and inference service can provide their respective services through corresponding applications based on corresponding algorithms, etc., and can access the storage service through the storage software development kit (SDK) provided by the storage service.

[0084] The storage service may store data, for example, data users' own data, data providers' third-party data, and derivative data generated based on other data (such as own data and third-party data).

[0085] Storage services can also store metadata, including information about owned data, third-party data, and derived data. This metadata can also include policies for subsequent steps to control the use of third-party data and other data. Furthermore, storage services can store data access records to help data providers understand how third-party data is being used.

[0086] In this way, data users can access the AI ​​platform to train AI models using the resources of the AI ​​platform.

[0087] Furthermore, data providers can store and sell their third-party data through the AI ​​platform. This allows data users to purchase third-party data from data providers through the AI ​​platform to train AI models.

[0088] Among them, in the embodiments of the present application, the data user can be understood as an individual, enterprise or organization, and can also be understood as a client device, account, network address or other form of identification represented by the subject, the identification and specific form of the data user can be multiple; similarly, the data provider can be understood as an individual, enterprise or organization, and can also be understood as a client device, account, network address or other form of identification represented by the subject, the identification and specific form of the data provider can be multiple, and there is no restriction on this in the application embodiments.

[0089] In this example, an isolated operating environment can be built in the AI ​​platform, and a data management system can be implemented in the isolated operating environment.

[0090] As shown in FIG3a , when the development space provided by the AI ​​platform to tenants (which may be data users) is logically multi-tenanted, the computing device cluster that implements the AI ​​platform can process requests from multiple tenants. That is, multiple tenants (such as development space 1 of tenant 1 and development space 2 of tenant 2 shown in FIG3a ) share the resources of the AI ​​platform. Then, the isolated operating environment may be the AI ​​platform, wherein the isolated operating environment may include network resources, computing resources, and storage resources of the AI ​​platform.

[0091] As shown in Figure 3b, when the development space provided by the AI ​​platform to tenants (which can be data users) is physically multi-tenanted, the resources of the development space leased by the tenant from the AI ​​platform are exclusive to that tenant, and the resources between tenants in the AI ​​platform are fully isolated. In other words, in the example shown in Figure 3b, the network resources, computing resources, and storage resources in Tenant 1's Development Space 1 are isolated operating environment 1, while the network resources, computing resources, and storage resources in Tenant 2's Development Space 2 are isolated operating environment 2. Isolated operating environment 1 and isolated operating environment 2 are isolated from each other.

[0092] It can be seen that in the embodiment of the present application, the isolated operating environment can be realized from one or more aspects of network, computing and storage.

[0093] They are introduced below one by one.

[0094] 1. Network isolation

[0095] In the example shown in FIG4 , the isolated operating environment is connected to other nodes outside the isolated operating environment through an API. For example, the isolated operating environment is connected to a data provider and a data user respectively through the API.

[0096] In this way, internal nodes in the isolated operating environment (such as execution nodes and storage nodes in the data management system) cannot directly communicate with external nodes outside the isolated operating environment (such as client devices of data users).

[0097] In this way, the information transmission between internal nodes and external nodes can be monitored through the monitoring API.

[0098] Specifically, in some embodiments, the access behavior of data users to the data management system through the API can be monitored, and the data output behavior of the data management system through the API can be monitored.

[0099] For example, the API can be managed through an API gateway or other forms of API management tools. The data management system can register the API with the API gateway to monitor the data interactions related to the API through the API gateway.

[0100] The API gateway can access internal nodes in the isolated operating environment. The API gateway can be deployed in the isolated operating environment. For example, the API gateway can be an elastic load balancing (ELB) service in the cloud platform. Alternatively, the API gateway can be located outside the isolated operating environment and connected to the isolated operating environment through technologies such as virtual private cloud (VPC) peering, source network address translation (SNAT), and elastic Internet Protocol (EIP).

[0101] In addition, the API gateway can be accessed by external nodes. For example, it can be connected to the external network where the external nodes are located through the ELB service in the cloud platform.

[0102] Furthermore, the network flow within the isolated runtime environment can be controlled through, for example, an API gateway. For example, the network flow can be unidirectional, from an external node to the API gateway, and then from the API gateway to an internal node. This means that an external node can only send a request to an internal node via the API, and the internal node, after processing the request, can send a response to the external node via the API. An internal node cannot actively send data to an external node via the API without receiving a request from an external node.

[0103] In this way, through the API gateway, etc., it is possible to monitor the access behavior of external nodes (data users) to internal nodes (data management systems) through APIs, and monitor the data output behavior of data management systems through APIs.

[0104] Access behavior can be monitored by monitoring one or more of the following: access frequency, corresponding response size, response content, etc., thereby identifying abnormal access behavior. Monitoring data output behavior can also include detecting the similarity between the data to be output and third-party data provided by the data provider, thereby preventing third-party data leakage.

[0105] The specific form of the isolated operating environment is not limited here.

[0106] For example, in a cloud platform, the isolated operating environment can be an independent network environment created using a virtual private cloud (VPC). In a non-cloud platform, the isolated operating environment can be a small network environment or an independent network area isolated by a firewall.

[0107] 2. Computational isolation

[0108] In some examples, a data management system in an isolated operating environment includes a management node, an execution node, and a storage node. The management node can be used to execute system operations of the data management system, the execution node is used to run business programs, and the business programs are used to execute the business of data users. The storage node can be used to provide storage services, for example, storing third-party data provided by data providers.

[0109] Among them, the specific forms of nodes such as management nodes, execution nodes and storage nodes, and the forms of the above-mentioned nodes can be the same or different, and are not limited here.

[0110] Exemplarily, any node can be a software module or plug-in in a data management system; or, it can be a virtual machine or container; or, it can be hardware in a computing device cluster that implements the data management system, for example, the execution node is a server or server cluster in the computing device cluster, and the storage node is a storage device in the computing device cluster.

[0111] In an embodiment of the present application, the business program that executes the data user's business (such as AI model training) is executed by the execution node, so that the data management system can isolate and manage the operation and data interaction of the business program, and avoid the leakage of third-party data and related derivative data during the execution of the business program.

[0112] In addition, the management node can also monitor the business programs run by the execution nodes. For example, it can monitor frequently used business programs and promptly detect abnormalities in the business programs.

[0113] In addition, the data management system can also limit users' viewing permissions for business program execution results, execution logs and other execution data. For example, the scale and frequency of users' viewing of execution data can be limited. For example, users can view up to 1,000 execution results within a specified time period, or can view up to 10MB of execution logs.

[0114] 3. Storage Isolation

[0115] The storage nodes of the data management system in the data management system can provide storage services.

[0116] Among them, data users can transfer, download, and delete their own data to storage nodes.

[0117] The data provider can transmit the third-party data provided by the data provider to the storage node, and the storage node stores the third-party data.

[0118] In this way, data users can apply to storage nodes for permission to use third-party data, for example, by purchasing the permission. In this way, the use of third-party data of data providers can be managed through storage nodes.

[0119] In actual scenarios, third-party data and related derivative data stored in storage nodes cannot be downloaded directly. If data users or other users need to download third-party data from storage nodes, they can initiate a download request for the third-party data to the data provider through the data management system. Only after the data provider of the third-party data approves the download request through the data management system will the data user or other users be allowed to download the corresponding third-party data from the storage node.

[0120] As shown in Figure 5, taking the data management system as an AI platform as an example, the relevant functions of the management node, execution node, and storage node in the training scenario of the AI ​​model are introduced exemplarily.

[0121] In the example shown in FIG5 , the management node of the AI ​​platform can provide platform-level AI services.

[0122] AI services can trigger execution nodes to perform business services based on instructions from data users.

[0123] Specifically, when executing a business program, the AI ​​service first initializes the execution node and then assigns the business program to the execution node for execution. The business program can use the AI ​​platform's infrastructure (such as a graphics processing unit (GPU) cluster) to train and infer the AI ​​model.

[0124] During the execution of a business program, the execution node can write temporary data generated during the execution process to local storage. After the business program is executed, the local storage is cleared and released. The execution node can write logs generated during the execution process to the log service, or write them to local storage for collection by the log agent to the log service. The execution node can also write persistent data such as AI models to the storage service. Users need to view logs and persistent data through the AI ​​service and configure policies in the AI ​​service to control data usage.

[0125] 2. Configure the strategy.

[0126] In the embodiment of the present application, the data management system may include one or more of the following strategies:

[0127] Data permission policy and data reading policy.

[0128] Through the above strategies, the data management system can manage the reading operations of data in the storage nodes.

[0129] The above strategies are introduced below respectively.

[0130] 1. Data permission strategy

[0131] In this example, the data permission policy is used to determine the read and write permissions for data based on the data source.

[0132] In some examples, the data permission policy may be a system-level policy, that is, the data permission policy may be applied to a data management system and pre-configured by a developer of the data management system.

[0133] The source of data may include one or more of the following: proprietary data, third-party data, and derived data. Proprietary data refers to data provided by data users, third-party data refers to data provided by data providers, and derived data refers to data obtained based on third-party data or other data. In some examples, a storage node may consider data to be stored on the storage node when an execution node executes a business program (also referred to as data to be stored) as derived data.

[0134] In some examples, the data permission policy can also be used to determine the read and write permissions for data based on the content of the data.

[0135] The content of the data may be classified in a variety of ways. For example, the content of the data may include ordinary data and model data. All data except the model data may be considered ordinary data.

[0136] In addition, data permission policies can also be used to determine the read and write permissions for data based on the purpose of the data.

[0137] The following uses a specific example to illustrate an exemplary content of the data permission policy.

[0138] Table 1: Data permission policy

[0139] It can be seen that in the example shown in Table 1, the data permission policy can indicate read permission for third-party data from the data provider, and thus the third-party data that the business program can read from the storage node can be determined subsequently based on the data read policy.

[0140] 2. Data reading strategy

[0141] In an embodiment of the present application, the data reading strategy can be a reading strategy for corresponding third-party data. The specific type of the third-party data is not limited here. For example, the third-party data can be a file or a group of files (such as a directory, a compressed data package, etc.). The third-party data can include attributes such as type, name, path, number of lines, and storage size.

[0142] In this example, the data read policy corresponding to a particular third-party data can be determined by the data provider of the third-party data. Specifically, the data management system can provide a configuration interface, and the data provider of the third-party data can enter configuration information through the configuration interface to configure the data read policy; alternatively, the data management system can provide the data provider of the third-party data with a candidate data read policy (which can be obtained from historical data read policies or default data read policies), and the data provider can determine the candidate data read policy as the data read policy for the third-party data.

[0143] The data reading policy includes policy items and operations determined based on the policy items. The number and content of the policy items can be various. The object of any policy item can be the data itself or a file in the data. The specific type of the object of each policy item can be pre-configured, for example, by defining it through PolicyType. Specifically, PolicyType of DATA indicates that the object of the policy item is the data itself, and PolicyType of FILE indicates that the object of the policy item is a file. The operation determined based on the policy item is the action indicated after the policy item is matched, for example, it can include allowing reading, denying reading, etc.

[0144] As shown in FIG6 , in some examples, the policy item includes one or more of the following items:

[0145] Data policy items, usage policy items, quantity policy items, time policy items, and subject policy items.

[0146] Below, various exemplary policy items are introduced respectively.

[0147] 1) Data Policy Item

[0148] Data policy items are used to describe the range of data that can be read from third-party data. This data policy item can segment the data and determine the data that can be read from the segmented data.

[0149] Exemplarily, the data policy items may include path filter (Path Filter), column selection (Columns) and row filter (Row Filter).

[0150] Path filtering involves selecting files from multiple files contained in third-party data. The default setting is to select all files. Specifically, you can use file extensions, wildcards, regular expressions, and other methods to filter. Wildcards can use simple, easy-to-use path matching expressions, such as those in the Ant style.

[0151] Selecting columns means selecting one or more columns. The default setting is to select all columns. For third-party data such as comma-separated values ​​(CSV), JavaScript object notation (JSON), and Excel, columns can be defined when transferring this data to the data management system. If this third-party data does not have columns defined, selecting columns does not apply. For example, binary file types such as images and videos do not support this option.

[0152] Row filtering involves filtering third-party data by row, reading only rows that meet the criteria. For example, "name in('beijing', 'nanjing') AND age>30," where name and age are columns. For third-party data in data types like CSV, JSON, and Excel, columns can be defined when transferring the data to the data management system. If a particular piece of third-party data doesn't have rows defined, row filtering doesn't apply. For binary file types like images and videos, you can filter based on metadata, such as "imageSize<10MB AND imageWidth<1024," where imageSize describes the image size and imageWidth describes the image width.

[0153] For example, the data policy item can be configured using the following code:

[0154] 2) Usage Policy Item

[0155] The usage policy item is used to describe the tasks in a business program that can use third-party data, that is, it is used to describe which functions can use third-party data.

[0156] For example, in the application scenario of the AI ​​model, the functions involved may include one or more of the following:

[0157] a) Labeling: labeling;

[0158] b) Feature: Feature engineering;

[0159] c) Train: training;

[0160] d)Reasoning: Reasoning.

[0161] For example, the usage policy item can be configured using the following code:

[0162] struct UsagePolicyItem{

[0163] String[]usageList;

[0164] }

[0165] 3) Count Policy Item

[0166] The quantity policy item is used to describe the usage threshold for third-party data. In other words, it describes the amount of third-party data that can be used. This usage threshold can be set in a variety of situations. The specific usage threshold can vary depending on whether the data policy item is for data or files.

[0167] In one example, PolicyType is DATA, indicating that the object of the policy item is the data itself. In this case, the usage of the data policy item can be the number of uses, specifically, the data policy item is used to describe the usage number threshold (maxUseTimes) of the third-party data.

[0168] The usage count threshold may be the maximum number of times the third-party data can be used. The calculation rules for the usage count may be defined by the user or the data management platform, and include but are not limited to:

[0169] a) Count only once: If the algorithm reads data once or more during a run, useTimes is accumulated only once. It is accumulated for multiple runs.

[0170] b) Multiple uses within a period of time are counted only once: When the algorithm program reads data once or more within a period of time, useTimes is accumulated only once.

[0171] c) One use counts as one: Each time the algorithm program uses the data, useTimes is accumulated once.

[0172] In another example, PolicyType is FILE, indicating that the object of the policy item is a file. In this case, the usage threshold may include one or more of the following:

[0173] a) The maximum number of rows (maxRowCount) refers to the maximum number of rows that can be used in a single file. This usage threshold does not apply to unstructured third-party data such as images and videos.

[0174] b) Maximum storage size (maxStorageSize) refers to the maximum amount of data that can be used by a single file.

[0175] For example, the Count Policy Item can be configured using the following code:

[0176] 4) Time Policy Item

[0177] The time policy item is used to describe the time range for using third-party data.

[0178] For example, the time policy item may specify a specific deadline, for example, may adopt the RFC 3339 date format, and may include a time zone, for example, the media time may be set to 2023-12-01T12:21:32Z.

[0179] Alternatively, the time policy item may specify a usage duration. For example, the start time may be the time when the third-party data is transferred to the data management system. The usage duration may be expressed in a variety of ways, such as using the Duration in RFC 3339. For example, the usage duration is expressed as P2DT3H4M, indicating that the usage duration is "2 days, 3 hours, and 4 minutes."

[0180] For example, the time policy item can be configured using the following code:

[0181] 5) Subject Policy Item

[0182] The subject policy item is used to describe the subject type to which the data read policy can be applied, thereby clarifying the type of subject object to which the corresponding data read policy is applied.

[0183] Considering that the data management system may be used by multiple users, different data reading policies can be specified for different users through the subject policy item.

[0184] There may be many specific cases for the subject type, which can be determined according to the specific application scenario.

[0185] Exemplarily, the subject type may include one or more of the following:

[0186] a) User: An account involved in the data management system, for example, a user in the cloud management platform or AI platform where the data management system is located, with an independent identity.

[0187] b) User Group: A user group is a collection of users. The data read policy applicable to the user group can be determined through the subject policy item, thereby simplifying the authorization operation of related read permissions.

[0188] c) Role: A role is a set of permissions defined according to the user's job function.

[0189] d) Organizational Unit: An organizational unit can be mapped to a department, subsidiary, project family, etc. of an enterprise or organization. Users can be assigned to an organizational unit.

[0190] For example, the Subject Policy Item can be configured using the following code:

[0191] Based on any of the above examples, one or more data reading policies can be configured for third-party data.

[0192] There are many ways to express the data reading strategy.

[0193] For example, when it comes to applications such as data management systems and the business programs therein, data reading strategies can be expressed in open digital rights language (ODRL), extensible markup language (XML), JSON, YAML, etc.

[0194] When it comes to users (such as data providers), text can be used to express it so that users can understand it more easily.

[0195] For example, when expressing a data read policy through text, you can use the format of "[Allow / Deny] [Subject] to use [Amount] of data for [Purpose] at [Time]", such as "Allow XXX to use no more than 1GB of avatar image data for training within 3 months".

[0196] It can be seen that the data reading strategy can be flexibly configured from one or more aspects through one or more strategy items. The configuration method is clear and easy to implement, which can improve configuration efficiency and facilitate understanding by data users and data providers, thereby facilitating negotiation and unification between data users and data providers.

[0197] Based on the above strategy, metadata information corresponding to each data can be recorded in the storage node.

[0198] For example, the specific content of the metadata information of each data may be as shown in Table 2.

[0199] Table 2: Metadata information of data stored in storage nodes

[0200] In this way, the use of data can be managed by using the metadata information of the data in the storage node. The specific management method can refer to the relevant embodiments such as the processing stage of using the strategy to read data, which will not be repeated here.

[0201] 3. Use strategies to read data.

[0202] In an embodiment of the present application, in an actual application scenario, the data user's reading operation on the third-party data in the storage node can be managed according to the data reading policy and / or the data permission policy.

[0203] Specifically, as shown in FIG. 7 , in some embodiments, the method includes steps 701 - 703 .

[0204] Step 701: The execution node runs the business program.

[0205] Business programs are used to execute the business of data users.

[0206] In the embodiment of the present application, the execution node is used to run the business program to execute the business of the data user.

[0207] The specific functions of this business program are not limited here.

[0208] For example, in the AI ​​model training scenario, the business program is used to train the AI ​​model. For example, resources such as the GPU cluster of the AI ​​platform and the training data stored in the storage node can be called to train the AI ​​model.

[0209] In some examples, the execution node can be created according to instructions from a management node in the data management system and start executing the business program.

[0210] For example, specifically, the management node may receive an instruction from a data user to instruct the execution of a specified service.

[0211] After receiving the instruction, the management node may initialize the execution node and instruct the execution node to execute the business program to perform the business of the data user.

[0212] After the execution node completes the current execution of the business program, the relevant operating data stored in the execution node about this execution, such as temporary data, etc., can be deleted, but the execution node can be retained to execute other businesses of the data user, or execute the business program again in the future; or, after the execution node completes the current execution of the business program, the execution node can be destroyed.

[0213] Step 702: The execution node sends a data read request to the storage node.

[0214] The data read request indicates that the running service program requests to read first data from the storage node, where the first data is third-party data.

[0215] In an embodiment of the present application, when an execution node needs to use first data belonging to third-party data during the execution of a business program, it can generate and send a data read request to the storage node.

[0216] It can be seen that the execution node's use of third-party data is controlled by the storage node, that is, the execution node cannot obtain third-party data without being monitored. Compared with the traditional data control method of controlling the use of third-party data through business programs that use third-party data, the solution of the embodiment of the present application can better monitor business programs that use third-party data, thereby better ensuring that third-party data is not abused or leaked.

[0217] Step 703: The storage node generates feedback information according to the data read request and the data read policy for the third-party data, and sends the feedback information to the execution node.

[0218] The storage node may store one or more data read policies, and in some examples, may also store data permission policies.

[0219] In the embodiment of the present application, the data read policy for third-party data can be matched according to the data read request, and the data permission policy for third-party data can also be matched. Depending on the different matching results, the specific content of the feedback information may be different.

[0220] For example, when it is determined based on the matching result that data reading is permitted, the feedback information may include the data that is permitted to be read; and when it is determined based on the matching result that data reading is not permitted, the feedback information may indicate that the data reading operation of the data read request is rejected.

[0221] The following is an exemplary introduction to the matching process related to data permission policy and / or data reading policy.

[0222] 1. Match data permission policies.

[0223] Specifically, in some embodiments, the storage node further includes a data permission policy, which is used to determine the read and write permissions for the data based on the source of the data, wherein the data permission policy indicates that the data user has read permission for third-party data from the data provider;

[0224] Before the storage node generates feedback information according to the first data and the data reading policy regarding the third-party data and sends the feedback information to the execution node, the method further includes:

[0225] The storage node determines, according to the data permission policy and based on the fact that the first data belongs to third-party data, that the business program has read permission for the first data.

[0226] In the embodiment of the present application, the relevant content of the data permission policy can refer to the relevant content of the above-mentioned configuration policy part, which will not be repeated here.

[0227] The data permission policy may be a system-level policy of the data management system, so that basic management of the read and write permissions of the data in the storage node can be performed through the system-level policy.

[0228] In an embodiment of the present application, the data permission policy may include indication information indicating that the data user has read permission for third-party data from the data provider. In this way, after determining that the user has read permission for the first data, the data can be read according to the data read policy.

[0229] It can be seen that in the embodiment of the present application, the reading operation of data in the storage node can be managed more comprehensively through at least two levels of strategies.

[0230] 2. Match data reading strategies.

[0231] Specifically, in some embodiments, the storage node includes one or more data read policies regarding third-party data;

[0232] Based on the data read request and the data read policy for third-party data, the storage node generates feedback information and sends it to the execution node, including:

[0233] The storage node matches the data read request with a policy item of at least one data read policy among the one or more data read policies;

[0234] If there is a data read policy that matches the data read request, feedback information is generated according to the operation in the data read policy that matches the data read request and the feedback information is sent to the execution node.

[0235] In the embodiment of the present application, the data reading policy includes policy items and operations determined based on the policy items.

[0236] There may be one or more data read policies to be matched. When there are multiple data read policies to be matched, the multiple data read policies may be matched sequentially. Specifically, matching may be performed based on the policy items in the data read policies. The multiple data read policies are matched sequentially until a matching data read policy is found, and feedback information is generated based on the operation in the data read policy. Alternatively, until it is determined that all data read policies do not match, feedback information may be generated based on a default operation, or feedback information indicating that data read is denied may be generated.

[0237] In addition, in some examples, in order to better manage the data reading operations of the execution node, the execution node itself can be prevented from obtaining the permission to interact with the storage node. Instead, the data management system manages the permission for the execution node to interact with the storage node. Only after the data management system authorizes the execution node so that the execution node obtains the permission to interact with the storage node can the execution node send a data reading request to the storage node.

[0238] Specifically, in some embodiments, the data management system includes a management node;

[0239] After the management node obtains the running instructions for the business program, it creates access credentials to the storage node;

[0240] The management node instructs the execution node to run the business program and passes the access credential information to the execution node;

[0241] The execution node sends a data read request to the storage node, including:

[0242] The execution node sends a data read request to the storage node based on the access credential information.

[0243] The management node can be considered a system-level node in the data management system. This management node manages the permissions for execution nodes to interact with storage nodes. The data management system uses the management node to manage storage node access credentials at the system level, rather than requiring execution nodes to independently apply for storage node access credentials. This prevents data users from improperly accessing third-party data through execution nodes.

[0244] As can be seen, in the embodiments of the present application, the management node can manage the execution node's access credentials to the storage node. In other words, the management node can manage the execution node's permissions to interact with the storage node. This prevents the execution node from independently obtaining access to the storage node, preventing the execution node from reading and using third-party data on the storage node without permission, thereby preventing the execution node from leaking and improperly using third-party data.

[0245] The specific form of the access credential can be various and is not limited here.

[0246] Exemplarily, the access credential may be a storage session, specifically, the access credential may be uniquely represented by an identifier of the storage session, or the access credential may be a designated key or other forms of credentials.

[0247] FIG8 is a schematic diagram showing an exemplary process of information interaction among a management node, an execution node, and a storage node.

[0248] In the example of FIG8 , the following steps may be included:

[0249] 1) The data user sends a start-up instruction to the management node to instruct the execution of the specified business through the business program.

[0250] 2) After receiving the start instruction, the management node may send instruction information to the storage node to instruct the storage node to create a storage session.

[0251] The indication information may include the ID of the isolated operating environment where the management node is located (for example, the ID of the development space), the subject corresponding to the storage session, and the purpose corresponding to the storage session, so as to request that the subject obtain permission to read data for this purpose from the storage node through the storage session.

[0252] 3) After receiving the indication information, the storage node may return the ID of the storage session to the management node, so that the subject of the corresponding subject type may obtain the permission to read the data for that purpose from the storage node through the ID of the storage session.

[0253] 4) After receiving the ID of the storage session, the management node instructs the execution node to run the business program and passes the ID of the storage session to the business program, for example, through parameters, environment variables, etc.

[0254] 5) When the execution node runs the business program, it requests the storage node to list the files included in the data of the storage node according to the ID of the storage session.

[0255] 6) The storage node returns the file list without verifying the policy.

[0256] 7) When running the business program, the execution node sends a data read request to the storage node according to the ID of the storage session. The data read request indicates that the running business program requests to read first data from the storage node, and the first data is third-party data.

[0257] 8) The storage node reads the metadata of the first data according to the data read request.

[0258] 9) The storage node verifies the policy based on the metadata of the first data, the subject corresponding to the storage session, the purpose corresponding to the storage session, etc., for example, verifies the data permission policy and then verifies the data reading policy.

[0259] 10) Return feedback information based on the matching results.

[0260] The matching results may be as follows:

[0261] A. If a data reading strategy does not match, determine the next data reading strategy;

[0262] B. If a data read strategy matches, then:

[0263] (a) If the operation in the data read policy is allowed (ALLOW), the data read policy is determined to be a matching policy, and the file content allowed to be read is fed back according to the data read policy, and subsequent policy matching operations are stopped.

[0264] (b) If the operation in the data reading policy is deny (DENY), access to the first data is denied, and subsequent policy matching operations are stopped.

[0265] C. If all do not match, the storage node may adopt a default operation, which may be specified by the data management system.

[0266] Specifically, determining whether a data reading policy matches can be achieved based on policy items of the data reading policy.

[0267] For example, a data reading policy includes a data policy item, a usage policy item, a quantity policy item, a time policy item, and a subject policy item.

[0268] The example matching method for each policy item is as follows:

[0269] A. Data policy items:

[0270] (a) Path filtering: The file path specified in the data read request must match the specified conditions;

[0271] (b) Selection column: does not participate in matching, but is applied when reading content;

[0272] (c) Row filtering: does not participate in matching, but is applied when reading content.

[0273] B. Usage strategy items:

[0274] The purpose specified in the data read request must be in the list of purposes defined in the purpose policy item.

[0275] C.Quantity strategy items:

[0276] The control object of the data reading policy can be the data itself (DATA) or the file (FILE) in the data, which is defined by PolicyType. The PolicyType in the metadata of the first data can be DATA or FILE.

[0277] (a) When the PolicyType in the metadata is DATA:

[0278] Use Count: Calculates the number of times the first data is used according to the calculation rules specified in the quantity policy item to determine whether it exceeds the range. If a piece of data is read multiple times within a storage session, it is counted as only one use.

[0279] (b) When the PolicyType in the metadata is FILE:

[0280] ①Number of lines: does not participate in matching, but is applied when reading content;

[0281] ② Storage size: does not participate in matching, but is applied when reading content.

[0282] D. Time strategy items:

[0283] The current time must be within the usage time range.

[0284] E. Subject policy items:

[0285] The subject corresponding to the data read request can match the definition of the subject policy item, such as the subject's userId is in the userId defined in the subject policy item.

[0286] If the storage node determines that data reading is allowed based on the matching result, the data that is allowed to be read is returned.

[0287] When a storage node returns data that is allowed to be read, it is necessary to control the read content based on the final matching data read policy. This can be done in one or more of the following ways:

[0288] Select columns: Eliminate unselected columns:

[0289] Row filtering: Filter the content:

[0290] Number of rows: limit the maximum number of rows that can be read;

[0291] Storage size: Limits the maximum size that can be read.

[0292] In this way, the storage node can complete the feedback of the data read request.

[0293] In some embodiments, in order to ensure the validity of the access credential, the access credential is deleted after the execution node stops running the service program.

[0294] In this way, each time an execution node starts a business program, it must re-acquire permission to interact with the storage node from the management node. This prevents execution nodes from independently obtaining access to storage nodes, nor can they abuse existing access rights. This effectively ensures controllability of each execution node's access to storage nodes, preventing execution nodes from reading and using third-party data on storage nodes without permission, thereby preventing the leakage and improper use of third-party data by execution nodes.

[0295] Furthermore, in some embodiments, the storage node may record data access behavior to third-party data for easy viewing by the data provider.

[0296] Specifically, in some embodiments, after step 703, the method further includes:

[0297] The storage node records data access behaviors related to data read requests.

[0298] Data access behavior includes access behavior in which data reading is successfully achieved, and may also include access behavior in which data reading fails.

[0299] In the embodiment of the present application, the data access behavior may include one or more information of a data read request, a user related to the data read request, and a response corresponding to the data read request (such as feedback information, etc.).

[0300] Illustratively, the data access behavior may include one or more of the following information:

[0301] Access time, user (including data user and / or data provider), purpose, operation (such as read operation or write operation), data, file, number of rows, storage size.

[0302] For example, the access record may be as shown in Table 3.

[0303] Table 3: Access Records

[0304] After the storage node stores the information of the above data access behavior, the data provider can conveniently and flexibly view the usage of the third-party data stored in the storage node.

[0305] Specifically, in some embodiments, the method further includes:

[0306] The storage node obtains the query request input by the data provider, which is used to request access records related to third-party data;

[0307] The storage node outputs access records, which include data access behaviors related to data read requests.

[0308] In an embodiment of the present application, a data provider may input a query request to a data management system based on an API through a client or the like to request access records related to third-party data provided by the data provider. After receiving the query request, the storage node may query the stored data access behavior information related to the third-party data based on the identifier of the data provider and / or the identifier of the third-party data, to obtain and output access records to the third-party data to the data provider and / or other devices. In this case, the access record may include data access behavior related to the data read request.

[0309] In this way, data providers can easily query the access and use of third-party data by data users through storage nodes, making the use of third-party data transparent.

[0310] In addition, data providers can promptly detect abnormal use of third-party data and adjust data access policies. Data providers can also initiate a policy negotiation process with data users through the data management system, using the data management system as an intermediary platform to negotiate new data access policies with data users to meet the requirements of both data providers and data users.

[0311] 4. Derivative data management.

[0312] In an embodiment of the present application, when an execution node is running a business program, after reading third-party data from a storage node, derivative data may be generated in the process of applying the third-party data.

[0313] In order to ensure the security of the data, in the embodiment of the present application, the derived data can be tracked and managed to avoid leakage of the derived data.

[0314] Specifically, as shown in FIG7 , in some embodiments, the method further includes:

[0315] Step 704: During the execution of the service program, the execution node sends the data to be stored to the storage node.

[0316] Step 705 : After receiving the data to be stored, the storage node determines and stores a derivative link of the data to be stored based on the third-party data read from the storage node by the execution node within a specified time period.

[0317] The specified time period includes the time period during which the execution node runs the business program. The derivative link is used to indicate data in the storage node for generating data to be stored.

[0318] In the embodiment of the present application, there may be various situations for the data to be stored.

[0319] For example, in one example, the execution node may detect derivative data generated based on third-party data read from the storage node during the execution of the business program, and use the derivative data as data to be stored.

[0320] In another example, while executing a business program, an execution node may generate temporary data and persistent data that requires persistent storage. Temporary data can be stored within the execution node, for example, in its cache or memory. Persistent data, however, must be stored by a storage node. Therefore, the execution node can send the persistent data as data to be stored to the node to be stored. In this example, there is no actual verification of whether the data to be stored is actually generated based on third-party data.

[0321] After receiving the data to be stored, the storage node may assume that the data to be stored is likely to be generated based on the third-party data read by the execution node from the storage node within the specified time period. Therefore, based on the third-party data read by the execution node from the storage node within the specified time period, the derivative link of the data to be stored is determined and stored.

[0322] The specified time period may include a time period between the time when the execution node starts running the service program and the time when the storage node receives the data to be stored.

[0323] For example, in the flowchart shown in FIG9 , after the management node instructs the execution node to run the business program and passes the ID of the storage session to the execution node as an access credential, the following steps may be included:

[0324] The execution node reads data A from the storage node. Data A is third-party data.

[0325] The storage node records access behaviors related to data A, for example, the storage session ID, data A information, time, etc.

[0326] The execution node reads data B from the storage node. Data B is third-party data.

[0327] The storage node records access behaviors related to data B, for example, the storage session ID, data B information, time, etc.

[0328] The execution node writes data C to the storage node. This data C can be considered as data to be stored.

[0329] Based on the recorded access behavior, the storage node can obtain the third-party data read by the execution node from the storage node within the specified time period after the ID of the current storage session is created and before the current write operation;

[0330] Based on the information of the third-party data, the information of the "derived from" field in the metadata of data C in the storage node is updated to indicate that data C is derived from data A and data B;

[0331] The storage node determines and stores the derived links of data C;

[0332] The storage node writes the contents of data C and returns information to the execution node to indicate that data C has been written;

[0333] The execution node reads data D from the storage node. Data D is third-party data, and the related reading operations are not described in detail.

[0334] The execution node writes data E to the storage node, and the related read operations are not repeated here.

[0335] After executing the above steps, the storage node may generate and store a derivative link as shown in FIG10 , wherein data C is derived from data A and data B, and data E is derived from data A, data B, data C, and data D.

[0336] Exemplarily, the derived links may be recorded using the fields shown in Table 4.

[0337] Table 4: Fields of derived links

[0338] Among them, in the derivative link of data C, data C can be considered as the target data, and data A and data B are the source data of data C.

[0339] In addition, within a development space, after the business program in the execution node is executed multiple times, the related derivative links can be automatically merged and updated. As shown in Figure 11, the derivative links related to storage session 1 and the derivative links related to storage session 2 can be merged.

[0340] When a data provider or data user views the derivative link of data A, the derivative link and related derivative data after data A can be found based on the record of the derivative link.

[0341] In addition, data providers may request deletion of third-party data provided by the data provider and may also request deletion of data derived from such third-party data.

[0342] Specifically, a data provider can initiate a deletion request to a storage node. The storage node can also provide a buffer period. For example, a data user can receive a deletion request notification and then review all derivative links corresponding to the third-party data to be deleted, thereby adjusting their business procedures in a timely manner. After the buffer period expires, the data management system can delete the third-party data and related derivative data based on these derivative links and notify the data user. If the data user cannot complete the deletion within the buffer period, they can request an extension. Upon the data provider's consent, the new deadline will apply.

[0343] 5. Stop running business procedures.

[0344] Specifically, in some embodiments, the method further includes:

[0345] After the execution node stops running the business program, the operation data related to the business program in the execution node is deleted.

[0346] After the execution node completes the execution of the business program, the relevant operating data stored in the execution node about this execution, such as temporary data, can be deleted, but the execution node can be retained to execute other businesses of the data user; or the business program can be executed again later; or, after the execution node completes the execution of the business program, the execution node can be destroyed.

[0347] In this way, it can effectively ensure that the traces of third-party data used in the process of running business programs are deleted, and avoid the execution nodes from unreasonably retaining the third-party data used in the process of running business programs, thereby ensuring that the third-party data is not leaked or abused.

[0348] The data management method provided by the embodiment of the present application has been introduced above from multiple aspects. The data management system provided by the embodiment of the present application will be introduced below with reference to the accompanying drawings.

[0349] As shown in Figure 12, an embodiment of the present application provides a data management system 120, which is set in at least one data center. The data management system 120 is located in an isolated operating environment, and the isolated operating environment is connected to the data provider and the data user respectively through the application programming interface API. The data management system 120 includes an execution node 1201 and a storage node 1202. The storage node 1202 stores the third-party data of the data provider.

[0350] Execution node 1201 is used to:

[0351] Run business programs, which are used to execute the business of data users;

[0352] Sending a data read request to the storage node, where the data read request indicates that the running business program requests to read first data from the storage node, where the first data is third-party data;

[0353] The storage node 1202 is configured to generate feedback information according to a data read request and a data read policy regarding third-party data, and send the feedback information to an execution node.

[0354] Optionally, the system 120 further includes a management node 1203;

[0355] The management node 1203 is used to monitor the access behavior of data users to the data management system through the API, and monitor the data output behavior of the data management system through the API.

[0356] Optionally, the data reading policy is determined by the data provider. The data reading policy includes policy items and operations determined based on the policy items. The policy items include one or more of the following items:

[0357] Data policy items, usage policy items, quantity policy items, time policy items, and subject policy items;

[0358] The data policy item is used to describe the data range that can be read from third-party data;

[0359] The usage policy item is used to describe the tasks in the business process that can use third-party data;

[0360] The quantity policy item is used to describe the usage threshold of third-party data;

[0361] The time policy item is used to describe the time range for using third-party data;

[0362] The subject policy item is used to describe the subject types to which data read policies can be applied.

[0363] Optionally, the storage node 1202 includes one or more data read policies regarding third-party data;

[0364] Storage node 1202 is used for:

[0365] matching the data read request with a policy item of at least one data read policy among the one or more data read policies;

[0366] If there is a data read policy that matches the data read request, feedback information is generated according to the operation in the data read policy that matches the data read request and the feedback information is sent to the execution node.

[0367] Optionally, the storage node 1202 further includes a data permission policy, which is used to determine the read and write permissions for the data based on the source of the data. The data permission policy indicates that the data user has read permission for third-party data from the data provider.

[0368] The storage node 1202 is also used for:

[0369] According to the data permission policy, based on the fact that the first data belongs to third-party data, it is determined that the business program has read permission for the first data.

[0370] Optionally, the management node 1203 is used to:

[0371] After obtaining the running instructions for the business program, create access credentials to the storage node;

[0372] Instruct the execution node to run the business program and pass the access credential information to the execution node;

[0373] Execution node 1201 is used to:

[0374] Based on the access credential information, a data read request is sent to the storage node.

[0375] Optionally, the execution node 1201 is further configured to delete the access credential after the execution node stops running the service program.

[0376] Optionally, the storage node 1202 is further configured to record data access behaviors related to data read requests.

[0377] Optionally, the storage node 1202 is further configured to:

[0378] Obtaining query requests input by data providers, where the query requests are used to request access records related to third-party data;

[0379] Output access records, which include data access behaviors related to data read requests.

[0380] Optionally, the execution node 1201 is further configured to: send data to be stored to the storage node during the execution of the business program;

[0381] The storage node 1202 is also used to: after receiving the data to be stored, determine and store the derivative link of the data to be stored based on the third-party data read by the execution node from the storage node within a specified time period, the specified time period includes the time period during which the execution node runs the business program this time, and the derivative link is used to indicate the data in the storage node used to generate the data to be stored.

[0382] Optionally, the execution node 1201 is further configured to: after the execution node stops running the business program, delete the running data related to the business program in the execution node.

[0383] In the embodiments of the present application, the execution nodes, storage nodes, and management nodes can all be implemented via software or hardware. For example, the implementation of the execution node will be described below using the execution node as an example. Similarly, the implementation of the storage node and management node can refer to the implementation of the execution node.

[0384] As an example of a software functional unit, a module may include code running on a computing instance. The computing instance may include at least one of a physical host (computing device), a virtual machine, and a container. Furthermore, the computing instance may be one or more. For example, the execution node may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed in the same region or in different regions. Furthermore, the multiple hosts / virtual machines / containers used to run the code may be distributed in the same availability zone (AZ) or in different AZs, each AZ including one data center or multiple geographically close data centers. Typically, a region may include multiple AZs.

[0385] Similarly, multiple hosts / virtual machines / containers running the code can be distributed within the same virtual private cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Cross-region communication between two VPCs within the same region, or between VPCs in different regions, requires a communication gateway within each VPC to interconnect the VPCs.

[0386] As an example of a hardware functional unit, a module may include at least one computing device, such as a server. Alternatively, the execution node may be implemented using a central processing unit (CPU), an application-specific integrated circuit (ASIC), or a programmable logic device (PLD). The PLD may be a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), a generic array logic (GAL), a data processing unit (DPU), a neural network processing unit (NPU), a system on chip (SoC), an offload card, an accelerator card, or any combination thereof.

[0387] The multiple computing devices included in an execution node can be distributed in the same region or in different regions. They can also be distributed in the same AZ or in different AZs. Similarly, they can be distributed in the same VPC or across multiple VPCs. These multiple computing devices can be any combination of servers, ASICs, PLDs, CPLDs, FPGAs, GALs, DPUs, NPUs, SoCs, offload cards, accelerator cards, and other computing devices.

[0388] It should be noted that, in other embodiments, the execution node can be used to execute any step in the data management method, the storage node can be used to execute any step in the data management method, and the management node can be used to execute any step in the data management method. The steps that the execution node, storage node, and management node are responsible for implementing can be specified as needed. The full functions of the data management system are realized by respectively implementing different steps in the data management method through the execution node, storage node, and management node.

[0389] The present application also provides a computing device 130. As shown in Figure 13, computing device 130 includes a bus 132, a processor 134, a memory 136, and a communication interface 138. Processor 134, memory 136, and communication interface 138 communicate with each other via bus 132. Computing device 130 can be a server or a terminal device. It should be understood that this application does not limit the number of processors and memories in computing device 130.

[0390] Bus 132 may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, among others. Buses may be classified as address buses, data buses, control buses, and the like. For ease of illustration, FIG13 shows a single line, but this does not imply a single bus or type of bus. Bus 134 may include a pathway for transmitting information between various components of computing device 130 (e.g., memory 136, processor 134, and communication interface 138).

[0391] The processor 134 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0392] The memory 136 may include volatile memory, such as random access memory (RAM). The memory 136 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0393] The memory 136 stores executable program code, and the processor 134 executes the executable program code to respectively implement the functions of the aforementioned execution node and storage node, thereby implementing the data management method applied to the computing device cluster in the above-mentioned embodiment. That is, the memory 136 stores instructions for executing the data management method applied to the computing device cluster in the above-mentioned embodiment.

[0394] The communication interface 138 uses a transceiver module such as, but not limited to, a network interface card or a transceiver to implement communication between the computing device 130 and other devices or a communication network.

[0395] Embodiments of the present application also provide a computing device cluster. The computing device cluster includes at least one computing device. The computing device can be a server, such as a central server, an edge server, or a local server in a local data center. In some embodiments, the computing device can also be a terminal device such as a desktop computer, a laptop computer, or a smartphone.

[0396] As shown in Figure 14, the computing device cluster includes at least one computing device 130. The memory 136 of one or more computing devices 130 in the computing device cluster may store the same instructions for executing the data management method.

[0397] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the data management method. In other words, the combination of one or more computing devices 130 can jointly execute the instructions for executing the data management method.

[0398] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster can store different instructions, each for executing a portion of the functions of the data management method. In other words, the instructions stored in the memory 136 in different computing devices 130 can implement the functions of one or more modules in the execution node and the storage node.

[0399] In some possible implementations, one or more computing devices in a computing device cluster may be connected via a network. The network may be a wide area network (WAN) or a local area network (LAN), etc. FIG. 15 shows a possible implementation. As shown in FIG. 15 , two computing devices 130A and 130B are connected via a network. Specifically, the network is connected via a communication interface in each computing device. In this type of possible implementation, the memory 136 in the computing device 130A may store instructions for executing the functions of an execution node. At the same time, the memory 136 in the computing device 130B may store instructions for executing the functions of a storage node. Alternatively, the memory 136 in the computing device 130A may store instructions for executing part of the functions of a storage node. At the same time, the memory 136 in the computing device 130B may store instructions for executing another part of the functions of a storage node.

[0400] It should be understood that the functionality of computing device 130A shown in FIG15 may also be performed by multiple computing devices 130. Similarly, the functionality of computing device 130B may also be performed by multiple computing devices 130.

[0401] The present application also provides another computing device cluster. The connection relationship between the computing devices in this computing device cluster can be similar to the connection relationship between computing device clusters in Figures 14 and 15. However, the memory 136 in one or more computing devices 130 in this computing device cluster can store the same instructions for executing the data management method.

[0402] In some possible implementations, the memory 136 of one or more computing devices 130 in the computing device cluster may also store partial instructions for executing the data management method. In other words, the combination of one or more computing devices 130 can jointly execute the instructions for executing the data management method.

[0403] It should be noted that the memory 136 in different computing devices 130 in the computing device cluster can store different instructions for executing portions of the data management method. In other words, the instructions stored in the memory 136 in different computing devices 130 can implement the functions of one or more modules in the execution node and the storage node.

[0404] Embodiments of the present application also provide a computer program product including instructions. The computer program product may be software or a program product including instructions that can be run on a computing device or stored on any available medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to execute the data management method.

[0405] The present application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that can be stored by a computing device or a data storage device such as a data center that contains one or more available media. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the data management method.

[0406] The present application also provides a chip system, which includes a processor configured to implement the steps performed by the computing device cluster. In one possible design, the chip system may also include a memory configured to store necessary program instructions and data. The chip system may be composed solely of a chip or may include a chip and other discrete components.

[0407] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0408] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of units is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0409] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0410] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0411] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

Claims

1. A data management method, characterized in that, Applied to a data management system, the data management system is set up in at least one data center, the data management system is located in an isolated operating environment, the isolated operating environment is connected to a data provider and a data user respectively through an application programming interface (API), the data management system includes an execution node and a storage node, the storage node stores third-party data of the data provider, and the method includes: The execution node runs a business program, and the business program is used to execute the business of the data user; The execution node sends a data reading request to the storage node, and the data reading request indicates that the running business program requests to read first data from the storage node, and the first data belongs to the third-party data; The storage node generates feedback information according to the data reading request and a data reading strategy for the third-party data, and sends the feedback information to the execution node.

2. The method according to claim 1, wherein The method further includes: Monitoring the access behavior of the data user to the data management system through the API, and monitoring the data output behavior of the data management system through the API.

3. The method according to claim 1 or 2, characterized in that, The data reading strategy is determined by the data provider, and the data reading strategy includes policy items and operations determined based on the policy items. The policy items include one or more of the following items: Data policy item, usage policy item, quantity policy item, time policy item, subject policy item; Among them, the data policy item is used to describe the data range that can be read in the third-party data; The usage policy item is used to describe the tasks in the business program that can use the third-party data; The quantity policy item is used to describe the usage quantity threshold of the third-party data; The time policy item is used to describe the usage time range of the third-party data; The subject policy item is used to describe the subject type to which the data reading strategy can be applied.

4. The method according to claim 3, wherein The storage node includes one or more data reading strategies for the third-party data; The storage node generates feedback information according to the data reading request and a data reading strategy for the third-party data, and sends the feedback information to the execution node, including: The storage node matches the data reading request with the policy items of at least one data reading strategy among the one or more data reading strategies; If there is a data reading strategy that matches the data reading request, the storage node generates the feedback information according to the operation in the data reading strategy that matches the data reading request, and sends the feedback information to the execution node.

5. The method according to any one of claims 1 to 4, characterized in that The storage node further includes a data permission policy, and the data permission policy is used to determine the read and write permissions for the data according to the data source. Among them, the data permission policy indicates that the data user has read permission for the third-party data from the data provider; Before the storage node generates feedback information according to the first data and a data reading strategy for the third-party data, and sends the feedback information to the execution node, it further includes: The storage node determines that the service program has read permission for the first data based on the data permission policy and that the first data belongs to the third-party data.

6. The method according to any one of claims 1-5, characterized in that The data management system includes a management node, and the method further includes: After obtaining a running instruction for the service program, the management node creates an access credential for the storage node; The management node instructs the execution node to run the service program and transfers information about the access credential to the execution node; The execution node sends a data read request to the storage node, including: Based on the information about the access credential, the execution node sends the data read request to the storage node.

7. The method according to claim 6, wherein The method further includes: After the execution node stops running the service program, the access credential is deleted.

8. The method according to any one of claims 1-7, characterized in that, After the storage node generates feedback information according to the first data and the data read policy for the third-party data and sends the feedback information to the execution node, it further includes: The storage node records the data access behavior related to the data read request.

9. The method according to claim 8, wherein The method further includes: The storage node obtains a query request input by the data provider, and the query request is used to request a query of the access records related to the third-party data; The storage node outputs the access records, and the access records include the data access behavior related to the data read request.

10. The method according to any one of claims 1-9, characterized in that, The method further includes: During the running of the service program, the execution node sends data to be stored to the storage node; After receiving the data to be stored, the storage node determines and stores the derivation link of the data to be stored according to the third-party data read by the execution node from the storage node within a specified time period. The specified time period includes the time period when the execution node runs the service program this time, and the derivation link is used to indicate the data in the storage node for generating the data to be stored.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: After the execution node stops running the service program, the running data related to the service program in the execution node is deleted.

12. A data management system, characterized in that, The data management system is set up in at least one data center. The data management system is located in an isolated running environment, and the isolated running environment is connected to the data provider and the data user respectively through an application programming interface (API). The data management system includes an execution node and a storage node, and the storage node stores the third-party data of the data provider; The execution node is used for: Running a service program, and the service program is used to execute the business of the data user; Sending a data read request to the storage node, and the data read request indicates that the running service program requests to read first data from the storage node, and the first data belongs to the third-party data; The storage node is used for: generating feedback information according to the data read request and the data read policy for the third-party data and sending the feedback information to the execution node.

13. The system according to claim 12, wherein, The system further includes a management node; The management node is used to monitor the access behavior of the data user to the data management system through the API, and monitor the data output behavior of the data management system through the API.

14. The system according to claim 12 or 13, wherein The data reading policy is determined by the data provider, and the data reading policy includes policy items and operations determined based on the policy items. The policy items include one or more of the following items: Data policy item, usage policy item, quantity policy item, time policy item, subject policy item; Among them, the data policy item is used to describe the data range that can be read in the third-party data; The usage policy item is used to describe the tasks in the business program that can use the third-party data; The quantity policy item is used to describe the usage threshold of the third-party data; The time policy item is used to describe the usage time range of the third-party data; The subject policy item is used to describe the type of subject to which the data reading policy can be applied.

15. The system according to claim 14, characterized in that, The storage node includes one or more data reading policies regarding the third-party data; The storage node is used for: Matching the data reading request with the policy items of at least one data reading policy among the one or more data reading policies; If there is a data reading policy that matches the data reading request, generate the feedback information according to the operation in the data reading policy that matches the data reading request and send the feedback information to the execution node.

16. The system according to any one of claims 12-15, characterized in that, The storage node further includes a data permission policy, and the data permission policy is used to determine the read and write permissions for the data according to the source of the data. Among them, the data permission policy indicates that the data user has read permission for the third-party data from the data provider; The storage node is further used for: According to the data permission policy, based on the fact that the first data belongs to the third-party data, determine that the business program has read permission for the first data.

17. The system according to any one of claims 12 - 16, characterized in that, The system further includes a management node; The management node is used for: After obtaining the operation instruction for the business program, create an access credential for the storage node; Instruct the execution node to run the business program and transmit the information of the access credential to the execution node; The execution node is used for: Based on the information of the access credential, send the data reading request to the storage node.

18. The system according to claim 17, wherein The execution node is further used for: after the execution node stops running the business program, delete the access credential.

19. The system according to any one of claims 12-18, wherein The storage node is further used for: recording the data access behavior related to the data reading request.

20. The system according to claim 19, wherein The storage node is further used for: Obtain a query request input by the data provider, where the query request is used to request to query the access records related to the third-party data; Output the access records, where the access records include the data access behavior related to the data reading request.

21. The system according to any one of claims 12-20, wherein: The execution node is further configured to: during the running of the service program, send data to be stored to the storage node; The storage node is further configured to: after receiving the data to be stored, determine and store the derivative link of the data to be stored according to the third-party data read by the execution node from the storage node within a specified time period, where the specified time period includes the time period when the execution node runs the service program this time, and the derivative link is used to indicate the data in the storage node for generating the data to be stored.

22. The system according to any one of claims 12-21, wherein: The execution node is further configured to: after the execution node stops running the service program, delete the running data related to the service program in the execution node.

23. A cluster of computing devices, characterized in that, Comprising at least one computing device, the at least one computing device includes a processor and a memory; The processor is configured to execute the instructions stored in the memory, so that the computing device cluster executes the method according to any one of claims 1-11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program runs on the processor, the processor is caused to execute the method according to any one of claims 1-11.

25. A computer program product comprising instructions, characterized in that, When the instructions are executed by the processor, the method according to any one of claims 1-11 is implemented.

Images