Script-based dataset sharing and collaboration method and system

Abstract

The embodiment of the present invention discloses a script-based dataset sharing and collaboration method and system. Specifically, it includes the following methods: using the connected database or data warehouse as a data source, and obtaining the table in the data source; using a scripting language to develop business logic, converting the table of the data source into a dataset; publishing the dataset as an API; calling the dataset API through code development or third-party application software; after identifying the dataset, querying whether the data cache has cached results, storing the data results in the data cache, and feeding back to the client as an API response. This application solves the problems of existing methods in which data privacy and permission management are not under the control of the publisher and there are security risks; the problems that data users have difficulty in reprocessing the data of the data publisher and publishing it as a new dataset and the data cannot be updated in real time; and the problems that data users only see black box results and cannot see the data generation logic.

Description

Technical Field

[0001] The present invention relates to the field of dataset technology, and in particular to a script-based dataset sharing and collaboration method and system. Background Art

[0002] Enterprises often have data sharing needs, typically within the company or among different employees. For example, the marketing operations department may ask the data warehouse team to export marketing campaign data, or they may share internal data externally, such as an e-commerce platform exporting data to brand owners. However, existing technologies often have numerous drawbacks. For example, once data is exported, data privacy and permission management are beyond the publisher's control, posing security risks. It is difficult for data users to reprocess the publisher's data and publish it as a new dataset, and data cannot be updated in real time. Data users only see black-box results and cannot understand the data generation logic, which can lead to a lack of confidence in the data they use.

[0003] In summary, the existing methods have the following problems:

[0004] 1. Data privacy and permission management are beyond the control of the publisher, posing a security risk.

[0005] 2. Data users face difficulties in reprocessing the data publisher's data and publishing it as a new dataset, and data cannot be updated in real time.

[0006] 3. Data users only see black box results and are unable to see the logic behind data generation. Summary of the Invention

[0007] The purpose of the embodiments of the present invention is to provide a script-based dataset sharing and collaboration method and system to solve the problems in existing methods, such as data privacy and permission management are not under the control of the publisher, which poses a security risk; the problem that it is difficult for data users to reprocess the data of the data publisher and publish it as a new dataset, the problem that the data cannot be updated in real time; and the problem that data users only see black box results and cannot see the data generation logic.

[0008] To achieve the above objectives, an embodiment of the present invention provides a script-based dataset sharing and collaboration method, which includes the following steps:

[0009] Use the connected database or data warehouse as the data source and obtain the tables in the data source;

[0010] Use scripting language to develop business logic and convert data source tables into data sets;

[0011] Publish the dataset as an API and define access rights management rules;

[0012] Calling dataset APIs through code development or third-party application software;

[0013] After identifying the data set, the data cache is checked to see if there is a cached result. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, storing the data result in the data cache, and feeding it back to the client as an API response. Furthermore, the specific process of defining access rights management rules is as follows:

[0014] In the API access rights management configuration interface, set up the corresponding dataset API administrator to assist in management operations and set up a blacklist or whitelist of access rights for the dataset API;

[0015] The dataset API administrator creates a developer token for the dataset API. When the client calls the dataset API, the developer token is used to gain access to the dataset API.

[0016] Furthermore, the audit logs in the dataset API are called through code development or third-party application software. The specific process includes:

[0017] Through the API call audit log interface, you can view and retrieve records of dataset API calls based on filter conditions on the audit log interface and export them into Excel files.

[0018] Furthermore, the filtering conditions include time range, call behavior type, and caller user name.

[0019] Furthermore, the version management page in the dataset API is called through code development or third-party application software. The specific process is as follows:

[0020] On the dataset API version management page, view the edit history of the corresponding dataset. Each edit generates a version. View the definition details of the dataset API in each version.

[0021] Furthermore, the specific process of secondary processing based on the published dataset includes:

[0022] Log in and view published datasets, use published datasets as data sources, develop scripts and define new datasets, publish new datasets as APIs, and define access control.

[0023] A script-based dataset sharing and collaboration system, comprising:

[0024] The acquisition module is used to use the connected database or data warehouse as a data source and obtain the tables in the data source;

[0025] The conversion module is used to develop business logic using scripting languages ​​and convert data source tables into data sets;

[0026] The processing module is used to publish the data set as an API and define access permission management rules;

[0027] Calling module, used to call dataset API through code development or third-party application software

[0028] The identification module is used to identify the data set and query whether the data cache has cached results. If there are cached results, the cached results are directly fed back. If there are no cached results, the data results are obtained by distributing the script to an external cluster for execution, storing the data results in the data cache, and feeding them back to the client as an API response.

[0029] Furthermore, the processing module further includes:

[0030] The access rights management configuration unit is used to set the corresponding dataset API administrator to assist in management operations and set the access rights blacklist or whitelist of the dataset API in the API access rights management configuration interface;

[0031] The creation unit is used to create a developer token for the dataset API through the dataset API administrator. The developer token is used to develop access rights to the dataset API when the client calls the dataset API.

[0032] Furthermore, the calling module further includes:

[0033] The query unit is used to call the audit log interface through the API. It can view and retrieve the records of dataset API calls on the audit log interface according to the filtering conditions and export them into Excel files.

[0034] Furthermore, the calling module further includes:

[0035] The version management unit is used to view the edit history of the corresponding dataset on the dataset API version management page. Each edit will form a version, and the definition details of the dataset API in each version can be viewed.

[0036] The embodiments of the present invention have the following advantages:

[0037] 1. The present invention uses the connected database or data warehouse as the data source and obtains the table in the data source; uses scripting language to develop business logic and converts the table of the data source into a data set; publishes the data set as an API and defines access permission management rules; calls the data set API through code development or third-party application software; after identifying the data set, queries the data cache to see if there is a cached result. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, and the data result is stored in the data cache and fed back to the client as an API response. This solves the problems of existing methods that data privacy and permission management are not under the control of the publisher and there are security risks; the problems that data users have difficulty in reprocessing the data publisher's data and publishing it as a new data set and the data cannot be updated in real time; and the problems that data users only see black box results and cannot see the data generation logic.

[0038] 2. This application provides more secure data privacy protection: the dataset is provided for downstream use in the form of an API, avoiding uncontrolled permissions and privacy leaks caused by data export.

[0039] 3. This application provides more convenient data secondary processing: any user can use an existing dataset to which they have access as a data source for secondary processing, forming a nested script or API, which can not only easily implement data secondary processing but also keep data synchronized with the dataset used.

[0040] 4. This application has clearer data business logic: data users can understand the data generation logic by viewing the script content, and understand the change history of the data generation logic by viewing the script version history. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] To more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are merely exemplary, and those skilled in the art can derive other implementation drawings based on the provided drawings without inventive effort.

[0042] The structures, proportions, sizes, etc. illustrated in this specification are intended only to complement the contents disclosed herein and to facilitate understanding and reading by persons familiar with the art. They are not intended to limit the conditions under which the present invention may be implemented and therefore have no substantive technical significance. Any structural modifications, changes in proportions, or adjustments in sizes, without affecting the efficacy and objectives of the present invention, shall still fall within the scope of the technical contents disclosed herein.

[0043] Figure 1 A method flow chart of a script-based dataset sharing and collaboration method provided by an embodiment of the present invention;

[0044] Figure 2 A framework diagram of a script-based dataset sharing and collaboration system provided by an embodiment of the present invention;

[0045] Figure 3 A diagram of the processing module framework in a script-based dataset sharing and collaboration system provided by an embodiment of the present invention;

[0046] Figure 4 A diagram of the calling module framework in the script-based dataset sharing and collaboration system provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The following describes the implementation of the present invention using specific embodiments. Those skilled in the art will readily understand the other advantages and benefits of the present invention from the disclosure herein. Obviously, the embodiments described are only a portion of the present invention, not all of it. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort are intended to fall within the scope of protection of the present invention.

[0048] Example 1

[0049] Combine Figure 1 This embodiment describes a script-based dataset sharing and collaboration method, which includes the following steps:

[0050] Use the connected database or data warehouse as the data source and obtain the tables in the data source;

[0051] Use scripting language to develop business logic and convert data source tables into data sets;

[0052] Publish the dataset as an API and define access rights management rules;

[0053] Calling dataset APIs through code development or third-party application software;

[0054] After identifying the data set, the data cache is queried to see if there is a cached result. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, the data result is stored in the data cache, and fed back to the client as an API response. In this embodiment, the connected database or data warehouse is used as the data source and the table in the data source is obtained; the business logic is developed using a scripting language to convert the table of the data source into a data set; the data set is published as an API and access rights management rules are defined; the data set API is called through code development or third-party application software; after identifying the data set, the data cache is queried to see if there is a cached result. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, the data result is stored in the data cache, and fed back to the client as an API response. This solves the problems of existing methods such that data privacy and permission control are not under the control of the publisher, which poses a security risk; the data user has difficulty in reprocessing the data publisher's data and publishing it as a new data set, the data cannot be updated in real time, and the data user only sees the black box results and cannot see the data generation logic.

[0055] In a preferred embodiment, the specific process of defining access rights management rules in this embodiment is as follows:

[0056] In the API access rights management configuration interface, set up the corresponding dataset API administrator to assist in management operations and set up a blacklist or whitelist of access rights for the dataset API;

[0057] The dataset API administrator creates a developer token for the dataset API. When the client calls the dataset API, the developer token is used to gain access to the dataset API.

[0058] In this implementation, in the API access rights management configuration interface, the corresponding dataset API administrator is set to assist in management operations and set the dataset API access rights blacklist or whitelist. Common application scenarios are:

[0059] When a business department data analyst uses a third-party BI tool (such as Tableau, PowerBI, etc.) to access the dataset API, the data analyst needs to use a personal account for authentication. Therefore, the dataset API administrator needs to open access rights to the dataset API for the data analyst.

[0060] In this implementation, a dataset API administrator creates a developer token for the dataset API. This token is then used by clients to grant access to the dataset API. A common application involves a business department developing server-side software or data products that require access to the dataset API as a service. Therefore, a dataset API administrator creates a developer token to grant access to the dataset API for the service.

[0061] In this implementation, a script is defined as an executable file written in a specific descriptive language and formatted in a specific format. Caching: This saves computational resources and reduces runtime by storing reusable computational results in memory and accessing them at runtime to avoid repeated computations.

[0062] In a preferred embodiment, this implementation uses code development or third-party application software to call the audit log in the dataset API. The specific process includes:

[0063] Through the API call audit log interface, you can view and retrieve records of dataset API calls based on filter conditions on the audit log interface and export them into Excel files.

[0064] Common applications of this implementation are: To prevent data from being abused, administrators need to regularly conduct compliance audits on data usage. All records of dataset usage can be viewed on this page.

[0065] In a preferred embodiment, the filtering conditions of this implementation include time range, calling behavior type and calling party user name.

[0066] In a preferred embodiment, this implementation method calls the version management page in the dataset API through code development or third-party application software. The specific process is as follows:

[0067] On the dataset API version management page, view the edit history of the corresponding dataset. Each edit generates a version. View the definition details of the dataset API in each version.

[0068] In a preferred embodiment, the specific process of secondary processing based on the published data set in this embodiment includes:

[0069] Log in and view published datasets, use published datasets as data sources, develop scripts and define new datasets, publish new datasets as APIs, and define access control.

[0070] Example 2

[0071] Combine Figures 2 to 4This embodiment describes a script-based dataset sharing and collaboration system, which includes:

[0072] The acquisition module 10 is used to use the connected database or data warehouse as a data source and obtain a table in the data source;

[0073] A conversion module 20 is used to develop business logic using a scripting language and convert the table of the data source into a data set 30;

[0074] Processing module 40, for publishing the dataset as an API and defining access rights management rules;

[0075] Calling module 50, used to call the dataset API through code development or third-party application software

[0076] The identification module 60 is used to query the data cache whether there is a cached result after identifying the data set. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, and the data result is stored in the data cache and fed back to the client as an API response.

[0077] This implementation method uses the connected database or data warehouse as a data source and obtains the tables in the data source; uses a scripting language to develop business logic and converts the tables in the data source into a data set; publishes the data set as an API and defines access rights management rules; calls the data set API through code development or third-party application software; after identifying the data set, queries the data cache to see if there is a cached result. If there is a cached result, the cached result is directly fed back. If there is no cached result, the data result is obtained by distributing the script to an external cluster for execution, and the data result is stored in the data cache and fed back to the client as an API response. This solves the problems of existing methods that data privacy and permission control are not under the control of the publisher and there are security risks; it is difficult for data users to reprocess the data of the data publisher and publish it as a new data set, and the data cannot be updated in real time; and data users only see black box results and cannot see the data generation logic.

[0078] In a preferred embodiment, the processing module 40 of this embodiment further includes:

[0079] The access permission management configuration unit 401 is used to set the corresponding dataset API administrator to assist in management operations and set the access permission blacklist or whitelist of the dataset API in the API access permission management configuration interface;

[0080] The creation unit 402 is configured to create a developer token for the dataset API through a dataset API administrator, and to develop access rights to the dataset API through the developer token when a client calls the dataset API.

[0081] In a preferred embodiment, the calling module 50 of this embodiment further includes:

[0082] The query unit 501 is used to call the audit log interface through the API, view and retrieve the records of the dataset API being called on the audit log interface according to the filtering conditions, and export them into an Excel file.

[0083] In a preferred embodiment, the calling module 50 of this embodiment further includes:

[0084] The version management unit 502 is used to view the editing history of the corresponding dataset on the dataset API version management page. Each edit will form a version, and view the definition details of the dataset API in each version.

[0085] Although the present invention has been described in detail above using general descriptions and specific embodiments, it will be apparent to those skilled in the art that modifications and improvements may be made thereto. Therefore, such modifications and improvements, without departing from the spirit of the present invention, are intended to be within the scope of protection claimed herein.

Claims

1. A script-based dataset sharing and collaboration method, characterized in that: The method comprises the following steps: Use the connected database or data warehouse as the data source and obtain the tables in the data source; Use scripting language to develop business logic and convert data source tables into datasets. Data users can obtain data generation logic by viewing the script content and view the script editing history and version definition details on the version management page. Publish the dataset as an API and define access rights management rules; Calling dataset APIs through code development or third-party application software; After identifying the dataset, the data cache is checked to see if there is any cached result. If so, the cached result is directly fed back. If not, the script is distributed to an external cluster for execution to obtain the data result, which is then stored in the data cache and fed back to the client as an API response. Perform secondary processing based on published datasets. The specific process includes: logging in and viewing published datasets, using published datasets as data sources, developing scripts and defining new datasets, publishing new datasets as APIs, and defining access control.

2. The script-based dataset sharing and collaboration method according to claim 1, wherein: The specific process of defining access rights management rules is as follows: In the API access rights management configuration interface, set up the corresponding dataset API administrator to assist in management operations and set up a blacklist or whitelist of access rights for the dataset API; The dataset API administrator creates a developer token for the dataset API. When the client calls the dataset API, the developer token is used to gain access to the dataset API.

3. The script-based dataset sharing and collaboration method according to claim 1, wherein: The specific process of calling the audit log in the dataset API through code development or third-party application software includes: Through the API call audit log interface, you can view and retrieve records of dataset API calls based on filter conditions on the audit log interface and export them into Excel files.

4. The script-based dataset sharing and collaboration method according to claim 3, wherein: Filtering criteria include time range, call behavior type, and caller username.

5. The script-based dataset sharing and collaboration method according to claim 1, wherein: Calling the version management page in the dataset API through code development or third-party application software involves the following steps: On the dataset API version management page, view the edit history of the corresponding dataset. Each edit generates a version. View the definition details of the dataset API in each version.

6. A script-based dataset sharing and collaboration system, characterized by: The system comprises: The acquisition module is used to use the connected database or data warehouse as a data source and obtain the tables in the data source; The conversion module is used to develop business logic using scripting languages ​​and convert data source tables into data sets. Data users can obtain data generation logic by viewing the script content and view the script editing history and version definition details on the version management page. The processing module is used to publish the data set as an API and define access permission management rules; Calling module, used to call dataset API through code development or third-party application software The identification module is used to identify the data set and then query whether the data cache has cached results. If there are cached results, the cached results are directly fed back. If there are no cached results, the data results are obtained by distributing the script to an external cluster for execution, storing the data results in the data cache, and feeding them back to the client as an API response. It is also used for secondary processing based on the published data set. The specific process includes: logging in and viewing the published data set, using the published data set as a data source, developing scripts and defining new data sets, publishing the new data set as an API, and defining access control.

7. The script-based dataset sharing and collaboration system according to claim 6, wherein: The processing module further includes: The access rights management configuration unit is used to set the corresponding dataset API administrator to assist in management operations and set the access rights blacklist or whitelist of the dataset API in the API access rights management configuration interface; The creation unit is used to create a developer token for the dataset API through the dataset API administrator. The developer token is used to develop access rights to the dataset API when the client calls the dataset API.

8. The script-based dataset sharing and collaboration system according to claim 6, wherein: The calling module also includes: The query unit is used to call the audit log interface through the API. It can view and retrieve the records of dataset API calls on the audit log interface according to the filtering conditions and export them into Excel files.

9. The script-based dataset sharing and collaboration system according to claim 8, wherein: The calling module also includes: The version management unit is used to view the edit history of the corresponding dataset on the dataset API version management page. Each edit will form a version, and the definition details of the dataset API in each version can be viewed.

Images