Data authority management method and device for data acquisition process

Abstract

The invention provides a data permission management method and device for a data acquisition process, and the method comprises the steps: distributing different permissions for the information of each dimension, such as a data source, a directory, a mode, a data table, a data field, a data line, a read operation, a write operation, an authorization starting time, an authorization ending time, a time period interval, and the like, for users with different permission levels in advance; the method comprises the steps of obtaining a data permission list of a user, obtaining a task configuration file corresponding to an ETL task request when the user initiates the ETL task request, obtaining various pieces of to-be-verified information corresponding to permission verification, comparing the to-be-verified information with actual permission information, and if the various pieces of to-be-verified information do not meet the permission range specified by the actual permission information, executing the permission verification. And if not, the data under the unified data management platform is not continuously used, so that the security of the data under the unified data management platform can be guaranteed in a fine-grained and multi-dimensional manner.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data authority management method and device for a data collection process. Background Art

[0002] Due to the different development and application times of databases and the different departments to which they belong, there are often many heterogeneous database application systems running on different software and hardware platforms within an enterprise or organization. The data sources of these database application systems are independent of each other and closed to each other, forming "information islands". In order to break this situation and achieve unified data integration, a unified data management platform for data extraction, conversion, loading, and development is built through technologies such as big data warehouses and data lakes.

[0003] As different departments and users are connected to a unified data management platform, using permission verification tools to verify and manage permissions for different users has become the key to ensuring the security of data under the unified data management platform. However, existing permission verification tools verify and manage user permissions, which is prone to problems such as insufficient data permission control coverage granularity and insufficient coverage dimensions. Summary of the invention

[0004] In view of this, the embodiments of the present application provide a data permission management method and device for the data collection process, which can realize refined permission verification for each user of a unified data management platform, and achieve multi-dimensional protection of the security of data under the unified data management platform.

[0005] In a first aspect, an embodiment of the present application provides a data rights management method for a data collection process, wherein the method comprises: Acquire an ETL task request passed in by a user, wherein the ETL task request carries: user identity information of the user and a unified address identifier of a task configuration file; Based on the unified address identifier, the task configuration file is obtained; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; According to the user identity information of the user, a data permission list of the user is obtained; wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is the permission information pre-assigned to the user for each item of information in the data description information, the permission action information and the permission time dimension information according to the permission level of the user; If the information to be verified meets the authority range specified by the actual authority information, it is determined to execute the ETL task corresponding to the ETL task request; otherwise, the ETL task is refused to be executed and task error information is output.

[0006] In some possible embodiments, the method is applied to a data rights management system, the data rights management system includes: a management platform, an execution platform, the execution platform includes: an ETL task scheduler, the management platform includes: a user interaction interface, and the method further includes: Obtaining data description information, permission action information, and permission time dimension information input by the user on the user interaction interface; Generate the task configuration file based on the input data description information, permission action information, and permission time dimension information, and generate the unified address identifier for the task configuration file; The ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier.

[0007] In some possible embodiments, the execution platform further includes: an ETL executor, wherein the ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier, including: Obtaining the task configuration file according to the unified address identifier through the ETL task scheduler; Obtaining the data permission list of the user from the management platform through the ETL task scheduler according to the user identity information of the user; If the information to be verified meets the authority range specified by the actual authority information, the ETL task scheduler generates an ETL task corresponding to the ETL task request, and schedules the ETL task to the corresponding ETL executor, which executes the ETL task.

[0008] In some possible embodiments, the method further includes: If the information to be verified does not satisfy the authority range specified by the actual authority information, the ETL task scheduler refuses to generate the ETL task corresponding to the ETL task request, and outputs the task error information to the user interaction interface.

[0009] In some possible embodiments, the data rights management system further includes: a business platform, the business platform is pre-installed with: a data source, and the method further includes: Obtaining environment description information of the deployment environment of the ETL executor, and if the environment description information is a trusted environment, sending an ETL instruction to the data source through a first data port, wherein the first data port is a data port preset between the ETL executor and the data source; The data source operates the target data corresponding to the actual authority information through the ETL instruction.

[0010] In some possible embodiments, the service platform is also pre-installed with: a data source adapter, and the method further includes: If the environment description information is not the trusted environment, sending the ETL instruction to the data source adapter through a second data port, wherein the second data port is a data port preset between the ETL executor and the data source adapter; The data source adapter obtains the information to be verified in the task configuration file and the data permission list of the user based on the ETL instruction; Determining, by the data source adapter, whether the information to be verified meets the authority scope specified by the actual authority information; If so, the data source adapter forwards the ETL instruction to the data source, and the data source operates the target data corresponding to the actual permission information through the ETL instruction; If not, the ETL instruction is rejected and forwarded to the data source, and the task error message is output.

[0011] In a second aspect, an embodiment of the present application provides a data rights management device for a data collection process, wherein the device includes: A request acquisition module, used to acquire an ETL task request passed in by a user, wherein the ETL task request carries: user identity information of the user and a unified address identifier of a task configuration file; A task configuration file acquisition module is used to acquire the task configuration file based on the unified address identifier; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; A data permission list acquisition module is used to acquire the data permission list of the user according to the user identity information of the user, wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is the permission information pre-assigned to the user according to the permission level of the user for each item of information in the data description information, the permission action information and the permission time dimension information; The authority verification module is used to determine to execute the ETL task corresponding to the ETL task request if the information to be verified meets the authority range specified by the actual authority information, otherwise refuse to execute the ETL task and output task error information.

[0012] In some possible embodiments, the device is applied to a data rights management system, the data rights management system includes: a management platform, an execution platform, the execution platform includes: an ETL task scheduler, the management platform includes: a user interaction interface; The request acquisition module is used to obtain data description information, permission action information, and permission time dimension information input by the user in the user interaction interface; The task configuration file acquisition module is used to generate the task configuration file based on the input data description information, permission action information, and permission time dimension information, and generate the unified address identifier for the task configuration file; The authority verification module is further configured to generate, through the ETL task scheduler, an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier.

[0013] In some possible embodiments, the execution platform further includes: an ETL executor, wherein the ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier, including: Obtaining the task configuration file according to the unified address identifier through the ETL task scheduler; Obtaining the data permission list of the user from the management platform through the ETL task scheduler according to the user identity information of the user; If the information to be verified meets the authority range specified by the actual authority information, the ETL task scheduler generates an ETL task corresponding to the ETL task request, and schedules the ETL task to the corresponding ETL executor, which executes the ETL task.

[0014] In some possible embodiments, the permission verification module is specifically used to: If the information to be verified does not satisfy the authority range specified by the actual authority information, the ETL task scheduler refuses to generate the ETL task corresponding to the ETL task request, and outputs the task error information to the user interaction interface.

[0015] In some possible embodiments, the data rights management system further includes: a business platform, in which: a data source is pre-installed, and the business platform is used to: Obtaining environment description information of the deployment environment of the ETL executor, and if the environment description information is a trusted environment, sending an ETL instruction to the data source through a first data port, wherein the first data port is a data port preset between the ETL executor and the data source; The data source operates the target data corresponding to the actual authority information through the ETL instruction.

[0016] In some possible embodiments, the service platform is also pre-installed with: a data source adapter, and the service platform is also used for: If the environment description information is not the trusted environment, sending the ETL instruction to the data source adapter through a second data port, wherein the second data port is a data port preset between the ETL executor and the data source adapter; The data source adapter obtains the information to be verified in the task configuration file and the data permission list of the user based on the ETL instruction; Determining, by the data source adapter, whether the information to be verified meets the authority scope specified by the actual authority information; If so, the data source adapter forwards the ETL instruction to the data source, and the data source operates the target data corresponding to the actual permission information through the ETL instruction; If not, the ETL instruction is rejected and forwarded to the data source, and the task error message is output.

[0017] In a third aspect, an embodiment of the present application provides an electronic device, wherein the electronic device includes: Processor; and Memory for storing programs, Wherein, the program includes instructions, and when the instructions are executed by the processor, the processor executes the data authority management method for the data collection process described in the first aspect.

[0018] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a computer to execute the data authority management method for a data collection process described in the first aspect.

[0019] Beneficial effects of this application: The present application provides a data permission management method and device for a data collection process, wherein the method obtains an ETL task request passed in by a user, obtains the user's data permission list based on the user identity information in the ETL task request, so as to obtain the user's actual permission information, and obtains the to-be-verified information corresponding to the ETL task request according to the unified address identifier of the task configuration file in the ETL task request, verifies the to-be-verified information based on the actual permission information, and determines whether the to-be-verified information satisfies the permission scope specified by the actual permission information. If so, it indicates that the ETL task request passed in by the user is in compliance with the pre-defined permission scope, and the corresponding ETL task can be allowed to be executed. If not, it indicates that the ETL task request passed in by the user does not meet the pre-defined permission scope and is refused to be executed, and a task error message is output at this time.

[0020] By selecting the embodiment of the present application, different permissions are assigned to users with different permission levels in advance for information of various dimensions such as data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval, etc. related to data request when the user uses a unified data management platform, and a data permission list of the user is obtained. Then, when the user uses the unified data management platform to read or write data, the user initiates a corresponding ETL task request, and then obtains the data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval waiting for verification information requested by the ETL task request, and compares it with the actual permission information corresponding to each item of information to be verified by the user, to determine whether the information to be verified meets the permission scope specified by the actual permission information. If there is any dissatisfaction, the data under the unified data management platform will not be used any more, which is helpful to ensure the security of each data under the unified data management platform in a fine-grained and multi-dimensional manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Further details, features and advantages of the present application are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which: Figure 1 A schematic diagram of a process flow of a data authority management method for a data collection process provided by an embodiment of the present application is shown; Figure 2 A schematic diagram of a system architecture of a data authority management system for a data collection process provided by an embodiment of the present application is shown; Figure 3 A schematic diagram showing an expression form of an ETL task request provided in an embodiment of the present application is shown; Figure 4 Another schematic diagram of a data authority management method for a data collection process provided in an embodiment of the present application is shown; Figure 5 Another schematic diagram of a data authority management method for a data collection process provided in an embodiment of the present application is shown; Figure 6 A structural block diagram of a data authority management device for a data collection process provided by an embodiment of the present application is shown; Figure 7 A structural block diagram of an exemplary electronic device that can be used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0022] The embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be construed as being limited to the embodiments described herein. Instead, these embodiments are provided to provide a more thorough and complete understanding of the present application. It should be understood that the drawings and embodiments of the present application are only for exemplary purposes and are not intended to limit the scope of protection of the present application.

[0023] It should be understood that the various steps described in the method implementation of the present application can be performed in different orders and / or performed in parallel. In addition, the method implementation may include additional steps and / or omit the steps shown. The scope of the present application is not limited in this respect.

[0024] The term "including" and its variations used in this document are open inclusions, that is, "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one other embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions of other terms will be given in the description below. It should be noted that the concepts of "first", "second", etc. mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] As described in the background technology, existing permission verification tools are prone to problems such as insufficient coverage granularity and insufficient coverage dimensions of data permission control.

[0027] Specifically, the existing permission verification tools are prone to the problem of insufficient coverage granularity of data permission control. It can be understood that the current mainstream ETL permission management tools only cover data permission control at the data table level or field level, and usually cannot implement data row level permission control. For example, suppose that user A is restricted to read data in data table sheet1. At this time, it is assumed that user A can read all data in the data table sheet1. In this way, if there is some sensitive data in sheet1 that user A does not want to access, the existing ETL permission management tools will cause those sensitive data to be accessed by user A, thereby causing data leakage.

[0028] Existing permission verification tools are prone to the problem of insufficient coverage dimensions. It can be understood that when the current ETL permission management tools control data access rights, they do not include dimensions such as time and cycle into the permission management scope. As a result, the backend database will be heavily loaded during certain time periods, but will be unable to control data access requests initiated by users.

[0029] In view of this, the present application provides a data permission management method and device for the data collection process, so as to realize refined permission verification for each user of the unified data management platform, and further realize multi-dimensional protection of the security of data under the unified data management platform. Among them, in the first aspect, the present application provides a data permission management method for the data collection process, which is applied to any electronic device with data permission management function, including but not limited to personal mobile terminals, computers or servers, etc. Figure 1As shown, the method comprises the following steps: S11, obtaining an ETL task request passed in by a user, wherein the ETL task request carries: user identity information of the user and a unified address identifier of a task configuration file; S12. Based on the unified address identifier, the task configuration file is obtained; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; S13. Acquire a data permission list of the user according to the user identity information of the user; wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is permission information pre-assigned to the user according to the permission level of the user for each item of information in the data description information, the permission action information, and the permission time dimension information; S14. If the information to be verified meets the authority range specified by the actual authority information, determine to execute the ETL task corresponding to the ETL task request; otherwise, refuse to execute the ETL task and output task error information.

[0030] The method provided by the embodiment of the present application obtains the ETL task request passed in by the user, obtains the user's data permission list based on the user identity identification information in the ETL task request, so as to obtain the actual permission information of the user, and obtains the to-be-verified information corresponding to the ETL task request according to the unified address identifier of the task configuration file in the ETL task request, verifies the to-be-verified information based on the actual permission information, and determines whether the to-be-verified information satisfies the permission scope specified by the actual permission information. If so, it indicates that the ETL task request passed in by the user is in compliance with the pre-defined permission scope, and the corresponding ETL task can be allowed to be executed. If not, it indicates that the ETL task request passed in by the user does not meet the pre-defined permission scope and is refused to be executed, and a task error message is output at this time.

[0031] By selecting the embodiment of the present application, different permissions are assigned to users with different permission levels in advance for information of various dimensions such as data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval, etc. related to data request when the user uses a unified data management platform, and a data permission list of the user is obtained. Then, when the user uses the unified data management platform to read or write data, the user initiates a corresponding ETL task request, and then obtains the data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval waiting for verification information requested by the ETL task request, and compares it with the actual permission information corresponding to each item of information to be verified by the user, to determine whether the information to be verified meets the permission scope specified by the actual permission information. If there is any dissatisfaction, the data under the unified data management platform will not be used any more, which is helpful to ensure the security of each data under the unified data management platform in a fine-grained and multi-dimensional manner.

[0032] The following is a detailed description of the above steps S11 to S14 with reference to specific examples: In the embodiments of the present application, the data collection process refers to the process of users obtaining data from the underlying database from the user's perspective. Among them, users may refer to various users who use the unified data management platform as described in the background technology. As an implementation method, the unified data management platform is a synonym for various data systems constructed through technologies such as big data warehouses and databases. The unified data management platform can also be referred to as a data rights management system. The core is to use the unified data management platform to uniformly respond to user requests and uniformly manage data corresponding to different data sources. The data collection process refers to the process in which users read or write data from the data stored in various databases under the unified data management platform through the unified data management platform.

[0033] The user can refer to an internal user or an external user. An internal user refers to a user within the organization that develops and operates the unified data management platform, such as employees of various departments within the organization. An external user refers to a user outside the organization. For example, if the main business of the organization is e-commerce, the external user refers to a consumer who consumes the e-commerce services provided by the organization.

[0034] Based on this, in the embodiment of the present application, the data permission management used in the data collection process refers to the user performing permission verification for the user when using the unified data management platform to operate the data under the data management platform. Only users who pass the permission verification can operate the data, and users who fail the permission verification cannot operate the data. In this way, the data security of the data under the data management platform can be ensured. As an implementation method, the unified data management platform can be implemented by using ETL (Extract-Transform-Load) technology to extract, transform, load, and develop data, and permission management can be performed through the self-developed ETL task scheduler, management platform, and execution platform provided by this application.

[0035] In the embodiment of the present application, when a user needs to operate the data in each database under the unified data management platform, he can submit an operation application to the unified data management platform. In the implementation of the present application, the operation application submitted by the user to the unified data management platform is determined as an ETL task request. In some possible embodiments, the data permission management method for the data collection process provided by the present application can be applied to the following: Figure 2 The data rights management system shown in the figure is a software rights management system, which is specifically divided into three subsystems: management platform, execution platform, and business platform. The three subsystems cooperate with each other to execute the data rights management method provided in this application to verify the user's rights.

[0036] Specifically, the management platform is built with a unified data acquisition platform authority configuration management module, and managers can configure the authority of users with different authority levels through the unified data acquisition platform authority configuration management module and the task configuration center to obtain the actual authority information of each user. As an implementation method, the management platform also includes a user interaction interface, and managers can interact with the data authority management system through the user interaction interface to configure the authority of different users. As another implementation method, the manager summarizes and sorts the authority information configured for each user (that is, the actual authority information of each user) to obtain the data authority list of each user, and then stores the data authority list of the user separately with the user's identity information as the index basis. When executing step S13 later, the data authority list of the user can be directly obtained from the corresponding storage path according to the user's identity information. In addition, the management platform can also generate a message queue based on the ETL task requests sent by different users to realize the scheduling of ETL task requests for different authority levels and different authority time dimensions to ensure the load balancing of the entire system.

[0037] In an embodiment of the present application, the actual permission information and the information to be verified are specifically subdivided into three categories: data description information, permission action information, and permission time dimension information. Among them, the data description information is used to describe the storage information of each data under the unified data acquisition platform, specifically including storage information on different dimensions, such as which data source is specifically stored, which data source namespace, which data table, etc. The permission action information is used to describe the action type for each data under the unified data acquisition platform, specifically including: read operation, write operation, access, etc. The permission time dimension information is used to describe the operation time for each data under the unified data acquisition platform, specifically including: start time, end time, time period interval, etc.

[0038] Specifically, as an implementation method, the content of the permission information included in the information to be verified and the user's data permission list in the embodiment of the present application can be as follows: Figure 3 As shown, it includes: permission mask, time configuration expression, authorization start time, and authorization end time. Among them, the permission information can be subdivided into: Data permission definition, data source, directory, mode, data table, data column, and data row.

[0039] The data permission definition includes: user role ID and data source set.

[0040] The data source includes: data source ID, category (data source category), data source permission information, and directory collection.

[0041] The directory contains: directory name, mode collection, and data source directory permission information.

[0042] The mode includes: mode name, data table collection, and mode permission information.

[0043] The data table contains: data table name, row set, column set, and data table permission information.

[0044] The data column contains: data column name and data column permission information.

[0045] The data row contains: split key, start row, end row and permission information of the data row.

[0046] The permission mask is identified by 2-bit binary, the high bit indicates read permission, and the low bit indicates write permission. The time expression and meaning are consistent with the time expression of the Linux system, which is used to represent the validity period corresponding to the data permission. The authorization start time can be stTimestamp, which is a timestamp. Similarly, the authorization end time can be edTimestamp, which is also a timestamp. In this way, the limited period of the data permission is [stTimestamp, edTimestamp).

[0047] When executing step S14 to determine whether it complies with the prescribed scope of authority, as an implementation method, entities can be configured for the above-mentioned data authority definition, data source, directory, mode, data table, data column, and data row. There is an aggregation relationship from the whole (outer layer or the previous layer) to the part (inner layer or the next layer) between each entity in turn. Except for the data authority definition, which is an entity, all other entities can be configured with data authority. The authority definition of the inner entity can be omitted and then inherited to the outer entity. When the inner entity has a specific data authority definition, the authority definition of the outer entity can also be omitted. When both the inner entity and the outer entity data authority exist at the same time, the authority level and validity period of the inner entity cannot exceed that of the outer entity.

[0048] For example, if the permission mask of the outer data source is configured as 10, read-only, then the highest permission of the inner directory, data row, data column and other entities can only be configured as read-only. Similarly, if the validity period of the data permission in the outer schema is: [stTimestamp1, edTimestamp1), then the validity period of the inner data table requires: stTimestamp2>=stTimestamp1, edTimestamp2<=edTimestamp1, that is, the validity period of the schema includes the validity period of the data table.

[0049] In the embodiment of the present application, the data source is a database set up by an organization or enterprise according to actual business, and the data source stores various types of data required for the organization or enterprise to provide normal business services. It can be subdivided into relational databases, non-relational databases, file systems, Web service interfaces, etc. Exemplarily, the data sources that may exist in an e-commerce company include: online transaction databases, customer relationship management (CRM) systems, and logistics systems.

[0050] A directory is a logical container used to organize and manage data stored in a database. The directory uses a unique identifier to distinguish different modes, such as data tables, data views, data storage procedures, etc. This unique identifier can avoid naming conflicts between data sources and help ensure the manageability and security of data sources.

[0051] Schema refers to various entities stored in the database that can be operated, managed, and accessed. Schemas can be subdivided into: data table, data view, index information, stored procedure, trigger, sequence, etc. Data table refers to the most basic schema in the database, usually a two-dimensional structure, consisting of data rows and data columns. Among them, data rows are called records, and data columns are called data fields.

[0052] The permission information configured by the administrator for the user is the same as the dimension of the permission parameters involved in the subsequent verification of the user's permissions, and each item corresponds to each other. For example, during the permission configuration stage, the administrator can configure the data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval and other permission parameters according to the user's user permission level.

[0053] For example, assuming that user A is a high-level user and can perform read or write operations on all data sources, directories, modes, data tables, data fields, and data rows at any time, for user A, the corresponding permissions for data sources, directories, modes, data tables, data fields, data rows, read operations, and write operations are all: allowed. The permission opening time is: always.

[0054] Assume that user B is a low-level user and can only read row 20 of the "Name" mode in data table 1 in data source A from 9:00 to 9:30 every Monday morning, and has no permissions for other data. For user B, the only data permission for the row 20 of data source A-data table 1-name from 9:00 to 9:30 every Monday morning is: Reading is allowed. Except for this row of data, other data cannot be read, written, browsed, or accessed. Except for the period from 9:00 to 9:30 every Monday morning, the data permissions for all data in other time periods are not read, written, browsed, or accessed. The authorization start time is: Monday morning 9:00, the authorization end time is Monday morning 9:30, and the time period interval is: 1 week.

[0055] Among them, the permission information configured by the administrator for the user is the actual permission information of the user, and the data table obtained by summarizing and sorting the actual permission information is the data permission list of the user. As described above, the actual permission information configured by the administrator for the user is the permission information obtained by the administrator in advance according to the permission parameters such as data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval, etc. The permission information can be regarded as the verification standard of permission verification. If the user uses the ETL task request submitted by the data permission management platform, the user fills in the data source, directory, mode, data table, data field, data row, read operation, write operation, authorization start time, authorization end time, time period interval and other information to be verified in the ETL task request submitted by the user, and compares it with the verification standard of the permission verification, and verifies the user. If the information to be verified meets the permission range specified by the actual permission information, it indicates that the user verification of the user has passed, and the ETL task request submitted by the user can continue to be executed.

[0056] In some possible embodiments, as described above, the data rights management system includes: a management platform and an execution platform. The management platform includes a user interaction interface. The execution platform is pre-installed with: an ETL task scheduler and an ETL executor. Figure 2 As shown, the number of ETL task schedulers can be multiple, specifically: ETL task scheduler 1, ETL task scheduler 2, ETL task scheduler 3... ETL task scheduler n. Similarly, the number of ETL executors can also be multiple, specifically ETL executor 1, ETL executor 2, ETL executor 3... ETL executor n. Based on this, when executing step S11 to obtain the ETL task request passed in by the user, it can be achieved through the following steps: Obtaining data description information, permission action information, and permission time dimension information input by the user on the user interaction interface; Generate the task configuration file based on the input data description information, permission action information, and permission time dimension information, and generate the unified address identifier for the task configuration file; The ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier.

[0057] Among them, as a possible implementation method, the user interaction interface displays various parameters such as data source, directory, mode, data table, data field, data row, read operation, write operation, usage time, etc. as checkable options. Users can check the options they need in the user interaction interface according to their own needs. The unified data acquisition platform authority configuration management module can automatically generate a task configuration file based on the options checked by the user, and generate a unified address identifier URL (Uniform Resource Locator) for the task configuration file.

[0058] As another implementation method, the ETL task scheduler may obtain the data description information, permission action information, permission time dimension information and other parameters input by the user from the user interaction interface, and then generate a task configuration file based on the data description information, permission action information, permission time dimension information and other parameters, and generate a unified address identifier for the task configuration file.

[0059] The unified address identifier is a standard resource address on the Internet, and the unified address identifier is used to fully describe the specific location and access method of the resource on the Internet. Thus, when executing step S12, the resource location path can be accessed based on the unified address identifier generated by the management platform or the ETL scheduler to obtain the task configuration file recorded under the URL path.

[0060] As an implementation method, the ETL task scheduler can extract data based on the data description information, permission action information, permission time dimension information, etc. input by the user in the user interaction interface, and then determine the data source, mode, data row, data field, etc. requested by the user, and then bind the result of the data extraction with the user's identity information and the unified address identifier of the task configuration file, and then generate the ETL task corresponding to the ETL task request. Exemplarily, data extraction can be performed on the data description information input by the user to determine the data source that the user needs to operate, and the permission action information input by the user can determine which mode in the data source the user needs to perform a read operation or a write operation.

[0061] The ETL task refers to the entire processing flow of obtaining the required data from the data source, converting the data according to the set conversion rules, and then loading it to the corresponding location. For example, assuming that the user needs to pull the real-time transaction data of the current period from the online transaction database of the e-commerce company, and then store it in the path A1 of the cloud disk A, the processing flow of the ETL task is to pull the real-time transaction data of the current period from the online transaction database and store it in the path A1 of the cloud disk A. The ETL task can also specifically specify the processing of the pulled real-time transaction data according to various processing rules. For example, the transaction data of men's clothing in the current real-time transaction data is screened out, and the transaction data of men's clothing is stored separately in the men's clothing data table under the path A1.

[0062] As an implementation method, during the entire process, the management platform may bind the user's identity information to the unique address identifier of the task profile, and then generate the ETL task request based on the bound information. In the embodiment of the present application, the user's identity information is the user's unique identification information, which may specifically be a user ID, and different users may be distinguished by the user ID. Based on this, in some possible embodiments, the ETL task scheduler generates the ETL task corresponding to the ETL task request based on the user's user identity information and the unique address identifier, which may be implemented specifically through the following steps: The task configuration file is obtained by the ETL task scheduler according to the unified address identifier. Specifically, the ETL task scheduler accesses the unified address identifier and obtains the pre-generated task configuration file from the path represented by the unified address identifier.

[0063] Obtaining the data permission list of the user from the management platform through the ETL task scheduler according to the user identity information of the user; If the information to be verified meets the authority range specified by the actual authority information, the ETL task scheduler generates an ETL task corresponding to the ETL task request, and schedules the ETL task to the corresponding ETL executor, which executes the ETL task.

[0064] Further, in some possible embodiments, if the information to be verified does not satisfy the authority scope specified by the actual authority information, the ETL task scheduler refuses to generate the ETL task corresponding to the ETL task request, and outputs the task error information to the user interaction interface.

[0065] That is to say, when the ETL task scheduler receives the ETL task request, it may not generate the ETL task first, but may first execute steps S12, S13 and S14 to perform the first permission check on the user to determine whether the various items of information to be checked in the task configuration file corresponding to the ETL task request submitted by the user meet the range specified by the actual permission information of the user. If so, an ETL task is further generated for the user, and the ETL task is scheduled to the corresponding ETL executor, which will specifically execute the ETL task. If not, there is no need to generate an ETL task for the user, which can effectively save the processing resources of the data permission management system.

[0066] In the embodiment of the present application, when executing step S14, it is determined whether the information to be verified meets the scope of authority specified by the actual authority information, specifically whether the information to be verified is the same as the actual authority information, or whether the information to be verified belongs to a subset of the actual authority information. Exemplarily, the information to be verified and the actual authority information are both the authorization start time and the authorization end time, and the authorization start time and the authorization end time constitute an authorization effective time period. When the user submits the ETL task request, whether the time period applied for use completely falls within the authorization effective time period, if so, it indicates that the authorization start time and the authorization end time in the ETL task request submitted by the user meet the scope of authority specified by the actual authority information. Similarly, taking the mode as an example, if the mode is a table, it further includes data rows and data columns, the data rows in the ETL task request submitted by the user are 1 to 10 rows, and the data rows that can be operated and accessed in the actual authority information of the user are 1 to 100 rows, and the data rows in the information to be verified belong to the subset under the actual authority information, and fall within the scope specified by the actual authority information. At this time, it can be determined that the information to be verified-the data row meets the scope of authority specified by the actual authority information.

[0067] In the embodiment of the present application, only when each piece of information to be verified meets the scope of authority specified by the actual authority information can it be determined that the user has passed the authority verification and can use the corresponding data in the back-end data source. At this time, an ETL task is generated, and the ETL executor executes the ETL task to respond to the user's ETL task request. As long as there is one item in the information to be verified that does not meet the scope of authority specified by the actual authority information, it can be determined that the user has not passed the authority verification and cannot continue to use the corresponding data in the back-end data source. At this time, the ETL task can be refused to be generated for the user, which effectively solves the processing resources required for the subsequent processing, and the data stored in the back-end data source can be protected in multiple dimensions. In addition, if there is one item in the information to be verified that does not meet the scope of authority specified by the actual authority, a task error message can be output to the user interaction interface in the management platform, and the task error message carries the type of information to be verified that does not meet the scope of authority specified by the actual authority information, so as to remind the user to update his submitted ETL task request.

[0068] In the embodiment of the present application, Figure 2 As shown, the data rights management system also includes a business platform, which is similar to the previous management platform and execution platform and belongs to the logical subsystem of the data rights management system. The core function of the business platform is to serve as the overall management subsystem of each database under the unified data management platform, and is used to respond to upper-level data requests and perform addition, modification, deletion, and rights verification on the data stored in the underlying database. Figure 2 As shown, a data source is pre-installed in the business platform. The data source is operated and maintained by an organization or enterprise and is used to store various data required for the normal daily operation of the organization or enterprise. It can be specifically divided into data source 1, data source 2, data source 3, ... data source m. Based on this, in the process of executing step S14 to perform the ETL task in the method provided in the embodiment of the present application, the following steps may also be included: Obtaining environment description information of the deployment environment of the ETL executor, and if the environment description information is a trusted environment, sending an ETL instruction to the data source through a first data port, wherein the first data port is a data port preset between the ETL executor and the data source; The data source operates the target data corresponding to the actual authority information through the ETL instruction.

[0069] The deployment environment of the ETL executor specifically refers to the software environment and hardware environment in which the ETL executor is installed after development. For example, it is installed on a host whose operating system is Windows. The ETL executor is developed based on the Java development language, and the corresponding installation path is AAAA. The network is equipped with a firewall and other software and hardware environments together constitute the environmental description information of the deployment environment of the ETL executor. By performing permission verification on the deployment environment of the ETL executor, it is determined whether the deployment environment is a trusted environment. Specifically, the trusted environment list can be queried to determine whether the deployment environment is a deployment environment in the trusted environment list. If so, it is determined that the deployment environment is a trusted environment. The trusted environment list summarizes and organizes the software environment and hardware environment that have been granted access permission by the management personnel in advance.

[0070] In an embodiment of the present application, if the deployment environment of the ETL executor is a trusted environment, it indicates that the deployment environment can directly allow access to the data in the underlying business platform. At this time, when executing step S14, the ETL instruction can be directly sent to the data source of the business platform through the first data port between the ETL executor and the data source in the business platform to read, write or access the corresponding data in the data source. Among them, the ETL instruction carries the index information and operation type of the target data filled in when the user submits the ETL task request. The storage location of the target data that the user needs to operate can be quickly located through the index information, and then the target data under the storage location is processed accordingly according to the operation type. Exemplarily, if the operation type is read, the data stored in the storage location can be directly read from the storage location indicated by the index information in the ETL instruction in the data source based on the ETL instruction.

[0071] On this basis, for the scenario where the deployment environment of the ETL executor is a trusted environment, the specific execution process of the method provided in the embodiment of the present application can be as follows: Figure 4 As shown, the following steps are included: 1: The user initiates an ETL task to the API service opened by the management platform.

[0072] 1.1: Initiate an ETL scheduling task (pass in the user ID and configuration file URL). Specifically, after receiving the ETL task, the management platform initiates the ETL scheduling task to the ETL task scheduler in the execution platform and passes in the user ID and configuration file URL to the execution platform.

[0073] 1.1.1: Obtaining task configuration information. Specifically, after receiving the ETL scheduling task, the ETL task scheduler obtains the task configuration information pre-configured by the task configuration center from the task configuration center of the management platform.

[0074] 1.1.2: Return task configuration information. Specifically, after receiving the request for obtaining task configuration information initiated by the ETL task scheduler, the task configuration center of the management platform obtains the task configuration file corresponding to the URL information according to the URL information carried in the request for the task configuration information, and returns the task configuration file to the execution platform.

[0075] 2: Obtain user data permissions. Specifically, the ETL task scheduler requests the user's data permissions from the unified permission configuration service of the management platform. The request carries the user's user ID, and then obtains the data permission list based on the user ID to obtain the user's actual permission information.

[0076] 2.1: Return user data permissions. Specifically, after the unified permission configuration service of the management platform queries and obtains the data permission list corresponding to the user ID, the data permission list is returned to the ETL task scheduler.

[0077] 3: Verify data permissions. Specifically, the ETL task scheduler compares the information to be verified in the received task configuration file with the actual permission information in the received user's data permission list to determine whether the verification is passed. If the verification is passed, continue to step 4. If the verification fails, refuse to execute step 4.

[0078] 4. Verify that the ETL task execution command is initiated. Specifically, the ETL task scheduler initiates the ETL task execution command to the ETL executor, and the ETL executor executes steps 4.1, 4.2, and 4.3.

[0079] 4.1: Read and write data from the data source. Specifically, the data source performs a read operation or a write operation on the target data requested in the ETL task execution command according to the ETL task execution command. 4.2: Return the task execution status. Specifically, the data source returns the data reading and writing results to the ETL executor.

[0080] 4.3: Return task execution status. Specifically, the ETL executor returns the data reading and writing results to the ETL task scheduler.

[0081] 5: Return the task execution status. Specifically, the ETL task scheduler returns the task execution status to the management platform.

[0082] 5.1: Return task execution status. Specifically, the management platform returns the task execution status to the user, and the user obtains the required target data.

[0083] In addition, in some possible embodiments, if the deployment environment of the ETL executor is not a trusted environment, that is, the deployment environment of the ETL executor has not been authorized in advance, then the method provided in the embodiment of the present application further includes the following steps: Sending the ETL instruction to the data source adapter through a second data port, wherein the second data port is a data port preset between the ETL executor and the data source adapter; The data source adapter obtains the information to be verified in the task configuration file and the data permission list of the user based on the ETL instruction; Determining, by the data source adapter, whether the information to be verified meets the authority scope specified by the actual authority information; If so, the data source adapter forwards the ETL instruction to the data source, and the data source operates the target data corresponding to the actual permission information through the ETL instruction; If not, the ETL instruction is rejected and forwarded to the data source, and the task error message is output.

[0084] As an implementation method, the environment description information is not a trusted environment, which may be that the environment description information is not in the trusted environment list. In this case, it indicates that either the ETL executor is a fake ETL executor generated by a network attack, or that the ETL executor is an ETL executor used for the first time. At this time, it is necessary to perform a secondary verification on the ETL executor whose deployment environment is not a trusted environment to determine whether it is a fake ETL executor. In an embodiment of the present application, a functional module for performing a secondary verification on the ETL executor is added between the data source and the ETL executor: a data source adapter. The ETL executor interacts with the data source adapter through a second data port pre-set by the developer for interaction between the two. Among them, it can be as follows Figure 2 As shown, the business platform includes several data source adapters: data source adapter 1, data source adapter 2, data source adapter 3, ... data source adapter k. Each data source adapter is a software module with ETL executor verification, the core of which is to receive the ETL executor through the second data port, especially the ETL instruction sent by the ETL executor, and then verify the legitimacy of the ETL executor.

[0085] Among them, the data source adapter obtains the ETL instruction sent by the ETL executor. The ETL instruction is similar to the ETL task request submitted by the user and also carries the user's identity information and the unique address identifier of the task configuration file. Then the data source adapter executes the above steps S12~S14 again to perform a secondary permission check on the user. Only when the user's information to be checked meets the permission range specified by the actual permission information, the ETL instruction sent by the ETL executor is allowed to reach the data source under the business platform. Then the ETL instruction is forwarded to the data source, and the data source responds to the ETL instruction to operate on the target data corresponding to the actual permission information.

[0086] If the information to be verified does not meet the authority range specified by the actual authority information, it indicates that there is a network attack simulation to generate an ETL executor to send an ETL instruction to the data source, but the specific authority range of the user is unclear. At this time, the data source adapter provided by the embodiment of the present application can perform a secondary authority check on the ETL instruction. As a possible implementation method, the data source adapter can also perform a secondary check on the ETL by verifying the user token carried in the ETL instruction, as well as verifying the port number, data source type, number of read and write channels, read and write rate, etc. of the ETL instruction interaction, to further ensure the security of the data in the underlying data source.

[0087] On this basis, for the scenario where the deployment environment of the ETL executor does not belong to the trusted environment, the specific execution process of the method provided in the embodiment of the present application can be as follows: Figure 5 As shown, the following steps are included: 1. Users open API services to the management platform to initiate ETL tasks.

[0088] 2. Initiate an ETL scheduling task (pass in the user ID and configuration file URL). Specifically, after receiving the ETL task, the management platform initiates the ETL scheduling task to the ETL task scheduler in the execution platform and passes in the user ID and configuration file URL to the execution platform.

[0089] 3. Obtaining task configuration information: Specifically, after receiving the ETL scheduling task, the ETL task scheduler obtains the task configuration information pre-configured by the task configuration center from the task configuration center of the management platform.

[0090] 4. Return task configuration information. Specifically, after receiving the request for obtaining task configuration information initiated by the ETL task scheduler, the task configuration center of the management platform obtains the task configuration file corresponding to the URL information according to the URL information carried in the request for the task configuration information, and returns the task configuration file to the execution platform.

[0091] 5. Obtain user data permissions. Specifically, the ETL task scheduler requests the user's data permissions from the unified permission configuration service of the management platform. The request carries the user's user ID, and then obtains the data permission list based on the user ID to obtain the user's actual permission information.

[0092] 6. Return user data permissions. Specifically, after the unified permission configuration service of the management platform obtains the data permission list corresponding to the user ID, it returns the data permission list to the ETL task scheduler.

[0093] 7. Verify data permissions. Specifically, the ETL task scheduler compares the information to be verified in the task configuration file received in step 4 with the actual permission information in the user's data permission list received in step 6 to determine whether the verification is passed. If the verification is passed, continue to step 8. If the verification fails, refuse to execute step 8. 8. Verify that the ETL task execution command is initiated. Specifically, the ETL task scheduler initiates the ETL task execution command to the ETL executor, and the ETL executor executes step 9.

[0094] 9. Read and write data from the data source. Specifically, a data read request or a data write request is initiated to the data source adapter in the business platform to request to read or write the corresponding target data in the data source.

[0095] 10. Request to verify data permissions (secondary verification). Specifically, the data source adapter initiates a secondary verification request to the ETL task scheduler for the task execution command initiated by the ETL executor, requiring the ETL task scheduler to perform another permission information verification on the ETL task. If the secondary verification passes, execute step 11. If the secondary verification fails, refuse to execute step 11.

[0096] 11. The data permission secondary verification is passed. Specifically, the ETL task scheduler sends the verification result to the data source adapter of the business platform.

[0097] 12. Read and write data from the data source. Specifically, the data source adapter forwards the data read request or write request sent by the ETL executor to the corresponding data source, and the data source returns the data read and write result according to the request.

[0098] 13. Return the data reading and writing results. Specifically, the data source returns the data reading and writing results to the data source adapter.

[0099] 14. Return the task execution status. Specifically, the data source adapter returns the task execution status to the ETL executor.

[0100] 15. Return the task execution status. Specifically, the ETL executor returns the task execution status to the ETL task scheduler.

[0101] 16. Return the task execution status. Specifically, the ETL executor returns the task execution status to the management platform.

[0102] 17. Return the task execution status. Specifically, the management platform returns the task execution status to the user, and the user obtains the required target data.

[0103] By selecting the embodiment of the present application, it is possible to perform a secondary permission check on the ETL executor based on the user's permission check in the task configuration file and whether the deployment environment of the ETL executor is a trusted environment. This can prevent network attacks from simulating the ETL executor to operate on the data source under the business platform, and help improve the data security of the data under the entire business platform.

[0104] In a second aspect, the present application provides a data rights management device for a data collection process, wherein, Figure 6 As shown, the device 60 includes: The request acquisition module 601 is used to acquire the ETL task request passed in by the user, wherein the ETL task request carries: the user identity information of the user and the unified address identifier of the task configuration file; The task configuration file acquisition module 602 is used to acquire the task configuration file based on the unified address identifier; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; The data permission list acquisition module 603 is used to acquire the data permission list of the user according to the user identity information of the user, wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is the permission information pre-assigned to the user according to the permission level of the user for each item of information in the data description information, the permission action information and the permission time dimension information; The authority verification module 604 is used to determine to execute the ETL task corresponding to the ETL task request if the information to be verified meets the authority range specified by the actual authority information, otherwise refuse to execute the ETL task and output task error information.

[0105] In some possible embodiments, the execution platform further includes: an ETL executor, wherein the ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier, including: Obtaining the task configuration file according to the unified address identifier through the ETL task scheduler; Obtaining the data permission list of the user from the management platform through the ETL task scheduler according to the user identity information of the user; If the information to be verified meets the authority range specified by the actual authority information, the ETL task scheduler generates an ETL task corresponding to the ETL task request, and schedules the ETL task to the corresponding ETL executor, which executes the ETL task.

[0106] In some possible embodiments, the permission verification module is specifically used to: If the information to be verified does not satisfy the authority range specified by the actual authority information, the ETL task scheduler refuses to generate the ETL task corresponding to the ETL task request, and outputs the task error information to the user interaction interface.

[0107] In some possible embodiments, the data rights management system further includes: a business platform, in which: a data source is pre-installed, and the business platform is used to: Obtaining environment description information of the deployment environment of the ETL executor, and if the environment description information is a trusted environment, sending an ETL instruction to the data source through a first data port, wherein the first data port is a data port preset between the ETL executor and the data source; The data source operates the target data corresponding to the actual authority information through the ETL instruction.

[0108] In some possible embodiments, the service platform is also pre-installed with: a data source adapter, and the service platform is also used for: If the environment description information is not the trusted environment, sending the ETL instruction to the data source adapter through a second data port, wherein the second data port is a data port preset between the ETL executor and the data source adapter; The data source adapter obtains the information to be verified in the task configuration file and the data permission list of the user based on the ETL instruction; Determining, by the data source adapter, whether the information to be verified meets the authority scope specified by the actual authority information; If so, the data source adapter forwards the ETL instruction to the data source, and the data source operates the target data corresponding to the actual permission information through the ETL instruction; If not, the ETL instruction is rejected and forwarded to the data source, and the task error message is output.

[0109] Among them, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved in this application are in compliance with the relevant laws and regulations and do not violate public order and good morals.

[0110] The names of the messages or information exchanged between multiple devices in the embodiments of the present application are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0111] In a third aspect, the exemplary embodiments of the present application further provide an electronic device, comprising: at least one processor; and a memory connected to the at least one processor in communication. The memory stores a computer program that can be executed by the at least one processor, and the computer program is used to enable the electronic device to perform a method according to an embodiment of the present application when executed by the at least one processor.

[0112] The exemplary embodiment of the present application further provides a non-transitory computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor of a computer, is used to cause the computer to perform a method according to an embodiment of the present application.

[0113] The exemplary embodiments of the present application further provide a computer program product, including a computer program, wherein when the computer program is executed by a processor of a computer, it is used to enable the computer to execute the method according to the embodiment of the present application.

[0114] refer to Figure 7 , the structural block diagram of the electronic device 700 that can be used as the server or client of the present application will now be described, which is an example of the hardware device that can be applied to various aspects of the present application. The electronic device is intended to represent various forms of digital electronic computer equipment, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices and other similar computing devices. The components shown herein, their connections and relationships, and their functions are only examples, and are not intended to limit the implementation of the present application described and / or required herein.

[0115] like Figure 7As shown, the electronic device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a ROM 702 or a computer program loaded from a storage unit 708 into a RAM 703. In the RAM 703, various programs and data required for the operation of the electronic device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An I / O interface 705 is also connected to the bus 704. Among them, RAM is a random access memory and ROM is a read-only memory.

[0116] A plurality of components in the electronic device 700 are connected to the I / O interface 705, including: an input unit 706, an output unit 707, a storage unit 708, and a communication unit 709. The input unit 706 may be any type of device capable of inputting information to the electronic device 700, and the input unit 706 may receive input digital or character information, and generate key signal inputs related to user settings and / or function control of the electronic device. The output unit 707 may be any type of device capable of presenting information, and may include but is not limited to a display, a speaker, a video / audio output terminal, a vibrator, and / or a printer. The storage unit 708 may include but is not limited to a disk, an optical disk. The communication unit 709 allows the electronic device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks, and may include but is not limited to a modem, a network card, an infrared communication device, a wireless communication transceiver, and / or a chipset, such as a Bluetooth™ device, a WiFi device, a WiMax device, a cellular communication device, and / or the like.

[0117] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above. For example, in some embodiments, the aforementioned data rights management method for the data acquisition process may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 700 via the ROM 702 and / or the communication unit 709. In some embodiments, the computing unit 701 may be configured to perform the aforementioned data rights management method for the data acquisition process in any other appropriate manner (e.g., by means of firmware).

[0118] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0119] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0120] As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0121] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0122] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0123] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

Claims

1. A data rights management method for a data collection process, characterized in that: The method comprises: Acquire an ETL task request passed in by a user, wherein the ETL task request carries: user identity information of the user and a unified address identifier of a task configuration file; Based on the unified address identifier, the task configuration file is obtained; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; According to the user identity information of the user, a data permission list of the user is obtained; wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is the permission information pre-assigned to the user according to the permission level of the user for each item of information in the data description information, the permission action information and the permission time dimension information; If the information to be verified meets the authority range specified by the actual authority information, it is determined to execute the ETL task corresponding to the ETL task request; otherwise, the ETL task is refused to be executed and task error information is output.

2. The method according to claim 1, characterized in that The method is applied to a data rights management system, the data rights management system includes: a management platform, an execution platform, the execution platform includes: an ETL task scheduler, the management platform includes: a user interaction interface, and the method further includes: Obtaining data description information, permission action information, and permission time dimension information input by the user on the user interaction interface; Generate the task configuration file based on the input data description information, permission action information, and permission time dimension information, and generate the unified address identifier for the task configuration file; The ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier.

3. The method according to claim 2, characterized in that The execution platform further includes: an ETL executor, wherein the ETL task scheduler generates an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier, including: Obtaining the task configuration file according to the unified address identifier through the ETL task scheduler; Obtaining the data permission list of the user from the management platform through the ETL task scheduler according to the user identity information of the user; If the information to be verified meets the authority range specified by the actual authority information, the ETL task scheduler generates an ETL task corresponding to the ETL task request, and schedules the ETL task to the corresponding ETL executor, which executes the ETL task.

4. The method according to claim 3, characterized in that The method further comprises: If the information to be verified does not satisfy the authority range specified by the actual authority information, the ETL task scheduler refuses to generate the ETL task corresponding to the ETL task request, and outputs the task error information to the user interaction interface.

5. The method according to claim 3, characterized in that: The data rights management system further includes: a business platform, in which: a data source is pre-installed, and the method further includes: Obtaining environment description information of the deployment environment of the ETL executor, and if the environment description information is a trusted environment, sending an ETL instruction to the data source through a first data port, wherein the first data port is a data port preset between the ETL executor and the data source; The data source operates the target data corresponding to the actual authority information through the ETL instruction.

6. The method according to claim 5, characterized in that The business platform also pre-installs: a data source adapter, and the method further includes: If the environment description information is not the trusted environment, sending the ETL instruction to the data source adapter through a second data port, wherein the second data port is a data port preset between the ETL executor and the data source adapter; The data source adapter obtains the information to be verified in the task configuration file and the data permission list of the user based on the ETL instruction; Determining, by the data source adapter, whether the information to be verified meets the authority scope specified by the actual authority information; If so, the data source adapter forwards the ETL instruction to the data source, and the data source operates the target data corresponding to the actual permission information through the ETL instruction; If not, the ETL instruction is rejected and forwarded to the data source, and the task error message is output.

7. A data rights management device for a data collection process, characterized in that: The device comprises: A request acquisition module, used to acquire an ETL task request passed in by a user, wherein the ETL task request carries: user identity information of the user and a unified address identifier of a task configuration file; A task configuration file acquisition module is used to acquire the task configuration file based on the unified address identifier; wherein the task configuration file includes information to be verified, and the information to be verified includes: data description information of the ETL task request, permission action information of the ETL task request, and permission time dimension information of the ETL task request; the data description information includes: data source, directory, mode, data table, data field, and data row requested; the permission action information includes: read operation or write operation; the permission time dimension information includes: authorization start time, authorization end time, and time period interval; A data permission list acquisition module is used to acquire the data permission list of the user according to the user identity information of the user, wherein the data permission list of the user includes the actual permission information of the user, wherein the actual permission information is the permission information pre-assigned to the user according to the permission level of the user for each item of information in the data description information, the permission action information and the permission time dimension information; The authority verification module is used to determine to execute the ETL task corresponding to the ETL task request if the information to be verified meets the authority range specified by the actual authority information, otherwise refuse to execute the ETL task and output task error information.

8. The device according to claim 7, characterized in that The device is applied to a data rights management system, the data rights management system comprises: a management platform, an execution platform, the execution platform comprises: an ETL task scheduler, the management platform comprises: a user interaction interface; The request acquisition module is used to obtain data description information, permission action information, and permission time dimension information input by the user in the user interaction interface; The task configuration file acquisition module is used to generate the task configuration file based on the input data description information, permission action information, and permission time dimension information, and generate the unified address identifier for the task configuration file; The authority verification module is further configured to generate, through the ETL task scheduler, an ETL task corresponding to the ETL task request according to the user identity information of the user and the unified address identifier.

9. An electronic device, characterized in that: The electronic device comprises: Processor; and Memory for storing programs, The program includes instructions, which, when executed by the processor, cause the processor to perform the method according to any one of claims 1 to 6.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-6.

Images