Data BI analysis method, system, medium and equipment based on data sandbox

Abstract

The present invention relates to a data BI analysis method, system, medium and device based on a data sandbox, which comprises: constructing a data sandbox and initializing the data sandbox; synchronizing table resources, library resources and organization resources of a data resource center to the data sandbox; the data sandbox sends a data synchronization request to a data push center, the data push center acquires data through the source library table information in the request parameter, and pushes the acquired data to the data sandbox; the data sandbox processes the data securely and then outputs it, and performs data BI analysis. The present invention can realize data processing and query in a secure sandbox environment, and is compatible with heterogeneous data sources.

Description

Technical Field

[0001] The present invention relates to the field of BI data analysis technology, and in particular to a data BI analysis method, system, medium and device based on a data sandbox. Background Art

[0002] As the demand for big data from various departments in the enterprise increases, access between cross-departmental databases involves many permissions and security issues. How to process and query data in a secure sandbox environment becomes increasingly important. Summary of the invention

[0003] In view of the above problems, the purpose of the present invention is to provide a data BI analysis method, system, medium and device based on data sandbox, which can realize data processing and query in a safe sandbox environment and is compatible with heterogeneous data sources.

[0004] To achieve the above-mentioned purpose, the present invention adopts the following technical scheme: a data BI analysis method based on a data sandbox, which includes: constructing a data sandbox and initializing the data sandbox; synchronizing the table resources, library resources and institutional resources of the data resource center to the data sandbox; the data sandbox sends a data synchronization request to the data push center, and the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; the data sandbox outputs the data after security processing, and performs data BI analysis.

[0005] Further, the building of the data sandbox includes:

[0006] Create a project according to user needs, and create a sandbox space in the project;

[0007] Determine table resources in the sandbox space, select a target table according to usage requirements, and perform desensitization processing;

[0008] A heterogeneous database table is established in the sandbox space, and all table data in different databases of the heterogeneous databases are synchronized to the sandbox space.

[0009] Furthermore, the target table is selected according to the usage requirements, and the desensitization process is performed, including: configuring the fields of the table that need to be desensitized in the target table, and then selecting the type of desensitization to complete the desensitization process.

[0010] Furthermore, the sandbox space is provided with a space size and a space expiration time; a time reminder is provided for the space expiration time, and the data in the data sandbox is cleared, deleted or created according to the time reminder.

[0011] Furthermore, synchronizing all the different table data in different databases among the heterogeneous databases to the sandbox space includes: setting a synchronization frequency of data increments in the sandbox space.

[0012] Furthermore, the step of synchronizing all the different table data in different databases in the heterogeneous databases to the sandbox space includes:

[0013] Apply for the sandbox space according to the data. If the applied space amount is less than the pre-set limited space amount, a normal application is made; otherwise, a special application is made;

[0014] Determine whether the general application and the special application meet the review requirements. If so, synchronize the data to the sandbox space for data synchronization, data upload, data download and BI analysis.

[0015] Furthermore, the special application includes: applying for expansion of the sandbox space, or applying according to the space expiration time of the sandbox space;

[0016] The review includes reviewing the sandbox space application, the capacity expansion and the extension of the space expiration time.

[0017] A data BI analysis system based on a data sandbox, comprising: a data sandbox construction module, a data synchronization module, a data push module and a BI analysis module; the data sandbox construction module constructs a data sandbox and initializes the data sandbox; the data synchronization module synchronizes table resources, library resources and institutional resources of a data resource center to the data sandbox; the data push module, the data sandbox sends a data synchronization request to the data push center, the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; the BI analysis module, the data sandbox outputs the data after security processing, and performs data BI analysis processing.

[0018] A computer-readable storage medium storing one or more programs, wherein the one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any of the above methods.

[0019] A computing device comprises: one or more processors, a memory and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any of the above methods.

[0020] The present invention adopts the above technical solution, which has the following advantages:

[0021] The present invention is compatible with heterogeneous data sources, has flexible space allocation and cleaning, is lightweight, safe and controllable, and can achieve incremental data synchronization. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the overall flow of the analysis method in one embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of a data sandbox architecture in one embodiment of the present invention;

[0024] Figure 3 It is a schematic diagram of the structure of a computing device in one embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0026] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] The present invention provides a data BI analysis method, system, medium and device based on a data sandbox. The present invention integrates and synchronizes heterogeneous data into a Haihe distributed database for BI analysis. That is, the present invention aggregates the business data of various departments into a data sandbox to achieve comprehensive BI analysis and management of data.

[0028] In one embodiment of the present invention, Figure 1 As shown, a data BI analysis method based on a data sandbox is provided. This embodiment takes the method applied to a terminal as an example. It can be understood that the method can also be applied to a server, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:

[0029] 1) Build a data sandbox and initialize the data sandbox;

[0030] 2) Synchronize the table resources, library resources and institutional resources of the data resource center to the data sandbox;

[0031] 3) The data sandbox sends a data synchronization request to the data push center. The data push center obtains data through the source database table information in the request parameters and pushes the obtained data to the data sandbox;

[0032] 4) The data sandbox processes the data securely and then outputs it for BI analysis.

[0033] When in use, when the data sandbox is initialized, the organization information, library resource information, and table resource information will be automatically synchronized from the data resource center to the data sandbox.

[0034] In the above step 1), a data sandbox is constructed, such as Figure 2 As shown, the following steps are included:

[0035] 1.1) Create a project according to user needs and create a sandbox space in the project;

[0036] 1.2) Determine table resources in the sandbox space, select the target table according to usage requirements, and perform desensitization processing;

[0037] 1.3) Create heterogeneous database tables in the sandbox space, and synchronize all table data in different databases in the heterogeneous databases to the sandbox space.

[0038] Among them, in step 1.2), the target table is selected according to the usage requirements and desensitization is performed, including: configuring the fields of the table that need to be desensitized in the target table, and then selecting the type of desensitization to complete the desensitization.

[0039] In this embodiment, since the table resource information is multiple-choice, the user selects the target table according to actual business needs, configures the fields of the table that need to be desensitized in the selected target table, and then selects the type of desensitization, such as identity desensitization, bank card number desensitization, etc.

[0040] In the above embodiment, the space size and space expiration time are set in the sandbox space; a time reminder is set for the space expiration time, and the data in the data sandbox is cleared, deleted or created according to the time reminder; it will not cause any impact on the source data, thus ensuring data security and protecting data usage permissions, thereby realizing flexible BI analysis across database tables.

[0041] In step 1.3), the synchronization frequency of data increments is set in the sandbox space, and all the data of different tables in different databases of the heterogeneous databases are synchronized to the sandbox space.

[0042] In step 1.3), all the data of different tables in different databases in the heterogeneous databases are synchronized to the sandbox space, including the following steps:

[0043] 1.3.1) Apply for sandbox space based on the data. If the applied space amount is less than the pre-set limited space amount, a normal application will be made; otherwise, a special application will be made;

[0044] 1.3.2) Determine whether general applications and special applications meet the review requirements. If so, synchronize the data to the sandbox space for data synchronization, data upload, data download and BI analysis.

[0045] Among them, special applications include: applying for sandbox space expansion, or applying based on the expiration time of the sandbox space.

[0046] In step 1.3.2), the review includes the review of sandbox space application, expansion and extension of space expiration time.

[0047] In one embodiment of the present invention, a data BI analysis system based on a data sandbox is provided, which includes: a data sandbox construction module, a data synchronization module, a data push module and a BI analysis module;

[0048] A data sandbox construction module is used to construct a data sandbox and initialize the data sandbox;

[0049] The data synchronization module synchronizes the table resources, library resources and institutional resources of the data resource center to the data sandbox;

[0050] Data push module: the data sandbox sends a data synchronization request to the data push center. The data push center obtains data through the source database table information in the request parameters and pushes the obtained data to the data sandbox.

[0051] In the BI analysis module, the data sandbox processes the data securely and then outputs it for BI analysis.

[0052] The system provided in this embodiment is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for specific processes and detailed contents, which will not be repeated here.

[0053] like Figure 3As shown, it is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. The computing device can be a terminal, which can include: a processor, a communication interface, a memory, a display screen and an input device. Among them, the processor, the communication interface and the memory complete mutual communication through a communication bus. The processor is used to provide computing and control capabilities. The memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. When the computer program is executed by the processor, an analysis method is implemented; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The communication interface is used to communicate with an external terminal in a wired or wireless manner. The wireless manner can be implemented through WIFI, a management network, NFC (near field communication) or other technologies. The display screen can be a liquid crystal display screen or an electronic ink display screen. The input device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the housing of the computing device, or an external keyboard, touchpad or mouse. The processor can call the logical instructions in the memory to execute the following method:

[0054] Build a data sandbox and initialize it; synchronize the table resources, library resources and institutional resources of the data resource center to the data sandbox; the data sandbox sends a data synchronization request to the data push center, and the data push center obtains data through the source library table information in the request parameters, and pushes the obtained data to the data sandbox; the data sandbox processes the data securely and outputs it for data BI analysis.

[0055] In addition, the logic instructions in the above-mentioned memory can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0056] Those skilled in the art will understand that Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the scheme of the present application, and does not constitute a limitation on the computing device to which the scheme of the present application is applied. The specific computing device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0057] In one embodiment of the present invention, a computer program product is provided, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the methods provided by the above-mentioned method embodiments, for example, including: constructing a data sandbox and initializing the data sandbox; synchronizing the table resources, library resources and institutional resources of a data resource center to the data sandbox; the data sandbox sends a data synchronization request to a data push center, and the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; the data sandbox outputs the data after security processing, and performs data BI analysis and processing.

[0058] In one embodiment of the present invention, a non-transitory computer-readable storage medium is provided, which stores server instructions. The computer instructions enable a computer to execute the methods provided in the above embodiments, for example, including: building a data sandbox and initializing the data sandbox; synchronizing the table resources, library resources and institutional resources of the data resource center to the data sandbox; the data sandbox sends a data synchronization request to the data push center, and the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; the data sandbox processes the data securely and outputs it for data BI analysis.

[0059] The above embodiment provides a computer-readable storage medium, whose implementation principle and technical effect are similar to those of the above method embodiment, and will not be repeated here.

[0060] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0061] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0062] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0063] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A data BI analysis method based on data sandbox, characterized in that: include: Constructing a data sandbox and initializing the data sandbox; Synchronize the table resources, library resources and institutional resources of the data resource center to the data sandbox; The data sandbox sends a data synchronization request to the data push center, and the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; The data sandbox processes the data securely and then outputs it for BI analysis and processing; The building of the data sandbox includes: Create a project according to user needs, and create a sandbox space in the project; Determine table resources in the sandbox space, select a target table according to usage requirements, and perform desensitization processing; Establishing heterogeneous database tables in the sandbox space, and synchronizing all the different table data in different databases in the heterogeneous databases to the sandbox space; The step of synchronizing all the different table data in different databases in the heterogeneous databases to the sandbox space includes: Apply for the sandbox space according to the data. If the applied space amount is less than the pre-set limited space amount, a normal application is made; otherwise, a special application is made; Determine whether the general application and the special application meet the review requirements. If they do, synchronize the data to the sandbox space to perform data synchronization, data upload, data download and BI analysis; The special application includes: applying for the expansion of the sandbox space, or applying according to the space expiration time of the sandbox space; The review includes reviewing the sandbox space application, the expansion and extension of the space expiration time; The step of synchronizing all the different table data in different databases among the heterogeneous databases to the sandbox space includes: setting a synchronization frequency of data increments in the sandbox space.

2. The data BI analysis method based on the data sandbox as claimed in claim 1, characterized in that: The target table is selected according to the usage requirements and desensitization is performed, including: Configure the fields of the table that need to be desensitized in the target table, then select the desensitization type to complete the desensitization process.

3. The data BI analysis method based on the data sandbox as claimed in claim 1, characterized in that: The sandbox space is provided with a space size and a space expiration time; a time reminder is provided for the space expiration time, and the data in the data sandbox is cleared, deleted or created according to the time reminder.

4. A data BI analysis system based on data sandbox, characterized in that: include: Data sandbox construction module, data synchronization module, data push module and BI analysis module; The data sandbox construction module constructs a data sandbox and initializes the data sandbox; The data synchronization module synchronizes the table resources, library resources and organization resources of the data resource center to the data sandbox; The data push module, the data sandbox sends a data synchronization request to the data push center, the data push center acquires data through the source library table information in the request parameters, and pushes the acquired data to the data sandbox; The BI analysis module, after the data sandbox processes the data securely, outputs the data and performs BI analysis on the data; The building of the data sandbox includes: Create a project according to user needs, and create a sandbox space in the project; Determine table resources in the sandbox space, select a target table according to usage requirements, and perform desensitization processing; Establishing heterogeneous database tables in the sandbox space, and synchronizing all the different table data in different databases in the heterogeneous databases to the sandbox space; The step of synchronizing all the different table data in different databases in the heterogeneous databases to the sandbox space includes: Apply for the sandbox space according to the data. If the applied space amount is less than the pre-set limited space amount, a normal application is made; otherwise, a special application is made; Determine whether the general application and the special application meet the review requirements. If they do, synchronize the data to the sandbox space to perform data synchronization, data upload, data download and BI analysis; The special application includes: applying for the expansion of the sandbox space, or applying according to the space expiration time of the sandbox space; The review includes reviewing the sandbox space application, the expansion and extension of the space expiration time; The step of synchronizing all the different table data in different databases among the heterogeneous databases to the sandbox space includes: setting a synchronization frequency of data increments in the sandbox space.

5. A computer-readable storage medium storing one or more programs, characterized in that: The one or more programs include instructions, which, when executed by a computing device, cause the computing device to perform any one of the methods of claims 1 to 3.

6. A computing device, characterized in that include: One or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs include instructions for executing any one of the methods described in claims 1 to 3.

Images