Data extraction method and device, computer equipment and readable storage medium

Abstract

The invention relates to a data extraction method and device, computer equipment and a readable storage medium. The method is applied to a data supply management and control system, the data supply management and control system comprises a data query node, and the method comprises the following steps: obtaining a data extraction request; the data extraction request comprises a preset data acquisition permission and an audit result identifier; if the auditing result identification is that auditing is passed, the data extraction request is added to a data processing task of a data processing object, and a target script fed back by the data processing object is loaded to a data query node in a data supply management and control system; according to the data query node and the target script, performing data extraction in the data supply management and control system to obtain target data; and feeding back the target data to the cloud space bound with the preset data acquisition permission. By adopting the method, the security of data extraction can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data security, in particular to a data extraction method and device, computer equipment and a readable storage medium. BACKGROUND

[0002] In the insurance field, business personnel need to obtain business data as analysis basis when performing data analysis.

[0003] In the traditional technology, the business personnel edits a form of the business data to be extracted, and sends the form to the developer through offline means such as email, the developer writes a script according to the data to be extracted contained in the form, and performs data extraction, and then feeds back the extracted business data as target data to the business personnel through offline means such as email, so as to realize data extraction.

[0004] However, the current offline operation for business data extraction has the risk of data leakage, resulting in poor security of data extraction. SUMMARY

[0005] Therefore, it is necessary to provide a data extraction method, device, computer equipment, computer readable storage medium and computer program product for the above technical problems.

[0006] In a first aspect, the present application provides a data extraction method, which is applied to a data supply control system, the data supply control system comprising a data query node, comprising:

[0007] obtaining a data extraction request; the data extraction request comprises a preset data acquisition permission and an audit result identifier;

[0008] if the audit result identifier is passed, adding the data extraction request to a data processing task of a data processing object, and loading a target script fed back by the data processing object into the data query node in the data supply control system;

[0009] performing data extraction in the data supply control system according to the data query node and the target script, and obtaining target data;

[0010] feeding back the target data to a cloud space bound with the preset data acquisition permission.

[0011] In one embodiment, if the audit result identifier is passed, the data extraction request is added to the data processing task of the data processing object, comprising:

[0012] If the audit result is identified as passing the audit, the data extraction request is matched with a task, and each data processing object corresponding to the data extraction request is determined;

[0013] For each data processing task, the data processing task is added to the data processing task of the data processing object, and the data processing object is instructed to perform data processing on the data processing task.

[0014] In one embodiment, the data processing object includes a task allocation object and a subtask processing object; the data extraction request is matched with a task, and each data processing object corresponding to the data extraction request is determined, including:

[0015] The data extraction request is added to the data processing task of the task allocation object, and the task allocation object is instructed to perform task splitting and allocation on the data extraction request to determine the data extraction subtask corresponding to the data extraction request.

[0016] The task allocation object submits each data extraction subtask corresponding to the subtask processing.

[0017] In one embodiment, after the target script fed back by the data processing object is loaded into the data query node in the data supply management system, the method further includes:

[0018] According to the preset label type of the target script, the target script is persistently stored in the target position corresponding to the preset label type in the script database.

[0019] In one embodiment, the data extraction is performed in the data supply management system according to the data query node and the target script to obtain target data, including:

[0020] The target script is parsed to determine the data level to be extracted corresponding to the target script.

[0021] According to the preset extraction mechanism allocation strategy and the data level to be extracted, the target extraction mechanism corresponding to the target script is determined, and the data query node is used to perform data extraction according to the target extraction mechanism to obtain initial data.

[0022] The initial data is verified for data compliance, and if the data compliance verification passes, target data is obtained.

[0023] In one embodiment, the target script is parsed to determine the data level to be extracted corresponding to the target script, including:

[0024] The target script is parsed to determine the expected extraction data amount corresponding to the target script;

[0025] The expected extraction data amount is compared with the first and second preset thresholds to determine the to-be-extracted data magnitude corresponding to the to-be-extracted data magnitude.

[0026] In a second aspect, the application further provides a data extraction device, which is applied to a data supply control system, the data supply control system comprising a data query node, comprising:

[0027] The acquisition module is configured to acquire a data extraction request, wherein the data extraction request comprises a preset data acquisition permission and an audit result identifier;

[0028] The loading module is configured to, if the audit result identifier indicates that the audit is passed, add the data extraction request to a data processing task of a data processing object, and load a target script fed back by the data processing object into the data query node in the data supply control system;

[0029] The data extraction module is configured to perform data extraction in the data supply control system according to the data query node and the target script, and obtain target data.

[0030] The feedback module is configured to feed back the target data to a cloud space bound with the preset data acquisition permission.

[0031] In one embodiment, the loading module is specifically configured to, if the audit result identifier indicates that the audit is passed, perform task matching on the data extraction request, and determine each data processing object corresponding to the data extraction request.

[0032] For each data processing task, the data processing task is added to the data processing task of the data processing object, and the data processing object is instructed to perform data processing on the data processing task.

[0033] In one embodiment, the data processing object comprises a task allocation object and a subtask processing object, and the loading module is specifically configured to add the data extraction request to the data processing task of the task allocation object, instruct the task allocation object to perform task splitting and allocation on the data extraction request, and determine a data extraction subtask corresponding to the data extraction request.

[0034] The subtask processing corresponding to each data extraction subtask submitted by the task allocation object is obtained.

[0035] In one embodiment, the device further comprises:

[0036] The storage module is used to persistently store the target script in the script database to the target location corresponding to the preset tag type, according to the preset tag type of the target script.

[0037] In one embodiment, the data extraction module is specifically used to parse the target script and determine the amount of data to be extracted corresponding to the target script;

[0038] Based on the preset extraction mechanism allocation strategy and the amount of data to be extracted, the target extraction mechanism corresponding to the target script is determined, and data is extracted in the data query node according to the target extraction mechanism to obtain the initial data;

[0039] The initial data is subjected to data compliance verification. If the data compliance verification passes, the target data is obtained.

[0040] In one embodiment, the data extraction module is specifically used to parse the target script and determine the expected amount of data to be extracted from the target script;

[0041] The expected amount of data to be extracted is compared with the first preset threshold and the second preset threshold to determine the data level to be extracted corresponding to the data level to be extracted.

[0042] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0043] Obtain a data extraction request; the data extraction request includes preset data acquisition permissions and an audit result identifier;

[0044] If the audit result is marked as approved, the data extraction request is added to the data processing task of the data processing object, and the target script fed back by the data processing object is loaded into the data query node in the data supply management system;

[0045] Based on the data query node and the target script, data is extracted within the data supply control system to obtain the target data;

[0046] The target data is fed back to the cloud space bound to the preset data acquisition permission.

[0047] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the following steps:

[0048] Obtain a data extraction request; the data extraction request includes preset data acquisition permissions and an audit result identifier;

[0049] If the audit result is marked as approved, the data extraction request is added to the data processing task of the data processing object, and the target script fed back by the data processing object is loaded into the data query node in the data supply management system;

[0050] Based on the data query node and the target script, data is extracted within the data supply control system to obtain the target data;

[0051] The target data is fed back to the cloud space bound to the preset data acquisition permission.

[0052] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, performs the following steps:

[0053] Obtain a data extraction request; the data extraction request includes preset data acquisition permissions and an audit result identifier;

[0054] If the audit result is marked as approved, the data extraction request is added to the data processing task of the data processing object, and the target script fed back by the data processing object is loaded into the data query node in the data supply management system;

[0055] Based on the data query node and the target script, data is extracted within the data supply control system to obtain the target data;

[0056] The target data is fed back to the cloud space bound to the preset data acquisition permission.

[0057] The aforementioned data extraction methods, devices, computer equipment, computer-readable storage media, and computer program products verify the review results of data extraction requests initiated by the user through a data supply control system. Approved data extraction requests are then added to the data processing tasks of the data processing object. This enables the verification and transmission of data extraction requests within the data supply control system, avoiding offline interaction between business personnel and developers. After the data supply control system loads the target script returned by the data processing object, business data is extracted at the data query node within the data supply control system. This ensures that data queries are performed at the data query node of the data supply control system, preventing developers from directly storing the queried business data on their local devices. Finally, the target data extracted from the data supply control system is fed back to the cloud space bound to the preset data access permissions. The target data in this cloud space can be displayed to the user. This realizes the verification of whether the user has the data receiving permission during the process of feeding back the target data to the user, ensuring the security of business data transmission in the data supply control system. As a result, the extracted target data is not locally written to the disk on the user's or developer's local device, which can reduce the exposure of sensitive data on the user's or developer's local device, reduce the risk of data leakage during the data extraction process, and improve the security of data extraction. Attached Figure Description

[0058] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0059] Figure 1 This is a flowchart illustrating a data extraction method in one embodiment;

[0060] Figure 2 Here is a business process diagram of a data extraction method in one embodiment;

[0061] Figure 3 This is a physical architecture diagram of a data supply control system in one embodiment;

[0062] Figure 4 This is a logical architecture diagram of a data supply control system in one embodiment;

[0063] Figure 5 This is a flowchart illustrating the process of determining a data processing object in one embodiment;

[0064] Figure 6This is a flowchart illustrating the process of obtaining the subtask processing object corresponding to the data extraction subtask in one embodiment.

[0065] Figure 7 This is a schematic diagram of the data extraction process based on the target script in one embodiment;

[0066] Figure 8 This is a flowchart illustrating the target extraction mechanism of the data volume hierarchical export engine in one embodiment;

[0067] Figure 9 This is a flowchart illustrating the process of determining the amount of data to be extracted in one embodiment;

[0068] Figure 10 This is a structural block diagram of a data extraction device in one embodiment;

[0069] Figure 11 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0070] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0071] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0072] In one embodiment, such as Figure 1 As shown, a data extraction method is provided. This embodiment illustrates the method's application to a terminal, but it is understood that the method can also be applied to a server, or to a system including both a terminal and a server, and implemented through interaction between the terminal and the server. In this embodiment, the method is applied to a data supply management system, which includes a data query node. The method includes the following steps:

[0073] Step 102: Obtain the data extraction request.

[0074] The data extraction request includes preset data acquisition permissions and an audit result identifier.

[0075] In this embodiment, a data supply management and control system is deployed in the terminal, which receives data extraction requests initiated by users. Specifically, after logging into an account with preset data acquisition permissions on the client, the user edits the business data to be extracted on the client's front-end page, thus forming a data extraction request. This data extraction request carries preset data acquisition permissions; for example, the data extraction request could be business data such as the total policy value or policy data of a specific insurance business in a certain region.

[0076] Then, because the business data is sensitive, further verification and approval are required, such as... Figure 2 As shown, Figure 2 This is a business process diagram for the data extraction method. The user client forwards the data extraction request to the approval system, which can be an application registry or an OA (Office Automation) platform. After the approval personnel complete the approval of the data extraction request in the approval system, the data extraction request contains an approval result identifier. Subsequently, the data supply control system obtains the data extraction request carrying preset data acquisition permissions and the approval result identifier. The data supply control system, through platform-based online processing, can also save records of user-submitted data extraction requests as audit evidence.

[0077] Step 104: If the audit result is marked as approved, add the data extraction request to the data processing task of the data processing object, and load the target script fed back by the data processing object into the data query node of the data supply control system.

[0078] In this embodiment, the terminal verifies the data extraction request using an audit result identifier. If the audit result identifier indicates approval, the current data extraction request is legitimate and meets business requirements. Subsequently, the data supply control system forwards the data extraction request to a data processing object. This data processing object can be a data processing personnel (i.e., a developer), or it can be a dedicated data extraction system. If the data processing object is a data processing personnel, the data supply control system adds the data extraction task corresponding to the data extraction request to the data processing personnel. The data processing personnel then writes a corresponding SQL (Structured Query Language) script based on this task, using it as the target script. This target script is uploaded to the script library of the data supply control system and synchronously loaded into the data query node of the data supply control system for subsequent data extraction. For example... Figure 3 As shown, Figure 3This is a physical architecture diagram of the data supply control system. The data query nodes include query nodes in the headquarters data extraction environment and query nodes in the branch office data extraction environment. These query nodes can be written in Java.

[0079] If the data processing object is a data extraction system, this system automatically generates SQL scripts as target scripts based on the data extraction tasks corresponding to the user's data query requests. This enables the interaction between the user and the data processing object, as well as the data processing flow based on the SQL script, to be executed within the data supply control system. Furthermore, the records of online interactions and data extraction processing within the data supply control system are stored there, facilitating auditing of the data extraction request processing.

[0080] Furthermore, if other users have previously submitted data extraction requests for the same data, the data supply management system can also query the script library based on the target tags of the data to be extracted in the data extraction request. If the target script corresponding to the target tag is found, the target script is directly loaded into the data query node of the data supply management system. The script library is managed based on script tags, which are categorized into task-related tags, policy tags, policy maintenance tags, data screening tags, group insurance business operations tags, and individual insurance business operations tags.

[0081] If the review result is marked as unreviewed, it means that the current data extraction request is an illegal request that violates business requirements. The data extraction request may be malicious. In this case, the data supply control system will intercept the data extraction request and report it.

[0082] In one specific embodiment, the example of a data processing worker being the data processing object is used for illustration. Figure 2 As shown, the data supply management and control system performs task acquisition, task splitting, script saving, and data verification based on the tasks to be processed. It also has functions such as online task flow records for data processing personnel, independent script writing and execution, script knowledge base sharing, and management perspective statistical dashboards.

[0083] Step 106: Based on the data query node and the target script, extract data within the data supply control system to obtain the target data.

[0084] In this embodiment, when the data supply control system detects that the target script corresponding to the data extraction request has been loaded into the data query node, the terminal triggers a data query through the data query node within the data supply control system, extracts data from the business database according to the target script, and obtains the target data. The business database is as follows: Figure 3As shown, it includes databases such as OceanBase, Gauss, and PolarDB, as well as Gauss clusters such as Gauss branch cluster, Gauss BI cluster, and Gauss hybrid cluster.

[0085] Specifically, the data supply control system reads uploaded and verified target scripts from the script library. Optionally, before executing the target script, the data supply control system performs a security check on the target script, including checking whether the target script contains any illegal data editing operations, such as data update, data deletion, or other data write operations. Then, as... Figure 3 As shown, the data supply control system first determines whether the current target script is used to extract data from headquarters or branch offices, thus defining the initial range of data query nodes for executing the target script. Then, the data supply control system further selects the final data extraction node from either a first set of candidate nodes in the headquarters data extraction environment or a second set of candidate nodes in the branch office data extraction environment. For example, selection is made based on the load status of each node in either the first or second candidate node set, ultimately resulting in the data query nodes. Each query node in the first candidate node set is connected to a database containing headquarters data, and each query node in the second candidate node set is connected to a database containing data from each branch office.

[0086] After determining the data query node to execute the target script, the data query node executes the target script in a secure sandbox environment, initiating a read-only query request to the target database. During the data extraction process, all extracted business data flows occur within the memory of the data supply control system, without being written to any developer's or user's terminal device or intermediate server disk. The query results, including the business data, are temporarily stored in a cache and converted into a standard-format dataset to obtain the target data. For example, the target data could be a summary of business data for a specific region and time period, or it could be detailed policy information such as a policy list. If the data extraction request targets detailed policy information, a more stringent approval process is applied to the application.

[0087] In an optional embodiment, such as Figure 2As shown, the data supply control system can determine whether to execute the target script directly within the system or send it to the data operation console for execution, based on the database type. For example, if the database type includes Gauss and OceanBase, the target script will be executed through the data operation console if it is OceanBase, and through the data supply control system if it is Gauss.

[0088] Step 108: Feed back the target data to the cloud space bound to the preset data access permission.

[0089] In this embodiment, the terminal uses a data supply control system to feed back the extracted target data to a cloud space bound to preset data access permissions, achieving secure and controllable data delivery. Specifically, after extracting the data, the data supply control system does not transmit the data via easily leaky methods such as email or local download. Instead, it automatically transfers the target data to a user-owned cloud storage space with controlled permissions, such as a designated directory in an enterprise cloud drive. This cloud space is bound to the user's preset data access permissions, ensuring that only authorized users can access the data under audit and supervision. This avoids the risk of data landing on personal devices, ensures the confidentiality and integrity of data flow, and supports operation logging for easy subsequent traceability and compliance auditing.

[0090] In a specific embodiment, such as Figure 4 As shown, Figure 4This is a logical architecture diagram of the data supply control system, which includes a presentation layer, a service layer, and a data layer. The service layer, involved in the data extraction process of the data supply control system, comprises a data extraction interaction module, a data extraction task module, a script management module, a query engine module, and a task scheduling module. Specifically, the data supply control system receives user-initiated and approved data extraction requests through the data extraction interaction module and performs task scheduling, dividing the data extraction request into multiple sub-tasks and distributing them to the task lists of corresponding data processing personnel. Then, after the data processing personnel complete the SQL script based on the sub-tasks, they submit it to the data supply control system. At this point, the data supply control system receives the uploaded SQL script through the data extraction task module as the target script and saves it to the script library according to the script management module. Finally, the data supply control system determines the appropriate query engine for the target script through the query engine module and determines the data query node to execute the target script according to the task scheduling module. Therefore, the data query node in the data supply control system executes the target script according to the query engine, performing data queries in the database cluster in the data layer to obtain the target data. By connecting application registration centers, OA systems, cloud storage and other platforms, the entire process of demand registration, script execution and result delivery is realized online. Data verification and supply are completed through cloud storage, and the result files are delivered directly to users. There is no local disk storage throughout the process, which meets the security requirement of "data not leaving the cloud".

[0091] In the data supply management system, after a user registers a data extraction request, the system registers the request as a data extraction task. Data processing personnel then use the system's graphical interface to complete the process, which includes task confirmation, task breakdown, task distribution, script development, script execution, data verification, and data supply. Finally, the data results are sent to the requesting user's cloud drive, achieving closed-loop management of data extraction requests. Basic information configuration includes data source, role permissions, and user management.

[0092] In the aforementioned data extraction method, the data supply control system verifies the review result of data extraction requests initiated by the user end and adds approved data extraction requests to the data processing task of the data processing object. This enables the verification and transmission of data extraction requests within the data supply control system, avoiding offline interaction between business personnel and developers. After the data supply control system loads the target script fed back by the data processing object, business data is extracted at the data query node within the data supply control system. This ensures that data queries are performed at the data query node of the data supply control system, preventing developers from directly storing the queried business data on their local devices. Finally, the target data extracted within the data supply control system is fed back to the cloud space bound to preset data access permissions. The target data in this cloud space can be displayed to the user. This verifies the user's data receiving permissions during the process of feeding back the target data to the user, ensuring the security of business data transmission within the data supply control system. This ensures that the extracted target data is not locally stored on the user's or developer's local devices, reducing the exposure of sensitive data on either user's or developer's local devices, lowering the risk of data leakage during data extraction, and improving the security of data extraction.

[0093] In one exemplary embodiment, such as Figure 5 As shown, step 104 includes steps 502 to 504. Wherein:

[0094] Step 502: If the review result is marked as approved, perform task matching on the data extraction request to determine the data processing objects corresponding to the data extraction request.

[0095] In this embodiment, the data supply control system pre-sets a process for processing user-initiated data extraction requests. The data processing object includes a task management object and subtask processing objects. The terminal, through the data supply control system, matches data extraction requests marked as approved to data processing tasks within the task management object according to the pre-set process. For example, a data processing task can be a list of pending tasks. The task management object can be an administrator for developers, used to break down data extraction requests and assign the resulting subtasks. Therefore, after adding the data extraction request to the pending tasks of the task management object, the data supply control system receives the subtasks divided by the task management object and the corresponding subtask processing objects, thus obtaining the task management object and subtask processing objects involved in the data extraction request.

[0096] Step 504: For each data processing task, add the data processing task to the data processing task list of the data processing object, and instruct the data processing object to perform data processing on the data processing task.

[0097] In this embodiment, after the data supply control system has determined the task management object corresponding to the data extraction request and the subsequent required sub-task processing objects, the data supply control system automatically adds each split and generated data processing task to the data processing task of the corresponding data processing object. For example, the data processing task can be the main task of "task splitting and allocation" which is the responsibility of the task management object, and the subsequent sub-tasks such as "script writing", "data verification", and "result generation". The main task of "task splitting and allocation" is added to the task list of the task management object, which represents the data processing tasks of the task management object. Alternatively, the data processing sub-tasks such as "script writing", "data verification", and "result generation" can be assigned to the task list of the developers.

[0098] In this embodiment, a data supply control system enables a fully online workflow for data extraction requests, from the user end to the task management object and sub-task processing object. Once approved, the data extraction request is automatically matched and assigned to the corresponding task management object by the data supply control system. It is then broken down into multiple sub-tasks and assigned to specific sub-task processing objects. All task allocation, execution, and feedback are completed within the control system, eliminating the need for offline operations or the offline transmission of sensitive data. This avoids the data leakage risks associated with traditional offline operations due to data storage or lack of process records, thus improving data security throughout the entire data extraction process.

[0099] In one exemplary embodiment, the data processing object includes a task allocation object and a subtask processing object; such as Figure 6 As shown, step 502 includes steps 602 to 604. Wherein:

[0100] Step 602: Add the data extraction request to the data processing task of the task allocation object, and instruct the task allocation object to split and allocate the data extraction request into tasks.

[0101] In this embodiment, the data supply control system, for each data processing task matched by a data extraction request, includes a request partitioning task and data extraction sub-tasks split by a task management object. Specifically, the data supply control system first adds the request partitioning task to the data processing tasks of the task management object, instructing the task management object to partition the current data extraction request into tasks. Then, the task management object determines the data extraction request and splits it into tasks, obtaining multiple data extraction sub-tasks corresponding to the data extraction request, and a sub-task processing object assigned by the task management object to each sub-task.

[0102] Step 604: Obtain the subtask processing object corresponding to each data extraction subtask submitted by the task allocation object.

[0103] In this embodiment of the application, the data supply management system receives information about the subtask processing objects specified by each subtask as the task allocation object, establishes the association between the data extraction subtask and each subtask processing object in the data supply management system, and finally adds the subtask to the task queue of each subtask processing object.

[0104] In this embodiment, the request is first pushed to the task allocation object, which then structurally breaks it down into multiple executable subtasks and assigns corresponding subtask processing objects. This automatically establishes a binding relationship between the subtasks and the processing objects, and the tasks are then distributed to the data processing tasks of each processing object. This achieves a fully online, standardized, and permission-based data processing workflow, improving task flow efficiency and execution accuracy. It also effectively avoids the risk of data leakage due to disconnected information transmission, unclear responsibilities, or unsupervised operations, thereby enhancing data security and constructing a traceable, auditable, and highly collaborative closed-loop management mechanism.

[0105] In an exemplary embodiment, after step 104, the method further includes step 1041. Wherein:

[0106] Step 1041: Based on the preset tag type of the target script, persist the target script in the script database to the target location corresponding to the preset tag type.

[0107] In this embodiment, the target script can be an SQL query script. After the data supply management system receives the target script uploaded by each subtask processing object, it reviews and verifies the target script. After the target script is generated and reviewed, the data supply management system identifies the category to which the current target script belongs based on the pre-configured preset tag type and persistently stores the script in a structured manner in the target storage location corresponding to the preset tag type in the script database. The preset script types include task-type tags, business analysis-type tags, summary-type tags, and data screening-type tags. The storage process includes not only the script code itself but also records of attribute information such as the creator, creation time, associated request number, data source information, and permission level. Optionally, the data supply management system can also automatically prioritize frequently used scripts within the last 15 days, which can improve the reuse rate of R&D assets.

[0108] In this embodiment, by constructing a script library and managing the scripts by tagging them according to the preset tag types of each target script, the data supply control system can retrieve existing tags in the script library according to the task name, tags, and content of the data extraction request. This enables script reuse, reduces redundant development, and improves the efficiency of data extraction.

[0109] In one exemplary embodiment, such as Figure 7As shown, step 106 includes steps 702 to 706. Wherein:

[0110] Step 702: Parse the target script to determine the data volume level to be extracted corresponding to the target script.

[0111] In this embodiment, the data supply control system parses the target script to determine the database tables, fields, and data types to be queried, and further determines the data volume level to be extracted from the target script based on the database tables, fields, and data types to be queried. For example, the data volume level to be extracted includes three levels: large, medium, and small.

[0112] Step 704: Based on the preset extraction mechanism allocation strategy and the amount of data to be extracted, determine the target extraction mechanism corresponding to the target script, and extract data in the data query node according to the target extraction mechanism to obtain the initial data.

[0113] In this embodiment, the preset extraction mechanism allocation strategy can be configured with different preset data extraction mechanisms for different data volumes to be extracted. The data supply management system matches the preset data extraction mechanisms with the data volumes to be extracted using a data volume grading export engine, determining the target extraction mechanism corresponding to the current data volume, i.e., obtaining the target extraction mechanism corresponding to the target script. Further, the data supply management system executes the target script in the data query node according to the target extraction mechanism to achieve data extraction and thus obtain the initial data.

[0114] Among them, the preset data extraction mechanisms in the preset extraction mechanism allocation strategy include direct connection read and write, streaming export and concurrent export of DN nodes. For example, direct connection read and write can be JDBC (Java Database Connectivity) direct connection read and write, streaming export can be CopyManager (part of the PostgreSQL JDBC extension) streaming export, and concurrent export of DN (Data Node) nodes can be GDS (GaussDB Data Server).

[0115] For example, such as Figure 8 As shown, when the amount of data to be extracted is large (more than 20 million rows), data extraction is performed according to the concurrent export mechanism of GDS's DN nodes; when the amount of data to be extracted is medium (between 100,000 and 20 million rows), data extraction is performed according to the CopyManager streaming export; and when the amount of data to be extracted is small (less than 100,000 rows), data extraction is performed according to JDBC direct connection read and write.

[0116] Step 706: Perform data compliance verification on the initial data. If the data compliance verification passes, the target data is obtained.

[0117] In this embodiment, the data supply control system performs data compliance verification on the initial data. If the verification passes, the initial data is confirmed as the target data; otherwise, the extracted initial data is determined to be non-compliant data, and an error message is fed back. Optionally, the data supply control system can also send an alert notification to the administrator's or auditor's email address via an email system. The compliance verification includes verifying the accuracy of business data and the legality of data extraction. For example, if a field requires "XX-digit business order number," its length and format are verified; if the data contains sensitive personal information, it is verified whether it has been anonymized or authorized.

[0118] In this embodiment, by setting matching data extraction mechanisms for data extraction scenarios with different data volumes to be extracted, the performance bottleneck of large data volume export can be solved, system crashes can be avoided, and the efficiency of data export can be improved.

[0119] In one exemplary embodiment, such as Figure 9 As shown, step 702 includes steps 902 to 904. Wherein:

[0120] Step 902: Parse the target script to determine the expected amount of data to be extracted.

[0121] In this embodiment, the data supply management system parses the target script through a database, performs semantic analysis on the target script, and determines information such as tables, fields, and join conditions contained in the target script. Then, based on the table and field information in the target script, the database determines the execution plan corresponding to data extraction. For example, the database estimates possible execution paths using a query optimizer and predicts the number of rows in the query result set for each execution path, ultimately obtaining the expected amount of data to be extracted.

[0122] Step 904: Compare the expected amount of data to be extracted with the first preset threshold and the second preset threshold to determine the data level to be extracted corresponding to the data level to be extracted.

[0123] In this embodiment, the data supply management system classifies the amount of data to be extracted based on a preset first threshold and a second threshold. For example, the first preset threshold could be 100,000 rows of data, and the second preset threshold could be 20 million rows of data. If the amount of data to be extracted is less than the first preset threshold, it is determined to be "small data volume"; if it is greater than or equal to the first preset threshold and less than the second preset threshold, it is determined to be "medium data volume"; and if it is greater than or equal to the second preset threshold, it is determined to be "large data volume".

[0124] In this embodiment, a hierarchical mechanism is used to determine the scale level of the data to be extracted, providing a basis for decision-making in selecting appropriate export methods, execution strategies and resource allocation schemes, thereby improving the efficiency of data extraction.

[0125] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0126] Based on the same inventive concept, this application also provides a data extraction apparatus for implementing the data extraction method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more data extraction apparatus embodiments provided below can be found in the limitations of the data extraction method described above, and will not be repeated here.

[0127] In one exemplary embodiment, such as Figure 10 As shown, a data extraction device 1000 is provided, which is applied to a data supply management and control system. The data supply management and control system includes a data query node, comprising: an acquisition module 1001, a loading module 1002, a data extraction module 1003, and a feedback module 1004, wherein:

[0128] The acquisition module 1001 is used to acquire data extraction requests; the data extraction request includes preset data acquisition permissions and an audit result identifier.

[0129] The loading module 1002 is used to add the data extraction request to the data processing task of the data processing object if the review result is marked as approved, and to load the target script fed back by the data processing object into the data query node of the data supply control system.

[0130] The data extraction module 1003 is used to extract data within the data supply control system based on the data query node and the target script to obtain the target data;

[0131] Feedback module 1004 is used to feed back the target data to the cloud space bound to the preset data acquisition permission.

[0132] In one embodiment, the loading module 1002 is specifically used to perform task matching on the data extraction request and determine the data processing objects corresponding to the data extraction request if the review result is marked as approved.

[0133] For each data processing task, add the data processing task to the data processing task list of the data processing object, and instruct the data processing object to perform data processing on the data processing task.

[0134] In one embodiment, the data processing object includes a task allocation object and a subtask processing object; the loading module 1002 is specifically used to add the data extraction request to the data processing task of the task allocation object, instruct the task allocation object to split and allocate the data extraction request into tasks, and determine the data extraction subtask corresponding to the data extraction request.

[0135] Retrieve the subtasks corresponding to the data extraction subtasks submitted by the task assignment object.

[0136] In one embodiment, the device 1000 further includes:

[0137] The storage module is used to persistently store the target script in the script database to the target location corresponding to the preset tag type, based on the preset tag type of the target script.

[0138] In one embodiment, the data extraction module 1003 is specifically used to parse the target script and determine the amount of data to be extracted corresponding to the target script;

[0139] Based on the preset extraction mechanism allocation strategy and the amount of data to be extracted, the target extraction mechanism corresponding to the target script is determined, and data is extracted in the data query node according to the target extraction mechanism to obtain the initial data;

[0140] Perform data compliance verification on the initial data. If the data compliance verification passes, the target data is obtained.

[0141] In one embodiment, the data extraction module 1003 is specifically used to parse the target script and determine the expected amount of data to be extracted from the target script.

[0142] The expected amount of data to be extracted is compared with the first preset threshold and the second preset threshold to determine the data level to be extracted. Each module in the above data extraction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the operations corresponding to each module.

[0143] In one exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be as follows: Figure 11 As shown, the computer device includes a processor, memory, input / output interfaces, a communication interface, a display unit, and an input device. The processor, memory, and input / output interfaces are connected via a system bus, and the communication interface, display unit, and input device are also connected to the system bus via the input / output interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The input / output interfaces are used for exchanging information between the processor and external devices. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, mobile cellular networks, Near Field Communication (NFC), or other technologies. When the computer program is executed by the processor, it implements a data extraction method. The display unit is used to form a visually visible image and can be a display screen, a projection device, or a virtual reality imaging device. The display screen can be an LCD screen or an e-ink screen. The input device of the computer device can be a touch layer covering the display screen, or buttons, trackballs, or touchpads set on the casing of the computer device, or external keyboards, touchpads, or mice, etc.

[0144] Those skilled in the art will understand that Figure 11 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0145] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to perform the following steps:

[0146] Obtain a data extraction request; the data extraction request includes preset data acquisition permissions and an audit result identifier;

[0147] If the audit result is marked as approved, the data extraction request is added to the data processing task of the data processing object, and the target script fed back by the data processing object is loaded into the data query node in the data supply management system;

[0148] Based on the data query node and the target script, data is extracted within the data supply control system to obtain the target data;

[0149] The target data is fed back to the cloud space bound to the preset data acquisition permission.

[0150] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0151] If the review result is marked as approved, the data extraction request is matched with tasks to determine the data processing objects corresponding to the data extraction request.

[0152] For each of the data processing tasks, the data processing task is added to the data processing task of the data processing object, and the data processing object is instructed to perform data processing on the data processing task.

[0153] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0154] Add the data extraction request to the data processing task of the task allocation object, instruct the task allocation object to split and allocate the data extraction request into tasks, and determine the data extraction sub-task corresponding to the data extraction request;

[0155] Obtain the subtask processing corresponding to each data extraction subtask submitted by the task allocation object.

[0156] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0157] Based on the preset tag type of the target script, the target script is persistently stored in the script database at the target location corresponding to the preset tag type.

[0158] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0159] The target script is parsed to determine the magnitude of the data to be extracted corresponding to the target script;

[0160] Based on the preset extraction mechanism allocation strategy and the amount of data to be extracted, the target extraction mechanism corresponding to the target script is determined, and data is extracted in the data query node according to the target extraction mechanism to obtain the initial data;

[0161] The initial data is subjected to data compliance verification. If the data compliance verification passes, the target data is obtained.

[0162] In one embodiment, the processor, when executing a computer program, also performs the following steps:

[0163] The target script is parsed to determine the expected amount of data to be extracted.

[0164] The expected amount of data to be extracted is compared with the first preset threshold and the second preset threshold to determine the data level to be extracted corresponding to the data level to be extracted.

[0165] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps in the above method embodiments.

[0166] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0167] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0168] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0169] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0170] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A data extraction method, characterized in that, The method is applied to a data supply control system, which includes a data query node, and the method includes: Obtain a data extraction request; the data extraction request includes preset data acquisition permissions and an audit result identifier; If the audit result is marked as approved, the data extraction request is added to the data processing task of the data processing object, and the target script fed back by the data processing object is loaded into the data query node in the data supply management system; Based on the data query node and the target script, data is extracted within the data supply control system to obtain the target data; The target data is fed back to the cloud space bound to the preset data acquisition permission.

2. The method according to claim 1, characterized in that, If the review result is marked as approved, the data extraction request is added to the data processing task of the data processing object, including: If the review result is marked as approved, the data extraction request is matched with tasks to determine the data processing objects corresponding to the data extraction request. For each of the data processing tasks, the data processing task is added to the data processing task of the data processing object, and the data processing object is instructed to perform data processing on the data processing task.

3. The method according to claim 2, characterized in that, The data processing object includes a task allocation object and a subtask processing object; the step of performing task matching on the data extraction request and determining the data processing objects corresponding to the data extraction request includes: Add the data extraction request to the data processing task of the task allocation object, instruct the task allocation object to split and allocate the data extraction request into tasks, and determine the data extraction sub-task corresponding to the data extraction request; Obtain the subtask processing corresponding to each data extraction subtask submitted by the task allocation object.

4. The method according to claim 1, characterized in that, After loading the target script fed back by the data processing object into the data query node of the data supply management system, the method further includes: Based on the preset tag type of the target script, the target script is persistently stored in the script database at the target location corresponding to the preset tag type.

5. The method according to claim 1, characterized in that, The step of extracting target data within the data supply control system based on the data query node and the target script includes: The target script is parsed to determine the magnitude of the data to be extracted corresponding to the target script; Based on the preset extraction mechanism allocation strategy and the amount of data to be extracted, the target extraction mechanism corresponding to the target script is determined, and data is extracted in the data query node according to the target extraction mechanism to obtain the initial data; The initial data is subjected to data compliance verification. If the data compliance verification passes, the target data is obtained.

6. The method according to claim 5, characterized in that, The step of parsing the target script and determining the data volume level to be extracted corresponding to the target script includes: The target script is parsed to determine the expected amount of data to be extracted. The expected amount of data to be extracted is compared with the first preset threshold and the second preset threshold to determine the data level to be extracted corresponding to the data level to be extracted.

7. A data extraction device, characterized in that, The device is used in a data supply control system, which includes a data query node, and the device includes: The acquisition module is used to acquire data extraction requests; the data extraction request includes preset data acquisition permissions and an audit result identifier; The loading module is used to add the data extraction request to the data processing task of the data processing object if the review result is marked as approved, and to load the target script fed back by the data processing object into the data query node in the data supply management system. The data extraction module is used to extract data within the data supply control system based on the data query node and the target script to obtain target data; The feedback module is used to send the target data back to the cloud space bound to the preset data acquisition permission.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images