Data processing method and device

Abstract

The invention discloses a data processing method and device, and relates to the technical field of big data processing. A specific embodiment of the method comprises the steps of receiving an execution instruction, and determining a program script, input definition information, output definition information and data merging information corresponding to the execution instruction; obtaining a target user in an idle state from a user pool, and constructing a temporary namespace in a database by using the target user; generating an input table in the temporary namespace according to the input definition information, and generating an output table in the temporary namespace according to the output definition information; running the program script; wherein the program script is used for processing data in the input table and storing generated output data in the output table; and according to the data merging information, merging the data in the output table into a target table corresponding to the output table. According to the embodiment, a database mirror image does not need to be manufactured, and different users can independently run the program scripts on the database.

Description

Technical Field

[0001] The present invention relates to the technical field of big data processing, and in particular to a data processing method and device. Background Art

[0002] Many company employees need to write their own program scripts and run them independently in the database to meet their respective job requirements. For example, employees in analysis positions write program scripts to obtain target customer groups and formulate corresponding marketing strategies. Employees in operation and maintenance positions write program scripts to obtain the current operation status of the system and formulate corresponding operation and maintenance strategies. This requires each employee to be able to execute program scripts independently without affecting each other. In the prior art, database mirroring technology is usually used to enable each employee to run program scripts independently on the database. However, mirroring technology has high requirements on hardware resources, and it is time-consuming to produce container images. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides a data processing method and device, which does not require the creation of a database mirror, and can reduce the requirements for hardware resources and the operation time.

[0004] In a first aspect, an embodiment of the present invention provides a data processing method, including:

[0005] Receive an execution instruction, and determine a program script, input definition information, output definition information, and data merging information corresponding to the execution instruction;

[0006] Obtain an idle target user from the user pool, and use the target user to build a temporary namespace in the database;

[0007] generating an input table in the temporary namespace according to the input definition information, and generating an output table in the temporary namespace according to the output definition information;

[0008] Running the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table;

[0009] According to the data merging information, the data in the output table is merged into the target table corresponding to the output table.

[0010] Optionally, before acquiring the target user in an idle state from the user pool, the method further includes:

[0011] Determining input data according to the input definition information;

[0012] Determine the user corresponding to the execution instruction;

[0013] In response to the user having the right to use the input data, the step of acquiring a target user in an idle state from a user pool is performed.

[0014] Optionally, before acquiring the target user in an idle state from the user pool, the method further includes:

[0015] Determining the target table according to the output definition information;

[0016] Determine the user corresponding to the execution instruction;

[0017] In response to the user having processing authority for the target table, the step of acquiring a target user in an idle state from a user pool is performed.

[0018] Optionally, generating an input table in the temporary namespace according to the input definition information includes:

[0019] Determine input table structure information and data extraction information according to the input definition information;

[0020] Creating the input table in the temporary namespace according to the input table structure information;

[0021] According to the data extraction information, input data is extracted from the data source, and the input data is stored in the input table.

[0022] Optionally, generating an output table in the temporary namespace according to the output definition information includes:

[0023] Determine output table structure information according to the output definition information;

[0024] The output table is created in the temporary namespace according to the output table structure information.

[0025] Optionally, merging the data in the output table into a target table corresponding to the output table according to the data merging information includes:

[0026] In response to the data merging information representation being updated, determining a key field corresponding to the data merging information;

[0027] Combining multiple values ​​of the key field in the output table into a key field set;

[0028] Delete the target record in the target table; wherein the value of the key field of the target record is in the keyword set;

[0029] Insert the data records in the output table into the target table.

[0030] Optionally, merging the data in the output table into a target table corresponding to the output table according to the data merging information includes:

[0031] In response to the data merge information indicating coverage, deleting all records in the target table;

[0032] Insert the data records in the output table into the target table.

[0033] Optionally, merging the data in the output table into a target table corresponding to the output table according to the data merging information includes:

[0034] In response to the data merging information representing the connection, determining a key field and a connection method corresponding to the data merging information;

[0035] According to the key field and the connection mode, the output table and the target table are connected, and the connection result is saved in the target view.

[0036] Optionally, after acquiring the target user in an idle state from the user pool, the method further includes:

[0037] Setting the target user to a busy state;

[0038] After merging the data in the output table into the target table corresponding to the output table according to the data merging information, the method further includes:

[0039] Cleaning up the temporary namespace;

[0040] The target user is set to an idle state.

[0041] In a second aspect, an embodiment of the present invention provides a data processing device, including:

[0042] An instruction receiving module, used to receive an execution instruction, and determine the program script, input definition information, output definition information, and data merging information corresponding to the execution instruction;

[0043] A space construction module is used to obtain a target user in an idle state from a user pool, and use the target user to construct a temporary namespace in a database;

[0044] A table generation module, configured to generate an input table in the temporary namespace according to the input definition information, and to generate an output table in the temporary namespace according to the output definition information;

[0045] A script running module, used to run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table;

[0046] A data merging module is used to merge the data in the output table into a target table corresponding to the output table according to the data merging information.

[0047] Optionally, it also includes:

[0048] An authority determination module, used to determine input data according to the input definition information;

[0049] Determine the user corresponding to the execution instruction;

[0050] In response to the user having the right to use the input data, the step of acquiring a target user in an idle state from a user pool is performed.

[0051] Optionally, it also includes:

[0052] A permission determination module, used for determining the target table according to the output definition information;

[0053] Determine the user corresponding to the execution instruction;

[0054] In response to the user having processing authority for the target table, the step of acquiring a target user in an idle state from a user pool is performed.

[0055] In a third aspect, an embodiment of the present invention provides an electronic device, including:

[0056] one or more processors;

[0057] A storage device is used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any of the above embodiments.

[0058] In a fourth aspect, an embodiment of the present invention provides a computer-readable medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the method described in any of the above embodiments.

[0059] In a fifth aspect, an embodiment of the present invention provides a computer program product, including a computer program, which implements the method described in any of the above embodiments when executed by a processor.

[0060] An embodiment of the above invention has the following advantages or beneficial effects: after receiving the execution instruction, a temporary namespace is constructed in the database using the target user in an idle state, and an input table and an output table are generated in the temporary namespace. A program script is run, and the program script is used to process the data in the input table, and the generated output data is saved in the output table, and the data in the output table is merged into the target table corresponding to the output table according to the data merge information. Through the target user and the temporary namespace, the program script corresponding to each execution instruction can be isolated, so that each employee can independently run the program script on the database. Since there is no need to create a database mirror, the requirements for hardware resources and the operation time are reduced.

[0061] The further effects of the above-mentioned non-conventional optional manner will be described below in conjunction with the specific implementation manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] The accompanying drawings are used to better understand the present invention and do not constitute an improper limitation of the present invention.

[0063] Figure 1 is a flowchart of a data processing method provided by the first embodiment of the present invention;

[0064] Figure 2 is a flow chart of a data processing method provided by the second embodiment of the present invention;

[0065] Figure 3 is a flow chart of a data processing method provided by the third embodiment of the present invention;

[0066] Figure 4 is a structural schematic diagram of a data processing device provided by an embodiment of the present invention;

[0067] Figure 5 It is a schematic diagram of the structure of a computer system of a terminal device or a server suitable for implementing an embodiment of the present invention. DETAILED DESCRIPTION

[0068] The following is a description of exemplary embodiments of the present invention in conjunction with the accompanying drawings, including various details of the embodiments of the present invention to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present invention. Similarly, for clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0069] The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations.

[0070] It should be noted that in the embodiments of the present application, certain software, components, models and other existing solutions in the industry may be mentioned, and they should be regarded as exemplary. Their purpose is only to illustrate the feasibility of implementing the technical solution of the present application, but it does not mean that the applicant has or will necessarily use the solution.

[0071] Figure 1 is a flow chart of a data processing method provided by the first embodiment of the present invention, such as Figure 1 As shown, the method includes:

[0072] Step 101: Receive an execution instruction, and determine the program script, input definition information, output definition information, and data merging information corresponding to the execution instruction.

[0073] The execution instruction is used to execute the program script. The program script is used to complete a specific function. The input definition information is the relevant information of the input information corresponding to the program script. The input definition information may include: data source information, input table structure information and data extraction information, etc. Among them, the data source information may include: source table name, field sequence number, field name, field description, screening conditions, etc.

[0074] The output definition information is the relevant information of the output information corresponding to the program script. The output definition information may include: output table structure information, target table information, etc. Among them, the output table structure information may include: output table name, field sequence number, field name, field description, etc.

[0075] The data merge information is used to merge the output data of the program script into the target table, which can be determined by the output definition information. The data merge information can represent update, overwrite, connection, etc. Different data merge information corresponds to different data merge methods.

[0076] Step 102: Obtain an idle target user from the user pool, and use the target user to build a temporary namespace in the database.

[0077] The user pool is used to manage at least one database user, and the database user is used to operate the database. The idle target user is a database user that is not in use. The idle target user is a user who has not created a temporary namespace and has not executed any instructions. Using the target user to build a temporary namespace in the database can reduce the conflict between different employees running different execution instructions. The temporary namespace can be a database temporary tablespace.

[0078] If there is at least one idle database user in the user pool, any one of the at least one idle database user is selected as the target user. If there is no idle database user in the user pool, after waiting for a preset period of time, it is determined again whether there is an idle database user in the user pool.

[0079] Step 103: Generate an input table in a temporary namespace according to the input definition information, and generate an output table in the temporary namespace according to the output definition information.

[0080] According to the input table structure information in the input definition information, the input table is constructed in the temporary namespace. The input table structure information may include: input table name, field name of the input table, field type of the input table, field description of the input table, etc. After the input table is constructed, the input data extracted from the data source is inserted into the input table.

[0081] In one embodiment of the present invention, an input table is generated in a temporary namespace according to input definition information, including: determining input table structure information and data extraction information according to the input definition information; creating an input table in the temporary namespace according to the input table structure information; extracting input data from a data source according to the data extraction information, and storing the input data in the input table.

[0082] The data extraction information can be extraction conditions or extraction codes, etc. The extraction conditions are flexibly set according to business needs. The extraction conditions can be: user number = XXX, occurrence time = October 9, 2024, status flag = timeout, etc.

[0083] The extraction code is used to extract output data from the data source. The system can determine at least one extraction template for the user to choose based on the source type of the data source. The user edits the selected extraction steps and generates the extraction code.

[0084] After extracting the input data from the data source, the mapping relationship between the data fields of the input data and the fields of the input table is determined. According to the mapping relationship, the input data is saved in the input table.

[0085] According to the output definition information in the input definition information, an output table is generated in a temporary namespace. The output table structure information may include: output table name, field name of the output table, field type of the output table, and field description of the output table. The table name of the generated output table is the output table name in the output table structure information. There are multiple output table fields in the output table. The output table fields correspond to the field name and field type of the output table in the output table structure information.

[0086] Step 104: Run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table.

[0087] The program script can be written in different programming languages. The program script reads the data in the input table, processes the data information, obtains the output data, and then saves the output data in the output table.

[0088] Before running the program script, all namespaces used in the program script are replaced with temporary namespaces to ensure that the operation of the program script is limited to the temporary namespace without affecting the data in other namespaces.

[0089] The input table and output table can be views, virtual tables or data tables, etc. Since the input table and output table are in the temporary namespace, they will not affect the normal execution of the execution programs from other employees in the system.

[0090] Step 105: According to the data merging information, merge the data in the output table into the target table corresponding to the output table.

[0091] The target table is a data table in the system. According to the data merge information, the data in the output table is merged into the target table corresponding to the output table. After the data in the output table is merged into the target table corresponding to the output table, the temporary namespace can be cleaned up, that is, the data and storage space in the temporary namespace can be released and cleaned up to improve the overall resource utilization and execution efficiency of the system. Subsequent analysis and processing can be performed based on the data in the target table.

[0092] It should be noted that the execution instructions sent by different employees may correspond to the same target table. The output data of multiple employees can be merged through the same target table, so that multiple employees can jointly execute the same data processing business.

[0093] In the solution of the embodiment of the present invention, after receiving the execution instruction, a temporary namespace is constructed in the database using the target user in an idle state, and an input table and an output table are generated in the temporary namespace. A program script is run, and the program script is used to process the data in the input table, and the generated output data is saved in the output table, and the data in the output table is merged into the target table corresponding to the output table according to the data merge information. Through the target user and the temporary namespace, the program script corresponding to each execution instruction can be isolated, so that each employee can independently run the program script on the database. Since there is no need to create a database mirror, the requirements for hardware resources and the operation time are reduced.

[0094] In one embodiment of the present invention, before obtaining a target user in an idle state from a user pool, it also includes: determining input data based on input definition information; determining a user corresponding to an execution instruction; and in response to the user having permission to use the input data, executing the step of obtaining a target user in an idle state from the user pool.

[0095] If the user has the permission to use the input data, the step of obtaining the target user in the idle state from the user pool is executed. The permission can be query permission, browsing permission, etc. If the user does not have the permission to use the input data, a reminder message of insufficient permission is sent. By verifying the permission of the user to use the input data, the security of the data in the system is guaranteed to prevent data from being abused.

[0096] In one embodiment of the present invention, before obtaining a target user in an idle state from a user pool, it also includes: determining a target table based on output definition information; determining a user corresponding to an execution instruction; and in response to the user having processing authority for the target table, executing the step of obtaining a target user in an idle state from the user pool.

[0097] Since the data in the output table needs to be merged into the target table corresponding to the output table, it is necessary to verify the user's processing permissions for the target table. Processing permissions can be query permissions, add permissions, modify permissions, delete permissions, etc. If the user has processing permissions for the target table, execute the step of obtaining the target user in the idle state from the user pool. If the user does not have processing permissions for the target table, send a reminder message of insufficient permissions. By verifying the user's processing permissions for the target table, the security of the data in the system is guaranteed to prevent data from being abused.

[0098] Figure 2 is a flow chart of a data processing method provided by the first embodiment of the present invention, such as Figure 2 As shown, the method includes:

[0099] Step 202: Receive an execution instruction, and determine the program script, input definition information, output definition information, and data merging information corresponding to the execution instruction.

[0100] Step 202: Verify the authority of the user corresponding to the execution instruction.

[0101] Step 203: In response to the user passing the authority verification, a target user in an idle state is obtained from the user pool, and a temporary namespace is constructed in the database using the target user.

[0102] Step 204: Generate an input table in a temporary namespace according to the input definition information, and generate an output table in the temporary namespace according to the output definition information.

[0103] Step 205: Run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table.

[0104] Step 206: According to the data merging information, merge the data in the output table into the target table corresponding to the output table.

[0105] In the solution of the embodiment of the present invention, the data merging information may represent update, overwrite, connection, etc. Different data merging information corresponds to different data merging methods. The output table in the temporary namespace is used to temporarily store the output data. After running the program script, the data and storage space in the temporary namespace need to be cleaned up. According to the data merging information, the data in the output table is merged into the target table corresponding to the output table. Subsequent analysis and processing are performed based on the data in the target table.

[0106] In one embodiment of the present invention, the data merge information represents an update. According to the data merge information, the data in the output table is merged into the target table corresponding to the output table, including: in response to the data merge information representing the update, determining the key field corresponding to the data merge information; combining multiple values ​​of the key field in the output table into a key field set; deleting the target record in the target table; wherein the value of the key field of the target record is in the key field set; and inserting the data record in the output table into the target table.

[0107] The data merge information may include: key fields. Key fields can be set by users according to business needs. Key fields can be: user ID, record ID, business ID, etc. Key fields can be the primary key or index field of the target table. First, multiple values ​​of the output table for the key fields are combined into a key field set, and the target records in the target table are deleted. The values ​​of the key fields of the target records are in the keyword set, that is, the data records in the target table and the output table with the same keywords are deleted. Finally, the data records in the output table are inserted into the target table to complete the data merge of the output table and the target table.

[0108] In one embodiment of the present invention, the data merge information represents an overwrite. According to the data merge information, the data in the output table is merged into the target table corresponding to the output table, including: in response to the data merge information representing an overwrite, deleting all records in the target table; inserting the data records in the output table into the target table. If the data merge information represents an overwrite, deleting all records in the target table, inserting the data records in the output table into the target table. That is, the target table is used to store all data records in the output table. These data records are obtained by running the execution instruction.

[0109] In one embodiment of the present invention, the data merge information represents a connection. According to the data merge information, the data in the output table is merged into the target table corresponding to the output table, including: in response to the data merge information representing the connection, determining the key field and the connection method corresponding to the data merge information; according to the key field and the connection method, performing connection processing on the output table and the target table, and saving the connection result in the target view.

[0110] The data merge information may include: key fields, connection methods, and view information. Key fields can be set by users according to business requirements. Key fields can be: user ID, record ID, business ID, etc. Key fields can be the primary key or index field of the target table. The connection methods can be left join, right join, and inner join. View information may include: view name, view ID, etc. Based on the key fields as the connection fields, the output table and the target table are connected, and the connection results are saved in the target view corresponding to the view information.

[0111] Figure 3 is a flow chart of a data processing method provided by the first embodiment of the present invention, such as Figure 3 As shown, the method includes:

[0112] Step 301: Receive an execution instruction, and determine the program script, input definition information, output definition information, and data merging information corresponding to the execution instruction.

[0113] Step 302: Obtain a target user in an idle state from the user pool, set the target user to a busy state, and use the target user to build a temporary namespace in the database.

[0114] Step 303: Generate an input table in a temporary namespace according to the input definition information, and generate an output table in the temporary namespace according to the output definition information.

[0115] Step 304: Run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table.

[0116] Step 305: According to the data merging information, merge the data in the output table into the target table corresponding to the output table.

[0117] Step 306: Clear the temporary namespace; set the target user to idle state.

[0118] In the solution of the embodiment of the present invention, after the target user is obtained, the target user is set to a busy state to avoid other execution instructions being associated with the target user. After the data in the output table is merged into the target table corresponding to the output table, the temporary namespace is cleaned up and the target user is set to an idle state, so that the space can be reclaimed in time, resource waste can be avoided, and the efficiency of instruction execution can be improved.

[0119] To facilitate understanding of the solution of the embodiment of the present invention, a data processing solution is described in detail below. The processing flow of the solution is as follows:

[0120] Step S01: Submit the program script. The program script can be a SQL (Structured Query Language) script. The user sends the written SQL script to the execution instruction. The execution instruction includes: SQL script, input definition information, output definition information and data merge information. The input definition information, output definition information and data merge information can be stored in the input and output definition file. Among them, the input and output definition file can be in JSON format.

[0121] Step S02: Initialize the script execution environment. The environment initialization module is mainly used to construct an isolated execution environment so that the scripts submitted by the users are executed in an isolated environment without interfering with each other.

[0122] The environment initialization module will first obtain an unused database user from the database user pool as the target user. A temporary namespace will be constructed and authorized to the target user. The database users in the user pool will be configured with a maximum resource usage limit in the database when the account is created. According to the input definition information in the input and output definition files submitted by the user, an input table pointing to the original data is constructed. The input table can use a view. Before constructing the input table, the permission management module will first verify whether the user has the permission to use the input data defined by him. If the data used by the user exceeds the permission, the execution of this script will be terminated.

[0123] After initializing the temporary running namespace, the namespaces used in the program script will be uniformly replaced with the temporary running namespace to ensure that the operation of the program script is limited to the temporary running namespace without affecting the data in other namespaces.

[0124] Step S03: Data authority control. The data authority control of the data user is mainly implemented by the data authority configuration and the input and output definition files of the script. Among them: the input definition information in the input and output definition files represents the input data required by the data user this time. The system administrator can specify the database table permissions owned by each user group through configuration, and then add the data users to the corresponding user groups to realize the data empowerment for the data users. Finally, the data usage data of the data users is compared with the data permissions they have to realize the control of data permissions.

[0125] Step S04: Run the program script. The execution module will push the SQL script to the database for execution. After the execution is completed, the output table is generated according to the output definition information of the input and output definition file. The output data generated by the SQL script is stored in the output table. According to the data merge information defined by the user, the output table is merged with the target table of the application storage namespace. The merge information may include: merge, overwrite and connect, etc. For the merge of the output table and the target table, it is also necessary to verify whether the user has the processing authority of the database table.

[0126] Finally, the data in the temporary namespace is cleared and the target user is returned to the user pool.

[0127] The solution of the embodiment of the present invention can achieve a similar effect to database containerization, ensuring that different data users do not interfere with each other when using the data. A lightweight solution is provided, which can use fewer machine resources to achieve the effect of containerization. By inputting and outputting definition files and configuring permissions, the user's data access rights can be controlled at a finer granularity. A configurable merging method is provided to achieve the merging, connection or overwriting of autonomous data usage results of different users.

[0128] The solution of the embodiment of the present invention has low requirements on machine hardware resources. It only requires a virtual machine that can deploy simple network services, and there is no need to make a mirror. Data sharing between different users can be accurate to the granularity of the table, and only simple configuration is required. The execution results of program scripts of different users can be merged, connected or overwritten. When different users execute their own program scripts, the execution environment is completely independent and does not interfere with each other. It is also possible to perform table-level permission control on data.

[0129] Figure 4 is a schematic diagram of the structure of a data processing device provided by an embodiment of the present invention, such as Figure 4 As shown, the device comprises:

[0130] The instruction receiving module 401 is used to receive an execution instruction, determine the program script, input definition information, output definition information and data merging information corresponding to the execution instruction;

[0131] The space construction module 402 is used to obtain a target user in an idle state from the user pool, and use the target user to construct a temporary namespace in the database;

[0132] A table generation module 403, used to generate an input table in a temporary namespace according to the input definition information, and to generate an output table in a temporary namespace according to the output definition information;

[0133] The script running module 404 is used to run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table;

[0134] The data merging module 405 is used to merge the data in the output table into the target table corresponding to the output table according to the data merging information.

[0135] Optionally, it also includes:

[0136] The permission determination module is used to determine the input data according to the input definition information;

[0137] Determine the user corresponding to the execution instruction;

[0138] In response to the user having the right to use the input data, a step of acquiring a target user in an idle state from a user pool is performed.

[0139] Optionally, it also includes:

[0140] The permission determination module is used to determine the target table according to the output definition information;

[0141] Determine the user corresponding to the execution instruction;

[0142] In response to the user having the processing authority for the target table, a step of acquiring an idle target user from a user pool is performed.

[0143] Optionally, the table generating module 403 is specifically used for:

[0144] Determine the input table structure information and data extraction information based on the input definition information;

[0145] Create an input table in a temporary namespace based on the input table structure information;

[0146] According to the data extraction information, input data is extracted from the data source and saved in the input table.

[0147] Optionally, the table generating module 403 is specifically used for:

[0148] Determine the output table structure information according to the output definition information;

[0149] Create an output table in a temporary namespace based on the output table structure information.

[0150] Optionally, the data merging module 405 is specifically used for:

[0151] In response to the data merge information representation update, determining a key field corresponding to the data merge information;

[0152] Combine multiple values ​​of key fields in the output table into a key field set;

[0153] Delete the target record in the target table; wherein the value of the key field of the target record is in the keyword set;

[0154] Insert the data records in the output table into the target table.

[0155] Optionally, the data merging module 405 is specifically used for:

[0156] In response to the data merge information representing the overwrite, deleting all records in the target table;

[0157] Insert the data records in the output table into the target table.

[0158] Optionally, the data merging module 405 is specifically used for:

[0159] In response to the data merging information representing the connection, determining key fields and connection methods corresponding to the data merging information;

[0160] According to the key fields and connection methods, the output table and the target table are connected, and the connection results are saved in the target view.

[0161] Optionally, it also includes:

[0162] A user control module, used to set the target user to a busy state;

[0163] Clean up the temporary namespace;

[0164] Set the target user to idle state.

[0165] An embodiment of the present invention provides an electronic device, including:

[0166] one or more processors;

[0167] The storage device is used to store one or more programs. When the one or more programs are executed by one or more processors, the one or more processors implement the method of any of the above embodiments.

[0168] An embodiment of the present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the data processing method in the embodiment of the present invention is implemented.

[0169] Reference below Figure 5 , which shows a schematic diagram of the structure of a computer system 500 of a terminal device suitable for implementing an embodiment of the present invention. Figure 5 The terminal device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0170] like Figure 5 As shown, the computer system 500 includes a central processing unit (CPU) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. In the RAM 503, various programs and data required for the operation of the system 500 are also stored. The CPU 501, the ROM 502, and the RAM 503 are connected to each other through a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0171] The following components are connected to the I / O interface 505: an input section 506 including a keyboard, a mouse, etc.; an output section 507 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 508 including a hard disk, etc.; and a communication section 509 including a network interface card such as a LAN card, a modem, etc. The communication section 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 510 as needed, so that a computer program read therefrom is installed into the storage section 508 as needed.

[0172] In particular, according to the embodiments disclosed in the present invention, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present invention include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 509, and / or installed from the removable medium 511. When the computer program is executed by the central processing unit (CPU) 501, the above-mentioned functions defined in the system of the present invention are executed.

[0173] It should be noted that the computer-readable medium shown in the present invention may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present invention, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0174] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0175] The modules involved in the embodiments of the present invention may be implemented in software or hardware. The modules described may also be arranged in a processor, for example, they may be described as: an instruction receiving module, a space construction module, a table generation module, a script running module, and a data merging module. Among them, the names of these modules do not constitute a limitation on the modules themselves under certain circumstances. For example, the instruction receiving module may also be described as a "module that receives execution instructions, determines the program script, input definition information, output definition information, and data merging information corresponding to the execution instructions."

[0176] As another aspect, the present invention further provides a computer-readable medium, which may be included in the device described in the above embodiment; or may exist independently without being assembled into the device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by a device, the device includes:

[0177] Receive an execution instruction, and determine a program script, input definition information, output definition information, and data merging information corresponding to the execution instruction;

[0178] Obtain an idle target user from the user pool, and use the target user to build a temporary namespace in the database;

[0179] generating an input table in the temporary namespace according to the input definition information, and generating an output table in the temporary namespace according to the output definition information;

[0180] Running the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table;

[0181] According to the data merging information, the data in the output table is merged into the target table corresponding to the output table.

[0182] According to the technical solution of the embodiment of the present invention, after receiving the execution instruction, a temporary namespace is constructed in the database using the target user in an idle state, and an input table and an output table are generated in the temporary namespace. A program script is run, which is used to process the data in the input table, and the generated output data is saved in the output table, and the data in the output table is merged into the target table corresponding to the output table according to the data merge information. Through the target user and the temporary namespace, the program script corresponding to each execution instruction can be isolated, so that each employee can run the program script independently on the database. Since there is no need to create a database mirror, the requirements for hardware resources and the operation time are reduced.

[0183] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may occur depending on design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A data processing method, characterized in that: include: Receive an execution instruction, and determine a program script, input definition information, output definition information, and data merging information corresponding to the execution instruction; Obtain an idle target user from the user pool, and use the target user to build a temporary namespace in the database; generating an input table in the temporary namespace according to the input definition information, and generating an output table in the temporary namespace according to the output definition information; Running the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table; According to the data merging information, the data in the output table is merged into the target table corresponding to the output table.

2. The method according to claim 1, characterized in that Before obtaining the target user in the idle state from the user pool, the method further includes: Determine input data according to the input definition information; Determine the user corresponding to the execution instruction; In response to the user having the right to use the input data, the step of acquiring a target user in an idle state from a user pool is performed.

3. The method according to claim 1, characterized in that Before obtaining the target user in the idle state from the user pool, the method further includes: Determine the target table according to the output definition information; Determine the user corresponding to the execution instruction; In response to the user having processing authority for the target table, the step of acquiring a target user in an idle state from a user pool is performed.

4. The method according to claim 1, characterized in that: Generating an input table in the temporary namespace according to the input definition information includes: Determine input table structure information and data extraction information according to the input definition information; Creating the input table in the temporary namespace according to the input table structure information; According to the data extraction information, input data is extracted from the data source, and the input data is stored in the input table.

5. The method according to claim 1, characterized in that Generating an output table in the temporary namespace according to the output definition information includes: Determine output table structure information according to the output definition information; The output table is created in the temporary namespace according to the output table structure information.

6. The method according to claim 1, characterized in that The step of merging the data in the output table into a target table corresponding to the output table according to the data merging information includes: In response to the data merging information representation being updated, determining a key field corresponding to the data merging information; Combining multiple values ​​of the key field in the output table into a key field set; Delete the target record in the target table; wherein the value of the key field of the target record is in the keyword set; Insert the data records in the output table into the target table.

7. The method according to claim 1, characterized in that The step of merging the data in the output table into a target table corresponding to the output table according to the data merging information includes: In response to the data merge information indicating coverage, deleting all records in the target table; Insert the data records in the output table into the target table.

8. The method according to claim 1, characterized in that The step of merging the data in the output table into a target table corresponding to the output table according to the data merging information includes: In response to the data merging information representing the connection, determining a key field and a connection method corresponding to the data merging information; According to the key field and the connection mode, the output table and the target table are connected, and the connection result is saved in the target view.

9. The method according to claim 1, characterized in that: After obtaining the target user in the idle state from the user pool, the method further includes: Setting the target user to a busy state; After merging the data in the output table into the target table corresponding to the output table according to the data merging information, the method further includes: Cleaning up the temporary namespace; The target user is set to an idle state.

10. A data processing device, characterized in that: include: An instruction receiving module is used to receive an execution instruction and determine the program script, input definition information, output definition information and data merging information corresponding to the execution instruction; A space construction module is used to obtain a target user in an idle state from a user pool, and use the target user to construct a temporary namespace in a database; A table generation module, configured to generate an input table in the temporary namespace according to the input definition information, and to generate an output table in the temporary namespace according to the output definition information; A script running module, used to run the program script; wherein the program script is used to process the data in the input table and save the generated output data in the output table; A data merging module is used to merge the data in the output table into a target table corresponding to the output table according to the data merging information.

11. The device according to claim 10, characterized in that Also includes: An authority determination module, used to determine input data according to the input definition information; Determine the user corresponding to the execution instruction; In response to the user having the right to use the input data, the step of acquiring a target user in an idle state from a user pool is performed.

12. The device according to claim 10, characterized in that Also includes: An authority determination module, used for determining the target table according to the output definition information; Determine the user corresponding to the execution instruction; In response to the user having processing authority for the target table, the step of acquiring a target user in an idle state from a user pool is performed.

13. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 9.

14. A computer readable medium having a computer program stored thereon, characterized in that When the program is executed by a processor, the method described in any one of claims 1 to 9 is implemented.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images