Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method, device and equipment and storage medium, which can be applied to the field of computer technology and the field of financial technology. The method is applied to a database server and includes the following steps: receiving data generation information sent by a local server, wherein the data generation information is obtained according to a target table in a database; generating target data related to the target table in the database server according to the data generation information; and importing the target data into the database, wherein the database is deployed in the database server.

Description

Technical Field

[0001] This disclosure relates to the fields of computer technology and financial technology, and in particular to a data processing method, apparatus, electronic device, storage medium and program product. Background Technology

[0002] When financial institutions modify or implement new business systems, extensive stress testing is required to ensure that the new business systems can be used normally, and a large amount of test data is needed during the testing process.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related art: the methods for generating test data in the related art are inefficient. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a data processing method, apparatus, electronic device, storage medium, and program product.

[0005] The first aspect of this disclosure provides a data processing method applied to a database server, comprising:

[0006] Receive data generation information sent by the local server, wherein the data generation information is obtained from the target table in the database;

[0007] Based on the data generation information, generate target data related to the target table in the database server; and

[0008] Import the target data into a database, which is deployed on a database server.

[0009] According to embodiments of this disclosure, target data related to a target table is generated in a database server based on data generation information, including:

[0010] The data generation information is parsed to determine the target data type of the target table;

[0011] Based on the target data type, determine the target data generation rules that match the target table; and

[0012] Based on the target data generation rules, generate the target data volume in the database server.

[0013] According to embodiments of this disclosure, the target data type includes at least one of string type, numeric type, and date type;

[0014] Based on the target data type, determine the target data generation rules that match the target table, including:

[0015] Based on the string type, the target data generation rule that matches the target table is the string generation rule, which is used to determine the string length of the data.

[0016] Based on the numeric type, the target data generation rule that matches the target table is the numeric generation rule, where the numeric generation rule is used to determine the numeric type of the data; and / or

[0017] Based on the date type, the target data generation rule that matches the target table is the date generation rule, which is used to determine the date of the data.

[0018] According to embodiments of this disclosure, target data of a target data volume is generated in a database server according to target data generation rules, including:

[0019] Based on the string generation rules, the first sub-data volume of string data is generated in the database server;

[0020] According to the number generation rules, generate digital data for the second sub-data volume in the database server; and / or

[0021] Based on the date generation rules, the date data of the third sub-data volume is generated in the database server.

[0022] According to embodiments of this disclosure, importing target data into a database includes:

[0023] Based on the size of the target data, the target data is divided into preset batches of data sets;

[0024] Based on the dataset and the preset data sharding quantity, determine multiple target data shards from the dataset; and

[0025] Import multiple sharded target data from each dataset into the database.

[0026] A second aspect of this disclosure provides a data processing method applied to a local server, comprising:

[0027] In response to the first input received by the data generation interface, the target table is determined from the table information associated with the database, wherein the database is deployed on a database server;

[0028] Based on the target data type of the target table, determine the target data generation rules that match the target table;

[0029] Based on the target data generation rules and the target data volume, determine the data generation information; and

[0030] Using the transmission path established between the local server and the database server, data generation information is sent to the database server so that the database server can receive the data generation information sent by the local server, generate target data related to the target table in the database server according to the data generation information, and import the target data into the database.

[0031] According to embodiments of this disclosure, the above-described data processing method applied to a local server further includes:

[0032] In response to the second input received from the data generation interface, determine the transmission information related to the database server;

[0033] Based on the transmitted information, a transmission path is established between the database server and the database server; and

[0034] If the transmission path is successfully established, retrieve the table information related to the database from the database server.

[0035] According to embodiments of this disclosure, data generation information is determined based on target data generation rules and target data volume, including:

[0036] Based on the target data generation rules and the target data volume, determine the original data generation instructions;

[0037] Convert the raw data generation instructions into target data generation instructions in the target language; and

[0038] The target data generation instructions are processed to determine the data generation information.

[0039] A third aspect of this disclosure provides a data processing apparatus applied to a database server, comprising:

[0040] The receiving module is used to receive data generation information sent by the local server, wherein the data generation information is obtained from the target table in the database;

[0041] The generation module is used to generate information based on the data, and to generate target data related to the target table in the database server; and

[0042] The import module is used to import target data into a database, which is deployed on a database server.

[0043] A data processing apparatus is provided according to a fourth aspect of this disclosure, applied to a local server, comprising:

[0044] The first determining module is used to determine the target table from table information related to the database in response to a first input received by the data generation interface, wherein the database is deployed on a database server;

[0045] The second determining module is used to determine the target data generation rules that match the target table based on the target data type of the target table.

[0046] The third determining module is used to determine data generation information based on the target data generation rules and the target data volume; and

[0047] The sending module is used to send data generation information to the database server using the transmission path established between the local server and the database server. The database server receives the data generation information sent by the local server, generates target data related to the target table in the database server according to the data generation information, and imports the target data into the database.

[0048] A fifth aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the methods described above.

[0049] A sixth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the methods described above.

[0050] The seventh aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0051] According to the data processing method, apparatus, device, medium, and program product provided in this disclosure, the method involves receiving data generation information sent by a local server, wherein the data generation information is obtained based on a target table in a database; generating target data related to the target table in a database server based on the data generation information; and importing the target data into the database, wherein the database is deployed on a database server. Since the target data is generated in the database server and the database is deployed in the database server, the import of the target data into the database server is a local transmission, which does not require network transmission, thereby reducing data transmission consumption and improving the efficiency of data generation. This at least partially solves the technical problem of low efficiency in methods for generating test data in related technologies. Attached Figure Description

[0052] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0053] Figure 1 This diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0054] Figure 2A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0055] Figure 3 A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown schematically;

[0056] Figure 4 A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown schematically;

[0057] Figure 5 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0058] Figure 6 A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0059] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure. Detailed Implementation

[0060] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0061] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0062] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0063] When using expressions such as "at least one of A, B, and C", they should generally be interpreted in accordance with the meaning that is commonly understood by a person skilled in the art (e.g., "a system having at least one of A, B, and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B, and C, etc.).

[0064] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0065] As financial institutions' business grows, the demands on their business systems also increase. This necessitates upgrading existing systems or launching new ones to support broader business needs. Stress testing of these systems requires generating substantial amounts of test data. One approach is for testers to manually copy and repeatedly insert single data entries to simulate production conditions. However, this method is time-consuming due to limitations in testing skills, and the data samples are limited and fail to reflect real-world scenarios. Another approach involves script generation, where test cases are rewritten for each new database table. Testers then import the data using a database client and the scripting language. This method is less efficient and time-consuming.

[0066] In view of this, embodiments of this disclosure provide a data processing method applied to a database server, comprising: receiving data generation information sent by a local server, wherein the data generation information is obtained based on a target table in the database; generating target data related to the target table in the database server based on the data generation information; and importing the target data into the database, wherein the database is deployed on the database server. Since the target data is generated in the database server, and the database is deployed on the database server, importing the target data into the database server is a local transmission, requiring no network transmission, reducing data transmission overhead, improving data generation efficiency, and at least partially solving the technical problem of low efficiency in methods for generating test data in related technologies.

[0067] Figure 1 The diagram illustrates an application scenario of the data processing method according to an embodiment of the present disclosure.

[0068] like Figure 1 As shown, application scenario 100 according to this embodiment may include a database server 101, a local server 102, and a network 103. The network 103 serves as a medium for providing a communication link between the database server 101 and the local server 102. The network 103 may include various connection types, such as wired or wireless communication links or fiber optic cables, etc.

[0069] Database server 101 can be a server with a database deployed on it.

[0070] Local server 102 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0071] It should be noted that the data processing method applied to a database server provided in this disclosure embodiment can generally be executed by the database server 101. Correspondingly, the data processing device applied to a database server provided in this disclosure embodiment can generally be disposed in the database server 101. The data processing method applied to a database server provided in this disclosure embodiment can also be executed by a server or server cluster that is different from the database server 101 and is capable of communicating with the database server 101 and the local server 102.

[0072] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0073] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0074] like Figure 2 As shown, the data processing method of this embodiment is applied to a database server and includes operations S210 to S230.

[0075] In operation S210, data generation information is received from the local server, whereby the data generation information is obtained from the target table in the database.

[0076] According to embodiments of this disclosure, the local server may be a server used to establish a connection with the database server locally.

[0077] According to embodiments of this disclosure, the target table may be a table pre-selected from a database, which may be a relational database. The database may be deployed on a database server.

[0078] According to embodiments of this disclosure, the data generation information may be information related to data generation, such as the quantity of data generated, the type of data generated, etc.

[0079] In operation S220, information is generated based on the data, and target data related to the target table is generated in the database server.

[0080] According to embodiments of this disclosure, data generation information can be pre-configured with information related to data generation, thereby generating target data related to the target table based on the data generation information.

[0081] According to embodiments of this disclosure, the target data may be data used to test a system associated with a database.

[0082] In operation S230, the target data is imported into the database.

[0083] According to embodiments of this disclosure, importing target data into the database can be done by importing the target data into a specified storage location within the database, or by importing the target data into a specified file location within the database. Since the database is deployed on a database server, importing the target data into the database does not require network transmission.

[0084] According to embodiments of this disclosure, by receiving data generation information sent by a local server, wherein the data generation information is obtained based on a target table in a database; generating target data related to the target table in a database server based on the data generation information; and importing the target data into the database, wherein the database is deployed on a database server, since the target data is generated in the database server and the database is deployed in the database server, importing the target data into the database server is a local transmission, which does not require network transmission, reduces data transmission consumption, improves data generation efficiency, and at least partially solves the technical problem of low efficiency in methods for generating test data in related technologies.

[0085] According to embodiments of this disclosure, generating target data related to a target table in a database server based on data generation information includes: parsing the data generation information to determine the target data type of the target table; determining target data generation rules matching the target table based on the target data type; and generating target data of the target data volume in the database server based on the target data generation rules.

[0086] According to embodiments of this disclosure, the data generation information can be parsed based on its data format to determine the target data type of the target table. The data format of the data generation information can be Javascript (JavaScript, JS code). JS code can provide customizable functions and methods, making data generation more flexible.

[0087] According to embodiments of this disclosure, data generation information is parsed, and different data types can correspond to different data generation rules. Different data types can have type rule matching relationships with data generation rules, thereby determining the target data generation rule that matches the target table in the type rule matching relationship based on the target data type of the target table.

[0088] According to embodiments of this disclosure, the target data volume can be preset, for example, it can be 100,000 data entries, 50,000 data entries, etc.

[0089] According to embodiments of this disclosure, target data generation rules are determined based on the target data type, and target data can be generated in the database server according to certain rules, making the data generation process more reasonable.

[0090] According to embodiments of this disclosure, the target data type includes at least one of string type, numeric type, and date type.

[0091] Based on the target data type, determine the target data generation rules that match the target table, including: based on the string type, determining that the target data generation rules that match the target table are string generation rules, wherein the string generation rules are used to determine the string length of the data; based on the numeric type, determining that the target data generation rules that match the target table are numeric generation rules, wherein the numeric generation rules are used to determine the numeric type of the data; and / or based on the date type, determining that the target data generation rules that match the target table are date generation rules, wherein the date generation rules are used to determine the date of the data.

[0092] According to embodiments of this disclosure, string-type data requires a defined string length; for example, the string length could be 5, 7, etc. String generation rules can be set to generate string-type data. These rules can also be used to determine whether to generate a fixed-length string.

[0093] According to embodiments of this disclosure, the data of a numeric type needs to have its numeric type determined. For example, the numeric type can be an integer, a decimal, etc. Numeric generation rules can be set to generate numeric data. The numeric generation rules can also determine the range of the numbers.

[0094] According to embodiments of this disclosure, date-type data requires determining the date of the data. The date of the data can use a custom format of template YYYY-MM-DD hh:mm:ss, for example, 2023-01-01 00:00:00. Therefore, date generation rules can be set to generate date-type data.

[0095] According to embodiments of this disclosure, matching different data generation rules based on string type, numeric type, and date type can further make the data generation process on the database server more orderly and reasonable.

[0096] According to embodiments of this disclosure, generating target data of a target data volume in a database server according to target data generation rules includes: generating string data of a first sub-data volume in the database server according to string generation rules; generating numeric data of a second sub-data volume in the database server according to numeric generation rules; and / or generating date data of a third sub-data volume in the database server according to date generation rules.

[0097] According to embodiments of this disclosure, the target data may be data comprising only one target data type, or it may be data comprising multiple target data types. For example, the target data may comprise only string data, or it may comprise both string and numeric data.

[0098] According to embodiments of this disclosure, corresponding data can be generated based on the amount of data for each target data type. For example, the amount of data for generating string type data can be set as the first sub-data amount, thereby generating string data of the first sub-data amount according to string generation rules; the amount of data for numeric type data can be set as the second sub-data amount, thereby generating string data of the second sub-data amount according to numeric generation rules; and the amount of data for date type data can be set as the third sub-data amount, thereby generating date data of the third sub-data amount according to date generation rules.

[0099] According to embodiments of this disclosure, string data of the first sub-data volume, numeric data of the second sub-data volume, and / or date data of the third sub-data volume can together constitute the target data.

[0100] According to embodiments of this disclosure, setting different data volumes for different target data types can make the data generation more detailed and further improve the orderliness of data generation.

[0101] According to embodiments of this disclosure, importing target data into a database includes: dividing the target data into preset batches of data sets based on the size of the target data; determining multiple sharded target data from the data sets based on the data sets and a preset number of data shards; and importing the multiple sharded target data from each data set into the database.

[0102] According to embodiments of this disclosure, if all the generated target data is imported into the database at once, it may affect the normal operation of the database server. Therefore, the target data can be divided into a preset batch of data sets, for example, 100,000 data entries.

[0103] According to embodiments of this disclosure, each data set can be sharded according to a preset data sharding quantity, for example, the preset data sharding quantity can be one thousand data entries. The data set can be divided into multiple sharding target data according to the preset data sharding quantity, and the sharding target data can be Structured Query Language (SQL) statements.

[0104] According to embodiments of this disclosure, the fragmented target data is imported into the database according to the size of the fragmented target data, making the data import more orderly and improving the data import efficiency.

[0105] Figure 3 A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown.

[0106] like Figure 3 As shown, the data processing method of this embodiment is applied to a local server and includes operations S310 to S340.

[0107] In operation S310, in response to the first input received by the data generation interface, the target table is determined from the table information associated with the database, wherein the database is deployed on a database server.

[0108] According to embodiments of this disclosure, the data generation interface can be an interface for inputting information during the data generation process, and can be deployed on a local server.

[0109] According to embodiments of this disclosure, the table information related to the database may include table name information, identification information, or number information used to identify the table, thereby allowing the target table to be determined from the table information.

[0110] In operation S320, the target data generation rules that match the target table are determined based on the target data type of the target table.

[0111] According to embodiments of this disclosure, the target data type of a target table can be determined using a table type query command, thereby determining a target data generation rule that matches the target table based on the target data type. The target data generation rule can be found in other embodiments of this disclosure, and will not be repeated here.

[0112] When operating S330, the data generation information is determined based on the target data generation rules and the target data volume.

[0113] According to embodiments of this disclosure, the configuration files for the target data generation rules and the target data volume can be compressed to determine the data generation information.

[0114] When operating S340, the data generation information is sent to the database server using the transmission path established between the local server and the database server. The database server receives the data generation information sent by the local server, generates the target data related to the target table in the database server according to the data generation information, and imports the target data into the database.

[0115] According to embodiments of this disclosure, the transmission path between the local server and the database server can be verified in advance to ensure it is normal. If the transmission path between the local server and the database server is determined to be normal, the data generation information is sent to the database server.

[0116] According to embodiments of this disclosure, determining the data generation information in a local server allows for better storage and availability to other testers.

[0117] According to embodiments of this disclosure, the data processing method applied to a local server further includes: in response to a second input received by a data generation interface, determining transmission information related to a database server; establishing a transmission path between the database server based on the transmission information; and, if the transmission path is successfully established, obtaining table information related to the database from the database server.

[0118] According to embodiments of this disclosure, the second input can be an operation of inputting transmission information, which may include username, password, IP address, database name, database tools, etc. in the database server.

[0119] According to embodiments of this disclosure, a connection between a local server and a database server can be established using transmission information. For example, a user can log in to the database server using a username and password.

[0120] According to embodiments of this disclosure, the transmission path can be verified. If the transmission path is successfully established, the table information in the database server can be transmitted back to the local server so that the target table can be selected in the data generation interface of the local server.

[0121] According to embodiments of this disclosure, determining data generation information based on target data generation rules and target data volume includes: determining original data generation instructions based on target data generation rules and target data volume; converting the original data generation instructions into target data generation instructions in the target language; and processing the target data generation instructions to determine data generation information.

[0122] According to embodiments of this disclosure, the target language can be Java. Since JS code can provide customizable functions and methods, making data generation more flexible, the target data generation rules and the original data generation instructions corresponding to the target data volume can be converted into target data generation instructions in the target language using the software program corresponding to the target language.

[0123] According to embodiments of this disclosure, compressing the target data generation instruction can determine the data generation information, so that the data generation information can be better transmitted to the database server.

[0124] Figure 4 A flowchart illustrating a data processing method according to yet another embodiment of the present disclosure is shown.

[0125] like Figure 4 As shown, the method includes operations S410 to S490.

[0126] In operation S410, in response to the second input received by the data generation interface, transmission information related to the database server is determined.

[0127] When operating the S420, a transmission path is established between the S420 and the database server based on the transmitted information.

[0128] When operating S430, if the transmission path is successfully established, retrieve the table information related to the database from the database server.

[0129] In operation S440, the target data generation rules that match the target table are determined based on the target data type of the target table.

[0130] When operating S450, the data generation information is determined based on the target data generation rules and the target data volume.

[0131] When operating the S460, the data generation information is sent to the database server using the transmission path established between the local server and the database server.

[0132] During operation S470, data generation information is received from the local server, which is obtained based on the target table in the database.

[0133] During operation S480, information is generated based on the data, and target data related to the target table is generated in the database server.

[0134] In operation S490, the target data is imported into the database.

[0135] According to the embodiments of this disclosure, operations S410 to S490 can be referred to other embodiments of this disclosure, and will not be repeated here.

[0136] Based on the above-described data processing method applied to a database server, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 5 The device is described in detail.

[0137] Figure 5 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0138] like Figure 5 As shown, the data processing apparatus 500 of this embodiment is applied to a database server and includes a receiving module 510, a generating module 520, and an importing module 530.

[0139] The receiving module 510 is used to receive data generation information sent by the local server, wherein the data generation information is obtained based on a target table in the database. In one embodiment, the receiving module 510 can be used to perform the operation S210 described above, which will not be repeated here.

[0140] The generation module 520 is used to generate information based on the data, and to generate target data related to the target table in the database server. In one embodiment, the generation module 520 can be used to perform the operation S220 described above, which will not be repeated here.

[0141] Import module 530 is used to import target data into a database, wherein the database is deployed on a database server. In one embodiment, import module 530 can be used to perform the operation S230 described above, which will not be repeated here.

[0142] According to embodiments of this disclosure, a generation module 520 for generating target data related to a target table in a database server based on data generation information includes:

[0143] The first generation unit is used to parse the data generation information and determine the target data type of the target table;

[0144] The second generation unit is used to determine the target data generation rules that match the target table based on the target data type; and

[0145] The third generation unit is used to generate target data of the target data volume in the database server according to the target data generation rules.

[0146] According to embodiments of this disclosure, the target data type includes at least one of string type, numeric type, and date type;

[0147] The second generation unit, used to determine the target data generation rules that match the target table based on the target data type, includes:

[0148] The first generation subunit is used to determine the target data generation rule that matches the target table based on the string type. The string generation rule is used to determine the string length of the data.

[0149] The second generation subunit is used to determine, based on the number type, the target data generation rule that matches the target table. This rule is a number generation rule, whereby the number generation rule is used to determine the number type of the data; and / or

[0150] The third generation subunit is used to determine the target data generation rule that matches the target table based on the date type. The date generation rule is used to determine the date of the data.

[0151] According to embodiments of this disclosure, a third generation unit for generating target data of a target data volume in a database server based on target data generation rules includes:

[0152] The fourth generation subunit is used to generate the string data of the first sub-data volume in the database server according to the string generation rules;

[0153] The fifth generation subunit is used to generate digital data for the second sub-data volume in the database server according to the digital generation rules; and / or

[0154] The sixth generation sub-unit is used to generate date data for the third sub-data volume in the database server according to the date generation rules.

[0155] According to embodiments of this disclosure, an import module 530 for importing target data into a database includes:

[0156] The first import unit is used to divide the target data into preset batches of data sets according to the size of the target data.

[0157] The second import unit is used to determine multiple target data fragments from the data set based on the data set and the preset data fragment size; and

[0158] The third import unit is used to import multiple sharded target data from each dataset into the database.

[0159] Based on the above-described data processing method applied to a local server, this disclosure also provides a data processing apparatus. The following will be combined with... Figure 6 The device is described in detail.

[0160] Figure 6 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0161] like Figure 6As shown, the data processing device 600 of this embodiment is applied to a local server and includes a first determining module 610, a second determining module 620, a third determining module 630, and a sending module 640.

[0162] The first determining module 610 is configured to determine a target table from table information related to the database in response to a first input received by the data generation interface, wherein the database is deployed on a database server. In one embodiment, the first determining module 610 may be used to perform the operation S320 described above, which will not be repeated here.

[0163] The second determining module 620 is used to determine the target data generation rules that match the target table based on the target data type of the target table. In one embodiment, the second determining module 620 can be used to perform the operation S320 described above, which will not be repeated here.

[0164] The third determining module 630 is used to determine data generation information based on the target data generation rules and the target data volume. In one embodiment, the third determining module 630 can be used to perform the operation S330 described above, which will not be repeated here.

[0165] The sending module 640 is used to send data generation information to the database server via the transmission path established between the local server and the database server. This allows the database server to receive the data generation information from the local server, generate target data related to the target table in the database server based on the data generation information, and import the target data into the database. In one embodiment, the sending module 640 can be used to perform the operation S340 described above, which will not be repeated here.

[0166] According to embodiments of this disclosure, the data processing apparatus 600 applied to a local server further includes:

[0167] The fourth determining module is used to determine the transmission information related to the database server in response to the second input received by the data generation interface;

[0168] The module establishes a transmission path between the database server and the server based on the transmitted information; and

[0169] The acquisition module is used to retrieve table information related to the database from the database server, provided that the transmission path has been successfully established.

[0170] According to embodiments of this disclosure, a third determining module 630 for determining data generation information based on target data generation rules and target data volume includes:

[0171] The first determining unit is used to determine the original data generation instruction based on the target data generation rules and the target data volume;

[0172] The second determining unit is used to convert the original data generation instructions into target data generation instructions in the target language; and

[0173] The third determining unit is used to process the target data generation instruction and determine the data generation information.

[0174] According to embodiments of this disclosure, any plurality of modules among the receiving module 510, generating module 520, and importing module 530 may be combined into one module, or any one of these modules may be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules may be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the receiving module 510, generating module 520, and importing module 530 may be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in any one of software, hardware, and firmware methods, or in a suitable combination of any of these. Alternatively, at least one of the receiving module 510, generating module 520, and importing module 530 may be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0175] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a data processing method according to an embodiment of the present disclosure.

[0176] like Figure 7 As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0177] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that the programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0178] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0179] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs, which, when executed, implement the data processing method according to the embodiments of this disclosure.

[0180] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0181] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to enable the computer system to implement the data processing methods provided in the embodiments of this disclosure.

[0182] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0183] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0184] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0185] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on the user's computing device, partially on the user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0186] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0187] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0188] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data processing method applied to a database server, comprising: Receive data generation information sent by a local server, wherein the data generation information is obtained based on a target table in a database; Based on the data generation information, target data related to the target table is generated in the database server; and Importing the target data into the database includes: Based on the size of the target data, the target data is divided into data sets in preset batches; Based on the data set and the preset data sharding quantity, determine multiple target data shards from the data set; and According to the size of the target data in each data set, the multiple target data fragments are imported into the database. The database is deployed on the database server.

2. The method according to claim 1, wherein, The step of generating target data related to the target table in the database server based on the data includes: The data generation information is parsed to determine the target data type of the target table; Based on the target data type, determine the target data generation rules that match the target table; and According to the target data generation rules, the target data of the target data volume is generated in the database server.

3. The method according to claim 2, wherein, The target data type includes at least one of string type, numeric type and date type; The step of determining the target data generation rule that matches the target table based on the target data type includes: Based on the string type, the target data generation rule that matches the target table is determined to be a string generation rule, wherein the string generation rule is used to determine the string length of the data; Based on the number type, the target data generation rule that matches the target table is determined to be a number generation rule, wherein the number generation rule is used to determine the number type of the data; and / or Based on the date type, the target data generation rule that matches the target table is determined to be a date generation rule, wherein the date generation rule is used to determine the date of the data.

4. The method according to claim 3, wherein, The step of generating the target data of the target data volume in the database server according to the target data generation rule includes: According to the string generation rules, a first sub-data volume of string data is generated in the database server; According to the stated number generation rules, generate digital data for a second sub-data volume in the database server; and / or According to the date generation rules, date data for a third sub-data volume is generated in the database server.

5. A data processing method applied to a local server, comprising: In response to a first input received by the data generation interface, a target table is determined from table information associated with a database, wherein the database is deployed on a database server; Based on the target data type of the target table, determine the target data generation rules that match the target table; Based on the target data generation rules and the target data volume, determine the data generation information; and Using the transmission path established between the local server and the database server, the data generation information is sent to the database server so that the database server receives the data generation information sent by the local server. Based on the data generation information, target data related to the target table is generated in the database server, and the target data is imported into the database. This includes: dividing the target data into a preset batch of data sets according to the size of the target data; determining multiple sharded target data from the data sets according to the data sets and a preset number of data shards; and importing the multiple sharded target data in each data set into the database according to the size of the sharded target data.

6. The method according to claim 5, further comprising: In response to a second input received by the data generation interface, transmission information related to the database server is determined; Based on the transmitted information, a transmission path is established between the database server and the database server; as well as If the transmission path is successfully established, table information related to the database is obtained from the database server.

7. The method according to claim 5, wherein, The step of determining data generation information based on the target data generation rules and the target data volume includes: Based on the target data generation rules and the target data volume, determine the original data generation instructions; Convert the original data generation instructions into target data generation instructions in the target language; and The target data generation instruction is processed to determine the data generation information.

8. A data processing apparatus, applied to a database server, comprising: A receiving module is used to receive data generation information sent by a local server, wherein the data generation information is obtained based on a target table in a database; A generation module is configured to generate information based on the data, and generate target data related to the target table in the database server; and The import module, used to import the target data into the database, includes: Based on the size of the target data, the target data is divided into data sets in preset batches; Based on the data set and the preset data sharding quantity, determine multiple target data shards from the data set; and According to the size of the target data in each data set, the multiple target data fragments are imported into the database. The database is deployed on the database server.

9. A data processing apparatus, applied to a local server, comprising: The first determining module is configured to determine a target table from table information related to a database in response to a first input received by the data generation interface, wherein the database is deployed on a database server; The second determining module is used to determine the target data generation rules that match the target table based on the target data type of the target table; The third determining module is used to determine data generation information based on the target data generation rules and the target data volume; and The sending module is used to send the data generation information to the database server using the transmission path established between the local server and the database server, so that the database server receives the data generation information sent by the local server, generates target data related to the target table in the database server according to the data generation information, and imports the target data into the database, including: dividing the target data into a preset batch of data sets according to the size of the target data, determining multiple sharded target data from the data sets according to the data sets and a preset number of data shards, and importing the multiple sharded target data in each data set into the database according to the size of the sharded target data.

10. An electronic device, comprising: One or more processors; Memory, used to store one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 4, or the method according to any one of claims 5 to 7.

11. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 4, or the method according to any one of claims 5 to 7.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 4, or the method according to any one of claims 5 to 7.

Images