Batch data unloading processing method and device, electronic equipment and readable storage medium

Abstract

The application provides a batch data unloading processing method and device, electronic equipment and readable storage medium, and relates to the technical field of computers. The method comprises the following steps: in response to an unloading processing request, determining first basic information of a database to be loaded and second basic information of a database to be unloaded; based on the first basic information and the second basic information, extracting a public unloading configuration item from an unloading task configuration file; based on the public unloading configuration item, generating a structured query statement; using a streaming query to execute the structured query statement to obtain unloading data exported from the database to be unloaded; and generating an unloading file from the unloading data and transmitting the unloading file to the database to be loaded. The application can accelerate the unloading processing speed, improve the configuration efficiency of the unloading task, and realize rapid horizontal expansion of the batch processing task quantity.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a batch data unloading processing method, a batch data unloading processing device, an electronic device, and a computer-readable storage medium. Background Technology

[0002] Offloading large volumes of data typically refers to the process of exporting or transferring large amounts of data stored in databases or other data storage systems. During this process, data is extracted from the original data source and is usually saved as files on the local file system or transferred to other locations for further processing, analysis, or backup.

[0003] Currently, most large-scale data unloading tasks are implemented using the existing Spring Batch framework. The main approach is to integrate the Spring Batch framework, use this component to code the unloading task logic, retrieve database data page by page using its pagination component, map the database data to data objects in memory, and then have its write component write the data to a specified file. However, practical experience has revealed the following problems with this approach:

[0004] (1) For large amounts of data, pagination query processing is required. After memory mapping, the data is written to the file via memory. This results in slow deep pagination queries, which burdens the database and ultimately affects the efficiency of the task.

[0005] (2) The learning curve of the framework is steep, and personalized development code needs to be assembled and written according to business needs. It is difficult to quickly complete the data unloading task development in a short period of time.

[0006] (3) The existing solution is not scalable enough for tasks. Different data unloading task code needs to be written according to different tables and query logic, which is not suitable for rapid horizontal expansion of task volume. Summary of the Invention

[0007] The purpose of this application is to provide a batch data unloading processing method, apparatus, electronic device, and readable storage medium to solve the technical problems of low processing efficiency, long processing time, and high difficulty in developing unloading tasks in the prior art.

[0008] To achieve the above objectives, a first aspect of this application provides a batch data unloading processing method, the unloading processing method comprising the following steps: in response to an unloading processing request, determining first basic information of the database to be loaded and second basic information of the database to be unloaded; based on the first basic information and the second basic information, extracting common unloading configuration items from the unloading task configuration file, wherein the unloading task configuration file stores common unloading configuration items under different business scenarios, the common unloading configuration items including data source information, unloading strategy information and database field information; generating a structured query statement based on the common unloading configuration items; executing the structured query statement using streaming query to obtain unloaded data exported from the database to be unloaded; generating an unloading file from the unloaded data and transmitting it to the database to be loaded.

[0009] Based on the aforementioned technical means, by encapsulating different types of data offloading task logic parameters in a suitable location accessible from multiple places (such as a common utility class or helper class), common data offloading configuration items for different business scenarios are formed. Thus, when data offloading is required for a specific type of database or system, only the common data offloading configuration item matching the specified database or system type needs to be extracted and assembled into a structured query language to achieve the configuration of the data offloading task code. Compared to the existing data offloading processing method that uses the Spring Batch framework to customize the data offloading task logic and then sequentially retrieves database data page by page, the data offloading processing method of this application not only achieves horizontal zero-code expansion of the database configuration of the data offloading task by extracting common method code (i.e., common data offloading configuration items), reducing the learning and calling costs for technical personnel, but also avoids the problem of deep pagination through streaming queries, significantly improving the processing efficiency of the data offloading task. Furthermore, since no modification to the source code or writing of the data offloading task code is required, the data offloading processing method of this application also reduces the workload of development and design and improves the iterative update speed of database data offloading.

[0010] In an exemplary embodiment of this application, the data source information may include: database connection parameters, data source type, and table name; the data unloading strategy information may include: incremental data unloading parameters, full data unloading parameters, general condition configuration parameters, and custom condition configuration parameters; the database field information may include: field name, field format, and field sorting.

[0011] Based on the aforementioned technical methods, defining the specific parameter types of data source information as database connection, data source type, and table name helps to clarify the source, access method, and processing method of the data to be offloaded, ensuring the consistency, integrity, and security of the offloaded data with the source data. Defining the specific parameter types of offload strategy information as incremental offload parameters, full offload parameters, general condition configuration parameters, and custom condition configuration parameters supports various filtering conditions (such as time range, field values, etc.) to select the data to be offloaded, thereby flexibly controlling the scope and granularity of the exported data to meet different data processing and analysis needs. Defining the specific parameter types of database field information as field name, field format, and field sorting not only clarifies the scope and content of the offloaded data but also improves the efficiency of the offload processing.

[0012] In an exemplary embodiment of this application, the step of using streaming query execution of structured query statements to obtain unloading data exported from the database to be unloaded may include: reading the data source information and database field information of the database to be unloaded, verifying the self-consistency of the structured query statement; injecting the verified structured query statement into the streaming query executor, obtaining strings that meet the query conditions in batches; and concatenating all strings that meet the query conditions to form unloading data.

[0013] Based on the aforementioned technical methods, on the one hand, by verifying the self-consistency of structured query statements before execution, the accuracy of the assembly logic of the structured query statements can be confirmed, thereby improving the query efficiency and accuracy of the unloaded data. On the other hand, by batch-acquiring strings that meet the query conditions in a streaming manner, and then concatenating the strings to form unloaded data and writing it to the unloaded data file, not only can the data receiver process and display the data in a timely manner, improving the overall response time and user experience, but it can also release the memory already processed while processing the current query results, thereby reducing memory consumption.

[0014] In one exemplary embodiment of this application, several unloaded number segments obtained after streaming query can be concatenated into a string that meets the query conditions according to a specified concatenation rule.

[0015] Based on the above technical means, by forming strings from the unloaded data segments according to specified connection rules (such as specified field order, interval fields, etc.), the unloaded data acquired in each batch can be marked, which makes it easier for the data receiver to parse the file after receiving the unloaded data file.

[0016] In an exemplary embodiment of this application, the data unloading processing method may further include: determining data volume information based on the data unloading file, the data volume information including: table name, data file byte count, and data file number of records; generating a data volume description file from the data volume information and transmitting it to the database to be loaded.

[0017] Based on the above technical means, by obtaining data volume information and generating a data volume description file, the data recipient can verify the data volume information and prevent data from being tampered with or lost.

[0018] In an exemplary embodiment of this application, the data unloading processing method may further include: determining data structure information based on the data unloading file, the data structure information including: table name, total number of fields, field name and field data type length; generating a data structure description file from the data structure information and transmitting it to the database to be loaded.

[0019] Based on the above technical means, by obtaining data structure information and generating a data structure description file, it is helpful for the data receiver to parse the data unloading file and to properly map the meaning of the fields, which facilitates the subsequent storage and use of the unloaded data.

[0020] A second aspect of this application provides a batch data unloading processing device, comprising: a basic information determination module, a configuration item extraction module, a query statement generation module, a data unloading module, and a file transfer module; wherein, the basic information determination module is used to determine, in response to an unloading processing request, first basic information of the database to be loaded and second basic information of the database to be unloaded; the configuration item extraction module is used to extract common unloading configuration items from the unloading task configuration file based on the first and second basic information, the unloading task configuration file storing common unloading configuration items under different business scenarios, the common unloading configuration items including data source information, unloading strategy information, and database field information; the query statement generation module is used to generate a structured query statement based on the common unloading configuration items; the data unloading module is used to execute the structured query statement using streaming query to obtain unloaded data exported from the database to be unloaded; and the file transfer module is used to generate an unloading file from the unloaded data and transfer it to the database to be loaded.

[0021] A third aspect of this application provides an electronic device comprising: a processor and a memory communicatively connected to the processor; the processor executes computer execution instructions stored in the memory to implement the above-described data offloading method.

[0022] A fourth aspect of this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, configure the processor to perform the aforementioned data offloading method.

[0023] The fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data offloading method.

[0024] The technical solution provided in this application has at least the following technical effects:

[0025] (1) This application realizes the database configuration of the data unloading task by extracting the common method code (that is, the common data unloading configuration item), which can quickly complete the zero-code development of the data unloading task in a short time. It can not only improve the configuration efficiency of the data unloading task, but also support the rapid horizontal expansion of the batch processing task volume.

[0026] (2) This application introduces a streaming query scheme to transmit the unloaded data to the data receiver in batches, avoiding the inefficiency caused by deep pagination queries when dealing with large amounts of data.

[0027] (3) This application does not require modification of the source code or writing of the data unloading task code, thus reducing the workload of development and design and improving the iterative update speed of database data unloading.

[0028] Other features and advantages of the embodiments of this application will be described in detail in the following detailed description section. Attached Figure Description

[0029] The accompanying drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the following detailed description to explain the embodiments of this application, but do not constitute a limitation on the embodiments of this application. In the drawings:

[0030] Figure 1 The schematic diagram illustrates a flowchart of a batch data unloading processing method according to an embodiment of this application;

[0031] Figure 2 This illustration schematically shows an application environment diagram of the batch data unloading processing method according to an embodiment of this application;

[0032] Figure 3 This schematic diagram illustrates a structural block diagram of a batch data unloading processing apparatus according to an embodiment of the present application;

[0033] Figure 4 The diagram schematically illustrates the internal structure of an electronic device according to an embodiment of this application.

[0034] Explanation of reference numerals in the attached figures

[0035] 102-Terminal, 104-Server, 200-Data offloading processing device, 201-Basic information determination module, 202-Configuration item extraction module, 203-Query statement generation module, 204-Data offloading module, 205-File transfer module, A01-Processor, A02-Network interface, A03-Internal memory, A04-Non-volatile storage medium, B01-Operating system, B02-Computer program. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are only for illustration and explanation of the embodiments of this application and are not intended to limit the embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0037] It should be noted that if the embodiments of this application involve directional indicators (such as up, down, left, right, front, back, etc.), the directional indicators are only used to explain the relative positional relationship and movement of the components in a certain specific posture (as shown in the figure). If the specific posture changes, the directional indicators will also change accordingly.

[0038] Furthermore, if the embodiments of this application involve descriptions such as "first" or "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, features defined with "first" or "second" may explicitly or implicitly include at least one of those features. Additionally, the technical solutions of various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. If the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed in this application.

[0039] The current mainstream approach to data offloading is generally as follows: first, the offloading task logic is constructed using relevant components of the Spring Batch framework; then, its pagination component retrieves data from the data provider's database page by page, mapping the data provider's database data to data objects in memory; finally, its write component writes the mapped database data to the offloading file specified by the data receiver. This approach requires customized code development for different offloading tasks and is not suitable for rapid horizontal scaling of large task volumes. Furthermore, when new data types are added to the data provider's database, or when the database version is updated, technical personnel need to modify the offloading task code again. This not only increases the workload of development personnel but also slows down the iterative update speed of database offloading. In addition, when performing large-scale paginated queries, this approach suffers from deep pagination issues, which burden the database and ultimately affect the efficiency of the offloading task.

[0040] To address the technical problems of low processing efficiency, inability to quickly scale horizontally, and high development difficulty in existing data offloading methods, this application proposes a batch data offloading processing method. This method encapsulates different types of offloading configuration parameters in offloading task configuration files, thus forming common offloading configuration items for different business scenarios. When a new offloading task is added, the offloading task logic can be quickly constructed by extracting the common offloading configuration items. Furthermore, this application employs a streaming query method to obtain database data from the data provider during the offloading process, avoiding deep pagination issues and significantly improving the efficiency of the offloading task.

[0041] To achieve the above technical solution, the first aspect of this application provides a batch data unloading processing method. Figure 1 A flowchart illustrating a batch data unloading processing method according to an embodiment of this application is shown, as follows: Figure 1 As shown, this batch data unloading method may include the following steps:

[0042] Step S101: In response to the unloading processing request, determine the first basic information of the database to be loaded and the second basic information of the database to be unloaded.

[0043] Here, data unloading can be understood as the process of extracting data from one data system or platform and saving or transferring it to another data system or platform in the form of files. Thus, the database to be loaded refers to the target database used by the data receiver to receive and store the unloaded data, while the database to be unloaded refers to the original database used by the data provider to store the data source.

[0044] Before the data receiver uses the unloading tool to unload data from the database to be unloaded into the database to be loaded, it needs to formulate corresponding unloading task logic based on the specific unloading task requirements of the database to be loaded and the types of database tables stored in the database to be unloaded. This logic is then executed by the unloading tool to complete the unloading process under the current task. Thus, the first basic information of the database to be loaded refers to the parameters characterizing the specific unloading task requirements, while the second basic information refers to the parameters characterizing the types of database tables stored in the database to be unloaded. For example, the first basic information of the database to be loaded may include the name, content, unloading method, unloading conditions, and unloading date of the unloading task; the second basic information may include the name, structure, attributes, type, and value range of the data within the tables.

[0045] After obtaining the first basic information of the database to be loaded and the second basic information of the database to be unloaded, step S102 can be executed.

[0046] Step S102: Based on the first basic information of the database to be loaded and the second basic information of the database to be unloaded, extract common unloading configuration items from the unloading task configuration file.

[0047] Common data offloading configuration items can include data source information, data offloading strategy information, and database field information. The data offloading task configuration file stores different types of data source information, different types of data offloading strategy information, and different types of database field information. By arranging and combining the above-mentioned various types of data source information, data offloading strategy information, and database field information, common data offloading configuration items for different business scenarios can be formed.

[0048] When it is necessary to form corresponding data unloading task logic based on the current data unloading task, the data unloading strategy information, database field information and data source information that match both the first basic information of the database to be loaded and the second basic information of the database to be unloaded can be filtered in the data unloading task configuration file. Then, the filtered data unloading strategy information, database field information and data source information are combined to form a common data unloading configuration item that meets the requirements of the current data unloading task.

[0049] It should be noted that the purpose of setting up the data unloading task configuration file in this application is to provide a suitable location accessible from multiple places to encapsulate common data unloading configuration items, thereby forming a reusable common method. When users need to use logic similar to the functionality of the common data unloading configuration items to implement data unloading tasks, they can directly extract the corresponding common data unloading configuration without having to rewrite the same code. Furthermore, when this data unloading logic needs to be modified, only the common data unloading configuration item in the data unloading task configuration file needs to be modified, which will affect all places that use the common data unloading configuration item, avoiding the risk of multiple modifications.

[0050] After obtaining a common unloading configuration item that matches both the first and second basic information, step S103 can be executed.

[0051] Step S103: Generate a structured query statement based on the public data configuration item.

[0052] Based on the pre-configured data offloading operation conditions, the data source information, data offloading strategy information, and database field information in the public data offloading configuration items can be written into a query statement that conforms to the SQL (Structured Query Language) syntax, forming a structured query statement that can ultimately execute the data offloading task.

[0053] For example, a single-table query statement can be assembled based on pre-configured regular conditions (i.e., conditions after the WHERE keyword, which usually only support single-table queries) and information in the common data unloading configuration items; if there is a custom SQL configuration (i.e., user-defined data unloading operation conditions), the custom SQL can be directly used as the data unloading logic SQL and assembled with information in the common data unloading configuration items to form the final structured query statement.

[0054] After the structured query statement is assembled, step S104 can be executed.

[0055] Step S104: Execute a structured query statement using streaming query to obtain the unloading data exported from the database to be unloaded.

[0056] The assembled structured query statement is injected into the streaming query executor. Through streaming queries, data that meets the data unloading task requirements can be extracted in real time from one or more data sources until all data that meets the conditions is obtained in streaming.

[0057] After a certain amount of unloading data is obtained from the unloading database, step S105 can be executed.

[0058] Step S105: Generate an unloading file from the unloading data and transfer it to the database to be loaded.

[0059] Thus, this application encapsulates different types of data offloading task logical parameters in the data offloading task configuration file, forming common data offloading configuration items for different business scenarios. In this way, when data offloading is required for a specific type of database or system, only the common data offloading configuration items matching the specified database or system type need to be extracted and assembled into a structured query language to achieve configurability of the data offloading task code. Furthermore, this application avoids deep pagination issues through streaming queries, significantly improving the processing efficiency of data offloading tasks.

[0060] Furthermore, in one possible implementation, the data source information may include database connection parameters, data source type, and table name; the data unloading strategy information may include incremental unloading parameters, full unloading parameters, regular condition configuration parameters, and custom condition configuration parameters; and the database field information may include field name, field format, and field sorting.

[0061] A database connection refers to establishing a communication link between an application and a database, enabling the application to interact and manipulate data with the database. Database connection parameters are a set of parameters used to describe and establish a database connection. For example, database connection parameters may include the database server's hostname or IP address, port number, database name, username, and password for accessing the database.

[0062] Data source type refers to the type of data origin, i.e., the method of data storage and management. Common data source types include relational databases (RDBMS), non-relational databases (NoSQL), file systems, and data warehouses. Since the data processing method in this application is implemented through structured query statements, the data source type in this application mainly refers to the specific type of relational database. For example, the data source type in this application may include MySQL, Oracle Database, Microsoft SQL Server, PostgreSQL, SQLite, IBM Db2, Amazon RDS, Sybase, Informix, SAP HANA, etc. These databases differ in functionality and performance, and are suitable for different application scenarios and needs.

[0063] A table name is the name of the table used to store data in a database. In relational databases, data is organized and stored in the form of tables, each consisting of columns (fields) and rows (records). The table name is the name that uniquely identifies the table. For example, consider a personnel management system database, which contains two tables: an employee information table and a department information table. The "Employee" table stores employee information, with each row representing a record for one employee, including employee_id (employee ID), first_name (first name), last_name (last name), and department_id (department ID). The "Department" table stores department information, including department_id (department ID) and department_name (department name). "Employee" and "Department" can be considered the two table names in the personnel management system database; they are used to uniquely identify each table and for data querying, inserting, and updating operations. When executing SQL statements, these table names can be used to specify the table to be operated on.

[0064] Incremental data export parameters are parameters used to characterize the data export method that only exports newly added or modified data from the database. They are usually used to identify recent changes by recording timestamps or incremental markers, in order to reduce the amount of data exported and processing time.

[0065] The full data export parameter refers to a parameter used to characterize the data export method that exports all data from the entire table or the entire database.

[0066] General condition configuration parameters refer to parameters used to characterize the data export method that exports only data that meets preset general conditions. For example, general condition configuration parameters can include the values ​​of certain fields, logical operators (such as ">", "<", "=", etc.), and comparison operators to set specific filtering conditions.

[0067] Custom condition configuration parameters refer to parameters used to characterize the data export method that only exports data based on user-defined conditions. For example, custom condition configuration parameters can include wildcards (such as "%", "_", etc.), string comparison functions (such as using the UPPER() function to convert strings to uppercase for comparison), aggregate functions (such as determining the sum or average of a column), subqueries (such as complex queries and multi-table operations), EXISTS clauses, expression evaluation (such as mathematical operations, logical operations, string concatenation, etc.), and CASE expressions.

[0068] The field name refers to the name of the specified field that can be included in the data scope in the current business scenario.

[0069] Field format refers to the format of a field in the data export file. There are various ways to specify the format of a field in the data export file, depending on the file type and requirements. Therefore, in this application, the field format can include text format, fixed-width format, CSV format, JSON format, XML format, Parquet format, and Avro format, etc. For example, the field format can be specified as text format, where each field is separated by commas, tabs, or other delimiters, such as the field format "John,Smith,25,HR".

[0070] Field sorting refers to the sorting position of a field in a database table. For example, you can specify the sorting position of a field in a database table as the first column, the third column, the nth column and the mth row, etc.

[0071] Furthermore, in one possible implementation, step S104, in which a structured query statement is executed using a streaming query to obtain the unloading data exported from the unloading database, can be further refined into the following sub-steps S1041 to S1043.

[0072] Sub-step S1041: Read the data source information and database field information of the database to be unloaded, and verify the self-consistency of the structured query statement.

[0073] Specifically, the consistency of the data source information configuration in the common data unloading configuration item can be verified by reading the data source information of the database to be unloaded; the consistency of the configuration of table fields and table information can be verified by reading the database field information of the database to be unloaded; and the preprocessing work can be done by confirming the assembly logic of the structured query statement.

[0074] Sub-step S1042 injects the validated structured query statement into the streaming query executor to obtain strings that meet the query conditions in batches.

[0075] Specifically, the various unloading data segments in the query results can be concatenated into a long string according to the specified concatenation rules, thereby marking the unloading data obtained in each batch, which makes it easier for the data receiver to parse the unloading data file after receiving it.

[0076] For example, each field can be linked together into a long string according to the specified field order, with special characters (such as symbols '|@|', '|#|', '|*|', '|&|', etc.) used as separators between fields, and the same special character is used as the end marker of the record.

[0077] Sub-step S1043: Concatenate all strings that meet the query conditions to form the unloaded data.

[0078] Furthermore, in one possible implementation, the unloading process may further include steps S106 and S107.

[0079] Step S106: Generate a data volume description file and transfer it to the database to be loaded.

[0080] Specifically, based on the public unloading configuration items and the relevant information in the unloading file, the data volume information during the unloading process can be determined, and a data volume description file (i.e., a ctrl format file) can be generated and transmitted to the database to be loaded.

[0081] Data volume information mainly includes table name, data file size in bytes, and number of rows in the data file. This information is used by the data receiver to verify the data volume and prevent data file tampering or data loss.

[0082] Step S107: Generate a data structure description file and transfer it to the database to be loaded.

[0083] Specifically, based on the public data unloading configuration items and relevant information in the data unloading file, the data structure information during the data unloading process can be determined, and a data structure description file (i.e., a DDL format file) can be generated and transmitted to the database to be loaded. The data structure information mainly includes table names, total number of fields, field names, and field data type lengths, etc., which are used by the data receiver to parse the data file and to properly map the meaning of the fields, facilitating subsequent data storage and use.

[0084] When users apply the above data unloading method, they only need to use the provided tool interface to pass in the first and second basic information related to the data unloading task to the data unloading tool. It will automatically generate a common data unloading configuration item that matches the database to be unloaded. Based on this common data unloading configuration item, fields can be added, removed, or adjusted, and custom SQL can be specified. The final SQL statement can then be assembled and executed to complete the data unloading task.

[0085] It is understood that the acquisition, collection, storage, use, processing, transmission, provision, disclosure and application of data in the technical solution of this application all comply with the provisions of relevant laws and regulations.

[0086] Figure 1 This is a flowchart illustrating a batch data unloading process in one embodiment. It should be understood that, although... Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0087] The batch data unloading method provided in this application can be applied to, for example... Figure 2In the application environment shown, terminal 102 communicates with server 104 via a network. Terminal 102 can send a data unloading request to server 104 via the network under user operation. Upon receiving the data unloading request, server 104 responds by obtaining the first basic information of the database to be loaded and the second basic information of the database to be unloaded. It then extracts common data unloading configuration items from the data unloading task configuration file using the first and second basic information to generate a structured query statement. Finally, it executes the structured query statement using streaming query and generates an unloading file. After the unloading file is generated, server 104 can also send the generated unloading file to terminal 102 via network, cloud, or external storage media. Terminal 102 can be any electronic product that can interact with the user through one or more methods such as a keyboard, touchpad, touchscreen, or voice interaction. For example, terminal 102 can include, but is not limited to, various personal computers, laptops, smartphones, tablets, and portable wearable devices. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers; Server 104 can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms.

[0088] A second aspect of this application provides a batch data unloading processing apparatus. In one embodiment, such as... Figure 3 As shown, the data unloading processing device 200 includes a basic information determination module 201, a configuration item extraction module 202, a query statement generation module 203, a data unloading module 204, and a file transfer module 205.

[0089] The basic information determination module 201 is used to determine the first basic information of the database to be loaded and the second basic information of the database to be unloaded in response to the unloading processing request.

[0090] The configuration item extraction module 202 is used to extract common data offloading configuration items from the data offloading task configuration file based on the first basic information and the second basic information. The data offloading task configuration file stores common data offloading configuration items for different business scenarios, which may include data source information, data offloading strategy information, and database field information.

[0091] The query statement generation module 203 is used to generate structured query statements based on common data configuration items.

[0092] The data unloading module 204 is used to execute structured query statements using streaming queries to obtain unloaded data exported from the database to be unloaded.

[0093] The file transfer module 205 is used to generate an unloading file from the unloading data and transfer it to the database to be loaded.

[0094] The data unloading processing device includes a processor and a memory. The aforementioned basic information determination module, configuration item extraction module, query statement generation module, data unloading module, and file transfer module are all stored as program units in the memory. The processor executes the aforementioned program modules stored in the memory to implement the corresponding functions.

[0095] The processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured, and the batch data unloading processing method can be implemented by adjusting kernel parameters.

[0096] The memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM, and the memory includes at least one memory chip.

[0097] The batch data unloading processing device provided in this application embodiment can be used to execute the technical solution of the batch data unloading processing method in the above embodiment. Its implementation principle and technical effect are similar, and will not be described again here.

[0098] It should be noted that the division of the various modules in the above device is merely a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, these modules can be implemented entirely in software via processing element calls; they can be fully implemented in hardware; or some modules can be implemented by processing element calls to software, while others are implemented in hardware. For example, the basic information determination module can be a separate processing element, or it can be integrated into a chip in the above device. Alternatively, it can be stored as program code in the memory of the above device, and its function can be called and executed by a processing element. The implementation of other modules is similar. Moreover, these modules can be fully or partially integrated together, or they can be implemented independently. The processing element here can be an integrated circuit with signal processing capabilities. In the implementation process, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0099] A third aspect of this application provides an electronic device, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown, the electronic device includes a processor A01, a network interface A02, a memory (not shown), and a database (not shown) connected via a system bus. The processor A01 provides computing and control capabilities. The memory includes internal memory A03 and a non-volatile storage medium A04. The non-volatile storage medium A04 stores an operating system B01, a computer program B02, and a database (not shown). The internal memory A03 provides an environment for the operation of the operating system B01 and the computer program B02 stored in the non-volatile storage medium A04. The database stores raw data. The network interface A02 of the computer device is used for communication with external terminals via a network connection. When the computer program B02 is executed by the processor A01, it implements a batch data unloading processing method.

[0100] Those skilled in the art will understand that Figure 4 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements.

[0101] In one embodiment, the batch data unloading processing apparatus provided in this application can be implemented as a computer program, and the computer program can be implemented as follows: Figure 4 The device operates on the electronic device shown. The memory of the electronic device can store various program modules that make up the batch data unloading processing device, and the computer program composed of the various program modules causes the processor to execute the steps in the batch data unloading processing methods of the various embodiments of this application described in this specification.

[0102] The fourth aspect of this application provides a storage medium on which a program is stored, which, when executed by a processor, implements the above-described batch data unloading processing method.

[0103] A fifth aspect of this application provides a computer program product, including a computer program that, when executed by a processor, implements the above-described batch data unloading processing method.

[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0108] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0109] Memory may include non-persistent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0110] Computer-readable media include both permanent and non-permanent, removable and non-removable media, which can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0111] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0112] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for batch data unloading and processing, characterized in that, The unloading process includes: In response to the unloading request, determine the first basic information of the database to be loaded and the second basic information of the database to be unloaded; Based on the first basic information and the second basic information, common data unloading configuration items that meet the current data unloading task requirements are extracted from the data unloading task configuration file. The data unloading task configuration file is used to encapsulate different types of data unloading task logical parameters to form common data unloading configuration items under different business scenarios. The common data unloading configuration items include data source information, data unloading strategy information and database field information. The data unloading strategy information includes incremental data unloading parameters, full data unloading parameters, regular condition configuration parameters and custom condition configuration parameters. Based on the aforementioned public data unloading configuration items, a structured query statement is generated; Streaming query is used to execute structured query statements to obtain unloading data exported from the database to be unloaded; The unloaded data is generated into an unloaded file and transferred to the database to be loaded.

2. The batch data unloading processing method according to claim 1, characterized in that, The data source information includes: database connection parameters, data source type, and table name; the data unloading strategy information includes: incremental data unloading parameters, full data unloading parameters, regular condition configuration parameters, and custom condition configuration parameters; the database field information includes: field name, field format, and field sorting.

3. The batch data unloading processing method according to claim 1, characterized in that, The process employs streaming query execution of structured query statements to obtain unloading data exported from the database to be unloaded, including: Read the data source information and database field information of the database to be unloaded, and validate the structured query statement; The validated structured query statement is injected into the streaming query executor to retrieve strings that meet the query conditions in batches. Concatenate all strings that match the query criteria to form the unloaded data.

4. The batch data unloading processing method according to claim 3, characterized in that, The several numeric segments obtained after the streaming query are concatenated into a string that meets the query conditions according to the specified concatenation rules.

5. The batch data unloading processing method according to claim 1, characterized in that, The unloading process also includes: The data volume information is determined based on the unloading file, and the data volume information includes: table name, data file size in bytes, and number of records in the data file; Generate a data volume description file from the data volume information and transfer it to the database to be loaded.

6. The batch data unloading processing method according to claim 1, characterized in that, The unloading process also includes: The data structure information is determined based on the unloading file, and the data structure information includes: table name, total number of fields, field name, and field data type length; Generate a data structure description file from the data structure information and transfer it to the database to be loaded.

7. A batch data unloading and processing device, characterized in that, The unloading processing device includes: The basic information determination module is used to determine the first basic information of the database to be loaded and the second basic information of the database to be unloaded in response to the unloading processing request. The configuration item extraction module is used to extract common data unloading configuration items that meet the requirements of the current data unloading task from the data unloading task configuration file based on the first basic information and the second basic information. The data unloading task configuration file is used to encapsulate different types of data unloading task logical parameters to form common data unloading configuration items under different business scenarios. The common data unloading configuration items include data source information, data unloading strategy information and database field information. The data unloading strategy information includes incremental data unloading parameters, full data unloading parameters, regular condition configuration parameters and custom condition configuration parameters. The query statement generation module is used to generate structured query statements based on the public data configuration items; The data unloading module is used to execute structured query statements using streaming queries to obtain unloaded data exported from the database to be unloaded; The file transfer module is used to generate an unloading file from the unloading data and transfer it to the database to be loaded.

8. An electronic device, characterized in that, include: A processor, and a memory communicatively connected to the processor; The processor executes the computer execution instructions stored in the memory to implement the data unloading processing method according to any one of claims 1 to 6.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that, When executed by a processor, the computer execution instructions cause the processor to be configured to perform the data offloading method as described in any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the data unloading processing method as described in any one of claims 1 to 6.

Images