Data processing method and device, electronic equipment and storage medium

Abstract

The present disclosure provides a data processing method and device, electronic equipment and storage medium, which can be applied to the technical field of big data. The method comprises the following steps: in response to receiving source data from a first database, parsing the source data to obtain parsed data, the parsed data comprising to-be-processed data; determining M processing steps and M database interfaces matched with the to-be-processed data, the database interfaces having a unique correspondence with the processing steps, M being greater than or equal to 1; calling N second databases by using the M database interfaces, so that the N second databases execute the M processing steps, N being greater than or equal to 1; and receiving N processing results and returning the N processing results to the first database, the N processing results coming from the N second databases.

Description

Technical Field

[0001] This disclosure relates to the field of big data technology, specifically to a data processing method, apparatus, electronic device, and storage medium. Background Technology

[0002] In database transformation scenarios, developers migrate data or services from a source database to a new database, modifying the data during the migration process to ensure the migrated data or services adapt to the new database. Alternatively, in data processing scenarios involving multiple databases, the source data in the source database is processed to adapt to the new database structure; for example, data tables are modified.

[0003] In existing technologies, data migration or processing scenarios involving multiple databases typically require developers to modify the source or target databases, or manually modify the data to be migrated, to ensure that the migrated or processed data can be accessed normally. However, modifying databases or data involves complex processing steps, resulting in long processing times, high costs, and low efficiency. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a data processing method, apparatus, electronic device and storage medium.

[0005] According to a first aspect of this disclosure, a data processing method is provided, comprising:

[0006] In response to receiving source data from the first database, the source data is parsed to obtain parsed data, which includes data to be processed.

[0007] Identify M processing steps and M database interfaces that match the data to be processed. There is a unique correspondence between the database interface and the processing step, and M is greater than or equal to 1.

[0008] M database interfaces are used to call N secondary databases, so that the N secondary databases execute M processing steps, where N is greater than or equal to 1; and

[0009] Receive N processing results and return N processing results to the first database. The N processing results come from N second databases.

[0010] According to embodiments of this disclosure, determining M processing steps and M database interfaces that match the data to be processed includes:

[0011] The execution path is determined based on the processing type of the data to be processed. The processing type represents the type of operation to be performed on the data to be processed.

[0012] Based on the execution path, determine M processing steps; and

[0013] Based on the driving information of the data to be processed, determine the database interface corresponding to the processing steps.

[0014] According to embodiments of this disclosure, the execution path includes a transmission path and a processing path; the processing steps include transmission steps.

[0015] Based on the execution path, M processing steps are determined, including:

[0016] If the execution path is determined to be a transmission path, determine the transmission steps that match the data to be processed;

[0017] When the execution path is determined to be the processing path, the first configuration parameter of the data to be processed is determined, and the first configuration parameter represents the processing logic of the data to be processed; the first configuration parameter is matched with the second configuration parameter to determine M processing steps, and the second configuration parameter represents the pre-determined standard processing logic.

[0018] According to embodiments of this disclosure, determining the database interface corresponding to the processing step based on the driving information of the data to be processed includes:

[0019] Based on the driver information, determine the database type that matches the data to be processed; and

[0020] Based on the database type and M processing steps, determine M database interfaces, which are used to call N second databases.

[0021] According to embodiments of this disclosure, parsing the data further includes data type; before determining the M processing steps and M database interfaces matching the data to be processed, it further includes:

[0022] Based on the data type, validate the data to be processed and obtain the validation results; and

[0023] If the verification result is successful, the data to be processed is stored in the data queue of the data transmission area.

[0024] According to embodiments of this disclosure, the data types include files and code command statements;

[0025] Based on the data type, the data to be processed is validated, and the validation results are obtained, including:

[0026] If the data type is determined to be a file, the data length of the data to be processed is determined based on the checksum of the data to be processed.

[0027] If the data length meets the preset conditions, the verification result is determined to be passed;

[0028] If the data type is determined to be a code command statement, the verification result is determined to be passed.

[0029] According to embodiments of this disclosure, the processing result includes execution result and execution data, wherein the execution data includes data obtained after processing the data to be processed by N second databases;

[0030] Receive N processing results and return N processing results to the first database, including:

[0031] Store N processing results into a result queue according to their timestamps; and

[0032] Based on the result queue, N processing results are returned in batches to the first database.

[0033] A second aspect of this disclosure provides a data processing apparatus, comprising:

[0034] The parsing module is used to respond to the received source data from the first database, parse the source data, and obtain parsed data, which includes the data to be processed.

[0035] The determination module is used to determine M processing steps and M database interfaces that match the data to be processed. There is a unique correspondence between the database interface and the processing step, and M is greater than or equal to 1.

[0036] The calling module is used to call N second databases using M database interfaces, so that the N second databases perform M processing steps, where N is greater than or equal to 1; and

[0037] The result return module is used to receive N processing results and return N processing results to the first database. The N processing results come from N second databases.

[0038] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data processing method described above.

[0039] A fourth aspect of this disclosure also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the data processing method described above.

[0040] The fifth aspect of this disclosure also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described data processing method.

[0041] The embodiments of this disclosure, in response to receiving source data from a first database, parse the source data to obtain parsed data, which includes data to be processed; determine M processing steps and M database interfaces matching the data to be processed; utilize the M database interfaces to call N second databases so that the N second databases execute the M processing steps; receive N processing results and return N processing results to the first database, thus realizing data processing. Since the data processing in these embodiments does not depend on the first and second databases, multiple database types can be flexibly switched through database interfaces, achieving decoupling of the data processing process from database types and deployment methods, thereby improving the flexibility and applicability of the data processing process.

[0042] Furthermore, since the data processing process interfaces with the database type and database deployment method, no specific program code modification is required for various database types. This enables data processing between multiple databases, reducing data processing time and costs and improving data processing efficiency. Attached Figure Description

[0043] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0044] Figure 1 This illustration schematically depicts an application scenario of the data processing method according to an embodiment of the present disclosure;

[0045] Figure 2 A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0046] Figure 3 A flowchart illustrating the processing steps and database interface determination method according to embodiments of the present disclosure is shown in the schematic diagram.

[0047] Figure 4 A flowchart illustrating a database interface determination method according to a specific embodiment of the present disclosure is shown schematically.

[0048] Figure 5 A flowchart illustrating a method for determining M database interfaces based on a first configuration parameter and a second configuration parameter according to an embodiment of the present disclosure is shown schematically.

[0049] Figure 6 This illustration schematically shows a data flow diagram of a data processing procedure according to a specific embodiment of the present disclosure;

[0050] Figure 7 A schematic diagram of a data processing architecture according to an embodiment of the present disclosure is shown.

[0051] Figure 8A schematic block diagram of a data processing apparatus according to embodiments of the present disclosure is shown; and

[0052] Figure 9 A block diagram of an electronic device suitable for a data processing method according to an embodiment of the present disclosure is shown schematically. Detailed Implementation

[0053] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0054] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0055] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0056] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0057] In the technical solutions disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure, and application of data (including but not limited to user personal information) comply with the provisions of relevant laws and regulations, necessary confidentiality measures have been taken, and they do not violate public order and good morals.

[0058] In data migration scenarios, it is necessary to migrate existing data and services from a source database to a new database. For the new database, the existing data in the source database is the source data. The source database and the new database have different database types. To ensure that the services from the source database function normally in the new database, and that the new database can access and use the source data, existing technologies generally employ the following methods.

[0059] One approach is to modify the storage format of the source data and the program code of the services in the source database based on information such as the database type and structure of the new database. After modifying the source data and services, the modified services and data are migrated to the new database, enabling the new database to access the data and services normally. Another approach is to modify the storage format of the source data and the program code of the services in the new database based on information such as the database type and structure of the source database after receiving the source data and services, enabling the new database to access the data and services normally.

[0060] However, existing technologies rely on either the original or new database for processing source data or services, and the processing of source data or services is only applicable to these two databases. Migrating data from the original or new database to other databases still requires modifying the program code. Furthermore, the data in the database involves multiple functions and tables, resulting in a large amount of code modification and complex operations for developers.

[0061] Therefore, data processing involving multiple databases presents technical problems such as long processing time, large amount of program modification, high processing cost, and low efficiency.

[0062] Embodiments of this disclosure provide a data processing method, comprising: in response to receiving source data from a first database, parsing the source data to obtain parsed data, the parsed data including data to be processed; determining M processing steps and M database interfaces that match the data to be processed, wherein there is a unique correspondence between the database interfaces and the processing steps, and M is greater than or equal to 1; using the M database interfaces to call N second databases so that the N second databases execute the M processing steps, and N is greater than or equal to 1; and receiving N processing results and returning the N processing results to the first database, wherein the N processing results come from the N second databases.

[0063] Figure 1 The illustration depicts an application scenario of the data processing method according to an embodiment of the present disclosure.

[0064] like Figure 1As shown, application scenario 100 according to this embodiment may include a first server cluster 101, a second server cluster 102, and a batch server 103. The network serves as a medium for providing communication links within the first server cluster 101 and the second server cluster 102, and between the first server cluster 101, the second server cluster 102, and the batch server 103. The network may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0065] The first server cluster 101 includes multiple first servers, and the second server cluster includes multiple second servers. Both the first and second servers include a database server. A batch server 103 is positioned between the first server cluster 101 and the second server cluster 102 for data processing.

[0066] It should be noted that the data processing method provided in this embodiment can generally be executed by the batch server 103. Correspondingly, the data processing apparatus provided in this embodiment can generally be located in the batch server 103. The data processing method provided in this embodiment can also be executed by a server or server cluster that is different from the batch server 103 and capable of communicating with the first server cluster 101, the second server cluster 102, and / or the batch server 103. Correspondingly, the data processing apparatus provided in this embodiment can also be located in a server or server cluster that is different from the batch server 103 and capable of communicating with the first server cluster 101, the second server cluster 102, and / or the batch server 103.

[0067] For example, batch server 103 can obtain source data from any one of the first servers in the first server cluster 101, parse the source data to obtain parsed data, determine M processing steps and M database interfaces that match the data to be processed, and use the M database interfaces to call any one or more second servers in the second server cluster 102 so that any one or more second servers can execute the M processing steps. Batch server 103 can also receive one or more processing results from the aforementioned one or more second servers and return the processing results to the aforementioned first servers.

[0068] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0069] The following will be based on Figure 1 The described scene, through Figures 2-7 The data processing method of the disclosed embodiments will be described in detail.

[0070] Figure 2A flowchart illustrating a data processing method according to an embodiment of the present disclosure is shown schematically.

[0071] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0072] In operation S210, in response to receiving source data from the first database, the source data is parsed to obtain parsed data, which includes data to be processed.

[0073] According to an embodiment of this disclosure, a first database serves as a source database and sends source data to a batch server so that the batch server can parse the source data, obtain parsed data, and perform subsequent operations on the parsed data.

[0074] According to embodiments of this disclosure, the first database can serve as a source database in a data migration scenario. The first database can also be understood as a database server, enabling it to retrieve source data from other servers or locally in response to user requests.

[0075] According to embodiments of this disclosure, a batch server is positioned between a first database and a second database, used to process source data suitable for the first database into data suitable for the second database, so that the processed data can be stored in the second database, or the second database can be invoked to process the processed data again. The second database can be understood as a database server.

[0076] According to an embodiment of this disclosure, after receiving source data from a first database, the source data is parsed to obtain parsed data, wherein the parsed data includes data to be processed.

[0077] According to embodiments of this disclosure, the data to be processed can be understood as various forms of information to be processed, such as files, requests, and command statements.

[0078] In operation S220, M processing steps and M database interfaces that match the data to be processed are determined. There is a unique correspondence between the database interface and the processing step, and M is greater than or equal to 1.

[0079] According to embodiments of this disclosure, based on a batch server, various operations with different execution logics can be performed on the data to be processed, such as calculation, statistics, and transmission.

[0080] According to embodiments of this disclosure, after obtaining the data to be processed by parsing the source data, M processing steps matching the data to be processed can be determined to determine the operation to be performed on the data to be processed based on the processing steps. Each processing step can perform at least one operation on the data to be processed.

[0081] According to embodiments of this disclosure, after determining the processing steps that match the data to be processed, the batch server can determine the database interface corresponding to the processing steps and call the second server through the database interface to process the data to be processed.

[0082] In operation S230, M database interfaces are used to call N second databases so that the N second databases can perform M processing steps, where N is greater than or equal to 1.

[0083] According to embodiments of this disclosure, the database interface is applicable to one or more database types, ensuring that source data can be migrated from a first database to a second database of multiple database types, and that data processing can be performed on the source data during the migration process.

[0084] According to embodiments of this disclosure, M database interfaces are used to invoke one or more second databases to execute the aforementioned M processing steps. One second database may correspond to multiple database interfaces to execute multiple processing steps.

[0085] According to embodiments of this disclosure, after determining M database interfaces, N second databases can be called using the M database interfaces to process the source data based on M processing steps to obtain a processing result. After the N second databases execute the M processing steps, each second database returns a processing result to the batch server.

[0086] In operation S240, N processing results are received and N processing results are returned to the first database. The N processing results come from N second databases.

[0087] According to embodiments of this disclosure, after calling N second databases using M database interfaces to execute M processing steps, N processing results are received from the N second databases. Since the batch server and the first database do not belong to the same physical machine or the same deployment environment, after receiving the N processing results, the batch server also returns the N processing results to the first database to ensure data consistency between the first and second databases.

[0088] The embodiments of this disclosure, in response to receiving source data from a first database, parse the source data to obtain parsed data, which includes data to be processed; determine M processing steps and M database interfaces matching the data to be processed; utilize the M database interfaces to call N second databases so that the N second databases execute the M processing steps; receive N processing results and return N processing results to the first database, thus realizing data processing. Since the data processing in these embodiments does not depend on the first and second databases, multiple database types can be flexibly switched through database interfaces, achieving decoupling of the data processing process from database types and deployment methods, thereby improving the flexibility and applicability of the data processing process.

[0089] Furthermore, since the data processing process interfaces with the database type and database deployment method, no specific program code modification is required for various database types. This enables data processing between multiple databases, reducing data processing time and costs and improving data processing efficiency.

[0090] Figure 3 The flowchart illustrating the processing steps and database interface determination method according to an embodiment of the present disclosure is shown in the illustration.

[0091] like Figure 3 As shown, the processing steps and database interface determination method 300 of this embodiment include operations S321 to S323, which can be used as a specific embodiment of operation S220.

[0092] In operation S321, the execution path is determined according to the processing type of the data to be processed. The processing type represents the type of operation to be performed on the data to be processed.

[0093] In operation S322, M processing steps are determined based on the execution path.

[0094] In operation S323, the database interface corresponding to the processing step is determined based on the driver information of the data to be processed.

[0095] According to embodiments of this disclosure, multiple processing types correspond to multiple execution paths. Processing types include transmission and processing, and each processing type corresponds to one execution path. For example, when the processing type is "transmission," it indicates that no operation is performed on the data to be processed, only transmission is performed, and the corresponding execution path is the transmission path; or, when the processing type is "processing," it indicates that simple data processing operations, such as statistics, are performed on the data to be processed, and the corresponding execution path is the processing path.

[0096] According to embodiments of this disclosure, for data migration scenarios, if the first database and the second database are of the same database type, such as both being MySQL databases, and the table structures in the databases have not changed, the data processing process can be: transferring the source data obtained from the first database to the second database without further processing.

[0097] According to embodiments of this disclosure, in data processing or data migration scenarios where the first and second databases are of different types, the data processing process may involve: obtaining source data from the first database, modifying or processing the source data, and then transmitting it to the second database. For example, modifying the source data may involve modifying the database table structure, adding fields, deleting fields, merging fields, etc. Database processing may involve calculating the amount of data under any given field.

[0098] According to embodiments of this disclosure, after determining the execution path, processing operations on the data to be processed can be performed based on the execution path. Each execution path can correspond to one or more processing steps to perform the corresponding processing operation.

[0099] According to embodiments of this disclosure, each processing step can be implemented by a different second database or a different execution method. Therefore, after determining M processing steps, the database interface corresponding to each processing step is determined based on the driving information of the data to be processed.

[0100] The embodiments of this disclosure determine the database interface through the driving information of the data to be processed, so as to call a second database that matches the database interface, thereby realizing indirect data interaction between the first database and the second database.

[0101] According to embodiments of this disclosure, the execution path includes a transmission path and a processing path; the processing steps include transmission steps; based on the execution path, M processing steps are determined, including: when the execution path is determined to be a transmission path, determining a transmission step that matches the data to be processed; when the execution path is determined to be a processing path, determining a first configuration parameter of the data to be processed, the first configuration parameter representing the processing logic of the data to be processed; matching the first configuration parameter with a second configuration parameter to determine M processing steps, the second configuration parameter representing a predetermined standard processing logic.

[0102] According to embodiments of this disclosure, when the execution path is determined to be a transmission path, since there is no need to process the data to be processed, it is only necessary to transmit the data to be processed to the second database. Therefore, the transmission operation of the data to be processed can be achieved through a single transmission step and a corresponding transmission interface.

[0103] For example, when the data to be processed is a file, the third and fourth configuration parameters of the data can be matched to determine a unique transmission step; then, the transmission operation of the data to be processed is implemented through the transmission interface corresponding to the transmission step. The database interface includes the transmission interface.

[0104] According to embodiments of this disclosure, the third configuration parameter is used to characterize transmission-related parameters in the data to be processed, such as the transmission protocol. The fourth configuration parameter characterizes predetermined transmission-related parameters.

[0105] Figure 4 The flowchart illustrating a database interface determination method according to a specific embodiment of the present disclosure is shown schematically.

[0106] like Figure 4 As shown in the figure, the database interface determination method 400 illustrates the process of determining the database interface.

[0107] According to embodiments of this disclosure, the execution path of the data to be processed 401 is determined based on the processing type of the data to be processed 401.

[0108] If the execution type is determined to be a processing path, the first configuration parameter 402 of the data 401 to be processed is determined. Matching the second configuration parameter 403 in the configuration area with the first configuration parameter 402 determines the processing step 406. The configuration area is a region in the data processing device used to perform parameter matching operations.

[0109] According to embodiments of this disclosure, both the first configuration parameter 402 and the second configuration parameter 403 may include multiple configuration parameters. For example, the first configuration parameter 402 may include M first configuration parameters, and each first configuration parameter may be matched with the second configuration parameter 403 to determine the processing steps that match each first configuration parameter, thereby obtaining the processing steps corresponding to each first configuration parameter.

[0110] If the execution type is determined to be a transmission path, the third configuration parameter 404 of the data to be processed 401 is determined. Matching the fourth configuration parameter 405 and the third configuration parameter 404 in the configuration area determines the transmission step 407. The configuration area is a region in the data processing device used to perform parameter matching operations.

[0111] According to an embodiment of this disclosure, the third configuration parameter 404 and the fourth configuration parameter 405 are matched to determine the transmission step 407 that matches the third configuration parameter 404.

[0112] According to embodiments of this disclosure, after determining processing step 406 or transmission step 407, the corresponding database interface 408 can be determined based on processing step 406 or transmission step 407. For example, the database interface corresponding to transmission step 407 is a transmission interface.

[0113] In this embodiment of the disclosure, determining M processing steps can break down the entire processing process into multiple parallel processing steps, thereby improving processing efficiency.

[0114] According to embodiments of this disclosure, determining the database interface corresponding to the processing step based on the driving information of the data to be processed includes the following steps:

[0115] Based on the driver information, determine the database type that matches the data to be processed.

[0116] Based on the database type and M processing steps, determine M database interfaces, which are used to call N second databases.

[0117] According to embodiments of this disclosure, driver information can be understood as a special identifier that can characterize the database type. The driver information for the data to be processed may include the database type of the first database.

[0118] For example, when the data to be processed is code, the code writing style or calling logic of the data to be processed can reflect the type of the source database. Therefore, the code writing style or calling logic can be used as driving information to identify the database type of the first database.

[0119] According to embodiments of this disclosure, based on the database type of the first database characterized by the driving information, a database interface matching the first database can be determined for each processing step.

[0120] According to embodiments of this disclosure, the database interface is written using application program interface (API) methods in an open-source driver. Embodiments of this disclosure allow direct invocation of pre-written database interfaces applicable to various database types.

[0121] According to embodiments of this disclosure, most databases are derived from PG through secondary development. Therefore, embodiments of this disclosure can support data interaction between multiple database types by directly calling pre-written database interfaces applicable to various database types, thereby improving the applicability of the data processing method.

[0122] Figure 5 The flowchart illustrating a method for determining M database interfaces based on a first configuration parameter and a second configuration parameter according to an embodiment of the present disclosure is shown schematically.

[0123] like Figure 5 As shown, when the data to be processed includes M first configuration parameters, the method 500 for determining M database interfaces based on the first configuration parameters and the second configuration parameters demonstrates the process of determining M database interfaces.

[0124] According to embodiments of this disclosure, the first configuration parameter 502 includes M first configuration parameters, such as the first first configuration parameter 502_1..., the mth first configuration parameter 502_m..., and the Mth first configuration parameter 502_M.

[0125] For each first configuration parameter, the first configuration parameter is matched with the second configuration parameter 503 to obtain M processing steps 506 that match the M first configuration parameters. For example, the first processing step 506_1..., the mth processing step 506_m..., and the Mth processing step 506_M.

[0126] For each processing step, a corresponding database interface 508 is determined. For example, the first database interface 508_1 corresponds to the first processing step 506_1, ..., the m-th database interface 508_m corresponds to the m-th processing step 506_m, ..., and the M-th database interface 508_M corresponds to the M-th processing step 506_M.

[0127] According to embodiments of this disclosure, the processing operations on the data to be processed are all integrated on a batch server outside of the first and second databases. Therefore, embodiments of this disclosure do not depend on the database types of the two parties interacting with the data, but only on the database driver API, and are essentially imperceptible to database migration.

[0128] According to embodiments of this disclosure, parsing the data further includes data type; before determining the M processing steps and M database interfaces matching the data to be processed, the method further includes: validating the data to be processed according to the data type to obtain a validation result. If the validation result is successful, the data to be processed is stored in the data queue of the data transmission area.

[0129] According to embodiments of this disclosure, data types are used to characterize the type of data to be processed. Data types include files and code command statements.

[0130] According to embodiments of this disclosure, different verification operations are performed on the data to be processed for different data types to obtain verification results. Only after the data to be processed has been verified will the data processing operation be performed by calling a second database through the database interface.

[0131] Specifically, if the data to be processed passes the verification, the data to be processed is stored in the data queue of the data processing area so that the data to be processed can be retrieved from the data queue later to determine the processing steps for the data to be processed.

[0132] According to embodiments of this disclosure, since data processing operations are integrated into a batch server, processing operations are implemented by calling a second database. By verifying the data before determining the processing steps, database interface, and the call to the second database, processing failures due to data issues can be avoided, further reducing invalid interactions between the batch database and the second database, and minimizing waste of computing resources.

[0133] According to embodiments of this disclosure, the data types include files and code command statements. Based on the data type, the data to be processed is validated to obtain a validation result, including the following steps:

[0134] If the data type is determined to be a file, the length of the data to be processed is determined based on the checksum of the data to be processed.

[0135] If the data length meets the preset conditions, the verification result is determined to be passed.

[0136] If the data type is determined to be a code command statement, the verification result is determined to be passed.

[0137] According to embodiments of this disclosure, when the data type is a file, it is necessary to verify whether the file is complete, and subsequent processing is performed if the file is complete. When the data type is a code command statement, since the code command statement conforms to the writing format of the first database, a verification operation can be skipped, and the verification result can be directly determined as passed.

[0138] According to embodiments of this disclosure, when the data type is determined to be a file, the verification operation includes: determining the data length of the data to be processed based on the verification code of the data to be processed, so as to determine the verification result based on whether the data length meets a preset condition. The preset condition may be that the data length reaches a length threshold.

[0139] According to embodiments of this disclosure, when the data type is determined to be a file, the verification operation further includes: determining whether the data is complete based on the hash code of the data to be processed.

[0140] According to embodiments of this disclosure, a batch server can call a second database with a single interaction and then obtain the processing result from the second database with another interaction. If the integrity of the files is not verified, and a file has a problem, only a failed call result will be received after two interactions, leading to wasted resources.

[0141] In the embodiments disclosed herein, file integrity is verified through a batch server. In the event of verification failure, no interactive operation is required, avoiding resource waste and improving processing efficiency. Furthermore, the verification operation is also handled by the batch server, eliminating the need for a first and second database, further reducing dependence on database types and lowering coupling.

[0142] According to embodiments of this disclosure, the processing result includes execution result and execution data, wherein the execution data includes data obtained after processing the data to be processed by N second databases.

[0143] According to embodiments of this disclosure, receiving N processing results and returning N processing results to a first database includes: storing the N processing results to a result queue according to timestamps; and returning the N processing results to the first database in batches based on the result queue.

[0144] According to embodiments of this disclosure, after a batch server invokes N second databases to perform M processing steps, it receives N processing results from the N second databases. Receiving N processing results indicates that the processing operation has been completed. The N processing results are stored in a result queue according to their timestamps so that, in the event of processing failure, the source of failure can be located based on the timestamps and processing results.

[0145] According to embodiments of this disclosure, the result queue can return N processing results one by one in a first-in, first-out manner. Alternatively, the N processing results in the result queue can be returned to the first database all at once.

[0146] Figure 6 A data flow diagram illustrating a data processing procedure according to a specific embodiment of the present disclosure is shown schematically.

[0147] like Figure 6 As shown, the data flow diagram 600 includes a first database 601, a batch server 602, a data processing device 603 deployed in the batch server 602, and a second database 604.

[0148] According to embodiments of this disclosure, the data processing device includes an input area A1, a control area A2, a processing area A3, and an output area A4. The input area A1 primarily receives scheduling commands or files and submits processing instructions to the control area A2 for processing. The control area A2 primarily controls the execution of various commands, including reading various configurations, obtaining batch processing steps, and scheduling the processing area A3 for processing. The processing area A3 mainly performs various atomic functions, primarily used to execute data processing-related functions according to the control area's scheduling, including content requiring interaction with the database, and feeds back the execution results to the control area A2. The output area A4 primarily receives the execution results and files from various commands in the control area A2.

[0149] According to an embodiment of this disclosure, the data processing procedure includes: a first database 601 transmitting code command statements or files to a batch server 602. At this time, if the input area A1 recognizes a file or code command statement, it transmits the file or code command statement to the control area A2.

[0150] Control area A2 reads the corresponding configuration parameters and currently active driver information based on the current file or code command statement. After calculating and assembling the processing steps, it invokes various unit services in processing area A3.

[0151] Processing area A3 calls the second database 604 through a database interface based on various unit services, obtains the processing results from the second database 604, and feeds the processing results back to control area A2. Control area A2 parses the processing results and feeds them back to output area A4.

[0152] The batch server 602 retrieves the processing results from output area A4, enabling seamless batch migration and providing stable and efficient background batch services for various systems, databases, and servers.

[0153] Figure 7 A schematic diagram of a data processing architecture according to an embodiment of the present disclosure is shown.

[0154] like Figure 7 As shown in the diagram, the data processing architecture diagram 700 illustrates the specific architecture of the input area A1, control area A2, processing area A3, and output area A4 within the data processing device.

[0155] like Figure 7 As shown, input area A1 includes an identification area, a file receiving area, a command receiving area, and a data transmission area. The identification area can automatically scan and check whether the data processing device has received a file or code command statement.

[0156] According to embodiments of this disclosure, when the identification area recognizes a code command statement, it transmits the code command statement to the command receiving area so that the command receiving area can perform preliminary processing on the code command statement and determine whether control area processing is required. If the command receiving area determines that control area processing of the code command statement is required, it transfers the code command statement to the data transmission area. When the identification area recognizes a file, it verifies information such as the file's integrity. If the verification passes, the file is transferred to the data transmission area. The data transmission area includes a data queue, where files or code command statements are stored.

[0157] Control area A2 includes a parsing area, a configuration area, and a scheduling area. The parsing area retrieves files or code command statements from the data transmission area, then parses them to obtain either a first or third configuration parameter. Next, the parsing area matches the first or third configuration parameter with the second or fourth configuration parameter from the configuration area. If a match is found, the processing step is determined. The scheduling area, based on the processing step, calls the units in processing area A3, receives the processing results returned by processing area A3, and determines whether to return a result to output area A4 based on whether the batch steps have been completed.

[0158] Processing zone A3 includes data processing 1, file import 1, file generation 1, ..., atomic service N. Processing zone A3 can include multiple atomic services, such as atomic service 1, atomic service 2, ..., atomic service N, each being a minimum processing unit. Processing zone A3 responds to scheduling requests from control zone A2, automatically selects the matching API based on driver information, and calls the database server for data processing; after the call is complete, it feeds back the return results from the second database to control zone A2. All processing operations are integrated within the atomic services of the processing zone, independent of the database type, relying only on the database driver's API, making database migration virtually imperceptible.

[0159] Output area A4 includes a result queue, which stores the processing results of all batch executions.

[0160] According to embodiments of this disclosure, the data processing apparatus can be implemented as a plug-in tool to implement the above-described data processing method.

[0161] Figure 8 A schematic block diagram of a data processing apparatus according to an embodiment of the present disclosure is shown.

[0162] like Figure 8 As shown, the data processing device 800 of this embodiment includes a parsing module 810, a determining module 820, a calling module 830, and a result returning module 840.

[0163] The parsing module 810 is used to parse the source data received from the first database in response to obtain parsed data, which includes data to be processed. In one embodiment, the parsing module 810 can be used to perform the operation S210 described above, which will not be repeated here.

[0164] The determining module 820 is used to determine M processing steps and M database interfaces that match the data to be processed. There is a unique correspondence between the database interfaces and the processing steps, and M is greater than or equal to 1. In one embodiment, the determining module 820 can be used to execute the operation S220 described above, which will not be repeated here.

[0165] The calling module 830 is used to call N second databases using M database interfaces, so that the N second databases perform M processing steps, where N is greater than or equal to 1. In one embodiment, the calling module 830 can be used to execute the operation S230 described above, which will not be repeated here.

[0166] The result return module 840 is used to receive N processing results and return N processing results to the first database, wherein the N processing results come from N second databases. In one embodiment, the result return module 840 can be used to perform the operation S240 described above, which will not be repeated here.

[0167] According to embodiments of this disclosure, the determining module 820 includes a first determining unit, a second determining unit, and a third determining unit.

[0168] The first determining unit is used to determine the execution path based on the processing type of the data to be processed, whereby the processing type represents the type of operation to be performed on the data to be processed. In one embodiment, the first determining unit can be used to execute the operation S321 described above, which will not be repeated here.

[0169] The second determining unit is used to determine M processing steps based on the execution path. In one embodiment, the second determining unit can be used to perform the operation S322 described above, which will not be repeated here.

[0170] The third determining unit is used to determine the database interface corresponding to the processing step based on the driving information of the data to be processed. In one embodiment, the third determining unit can be used to perform the operation S323 described above, which will not be repeated here.

[0171] According to embodiments of this disclosure, the execution path includes a transmission path and a processing path; the processing steps include transmission steps.

[0172] According to embodiments of this disclosure, the second determining unit includes a first determining subunit and a second determining subunit.

[0173] The first determining subunit is used to determine the transmission steps that match the data to be processed when the execution path is determined to be a transmission path.

[0174] The second determining subunit is used to determine the first configuration parameter of the data to be processed when the execution path is determined to be the processing path. The first configuration parameter represents the processing logic of the data to be processed. The first configuration parameter is matched with the second configuration parameter to determine M processing steps. The second configuration parameter represents the pre-determined standard processing logic.

[0175] According to embodiments of this disclosure, the third determining unit includes a third determining subunit and a fourth determining subunit.

[0176] The third determining subunit is used to determine the database type that matches the data to be processed based on the driving information.

[0177] The fourth determining subunit is used to determine M database interfaces based on the database type and M processing steps. These M database interfaces are used to call N second databases.

[0178] According to embodiments of this disclosure, the parsed data also includes a data type, and the data processing device 800 further includes a verification module and a storage module.

[0179] The validation module is used to validate the data to be processed according to the data type and obtain the validation result.

[0180] The storage module is used to store the data to be processed into the data queue of the data transmission area when the verification result is determined to be successful.

[0181] According to embodiments of this disclosure, the data types include files and code command statements. The verification module includes a first verification unit, a second verification unit, and a third verification unit.

[0182] The first verification unit is used to determine the data length of the data to be processed based on the verification code of the data to be processed, provided that the data type is determined to be a file.

[0183] The second verification unit is used to determine that the verification result is passed if the data length meets the preset conditions.

[0184] The third verification unit is used to determine that the verification result is passed if the data type is determined to be a code command statement.

[0185] According to embodiments of this disclosure, the processing result includes execution result and execution data, whereby the execution data includes data obtained after processing the data to be processed from N second databases. The result return module 840 includes a result storage unit and a return unit.

[0186] The result storage unit is used to store N processing results into the result queue according to their timestamps.

[0187] The return unit is used to return N processing results in batches to the first database based on the result queue.

[0188] According to embodiments of this disclosure, any and multiple modules among the parsing module 810, determining module 820, calling module 830, and result returning module 840 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least some of the functionality of one or more of these modules can be combined with at least some of the functionality of other modules and implemented in one module.

[0189] According to embodiments of this disclosure, at least one of the parsing module 810, determining module 820, calling module 830, and result returning module 840 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable method of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three methods. Alternatively, at least one of the parsing module 810, determining module 820, calling module 830, and result returning module 840 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0190] Figure 9 A block diagram of an electronic device suitable for a data processing method according to an embodiment of the present disclosure is shown schematically.

[0191] like Figure 9 As shown, an electronic device 900 according to an embodiment of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage portion 908 into a random access memory (RAM) 903. The processor 901 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 901 may also include onboard memory for caching purposes. The processor 901 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0192] RAM 903 stores various programs and data required for the operation of electronic device 900. Processor 901, ROM 902, and RAM 903 are interconnected via bus 904. Processor 901 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 902 and / or RAM 903. It should be noted that the programs may also be stored in one or more memories other than ROM 902 and RAM 903. Processor 901 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0193] According to embodiments of this disclosure, the electronic device 900 may further include an input / output (I / O) interface 905, which is also connected to a bus 904. The electronic device 900 may also include one or more of the following components connected to the input / output I / O interface 905: an input section 906 including a keyboard, mouse, etc.; an output section 907 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as needed. A removable medium 911, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 910 as needed so that computer programs read from it can be installed into the storage section 908 as needed.

[0194] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0195] According to embodiments of this disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 902 and / or RAM 903 and / or one or more memories other than ROM 902 and RAM 903 described above.

[0196] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the methods provided in the embodiments of this disclosure.

[0197] When the computer program is executed by the processor 901, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0198] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 909, and / or installed from a removable medium 911. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0199] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 909, and / or installed from the removable medium 911. When the computer program is executed by the processor 901, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0200] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0201] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0202] Those skilled in the art will understand that the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0203] The specific embodiments described above further illustrate the purpose, technical solutions, and beneficial effects of this disclosure. It should be understood that the above descriptions are merely specific embodiments of this disclosure and are not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. A data processing method applied to a batch server between a first database and a second database, wherein the physical machine or deployment environment of the batch server is different from that of the first database, the method comprising: In response to receiving source data from a first database, the source data is parsed to obtain parsed data, which includes data to be processed; Identify M processing steps and M database interfaces that match the data to be processed, wherein there is a unique correspondence between the database interface and the processing step, and M is greater than or equal to 1; The M database interfaces are used to call N second databases so that the N second databases can execute the M processing steps, where N is greater than or equal to 1; as well as Receive N processing results and return the N processing results to the first database, wherein the N processing results come from the N second databases; The determination of the M processing steps and M database interfaces that match the data to be processed includes: The execution path is determined based on the processing type of the data to be processed, whereby the processing type represents the type of operation performed on the data to be processed; multiple processing types correspond to multiple execution paths. If the execution path is determined to be a transmission path, a transmission step matching the data to be processed is determined; if the data to be processed is a file, the third configuration parameter and the fourth configuration parameter of the data to be processed are matched to determine a unique transmission step; and the transmission operation of the data to be processed is realized through the transmission interface corresponding to the transmission step; wherein, the database interface includes a transmission interface, the third configuration parameter is used to characterize the transmission-related parameters in the data to be processed, and the fourth configuration parameter characterizes the predetermined transmission-related parameters; If the execution path is determined to be a processing path, a first configuration parameter for the data to be processed is determined, wherein the first configuration parameter represents the processing logic of the data to be processed; the first configuration parameter is matched with a second configuration parameter to determine the M processing steps, wherein the second configuration parameter represents a pre-determined standard processing logic; and Based on the driving information of the data to be processed, determine the database interface corresponding to the processing step.

2. The method according to claim 1, wherein, The step of determining the database interface corresponding to the processing step based on the driver information of the data to be processed includes: Based on the driving information, determine the database type that matches the data to be processed; and Based on the database type and the M processing steps, the M database interfaces are determined, and the M database interfaces are used to call the N second databases.

3. The method according to claim 1, wherein, The parsed data also includes data types; before determining the M processing steps and M database interfaces that match the data to be processed, the following is also included: Based on the data type, the data to be processed is validated to obtain a validation result; and If the verification result is found to be successful, the data to be processed is stored in the data queue of the data transmission area.

4. The method according to claim 3, wherein, The data types include files and code command statements; The step of validating the data to be processed according to the data type and obtaining the validation result includes: If the data type is determined to be a file, the data length of the data to be processed is determined based on the checksum of the data to be processed. If the data length meets the preset conditions, the verification result is determined to be passed; If the data type is determined to be a code command statement, the verification result is determined to be passed.

5. The method according to claim 1, wherein, The processing result includes the execution result and the execution data, wherein the execution data includes the data obtained by the N second databases after processing the data to be processed; The step of receiving N processing results and returning the N processing results to the first database includes: Store the N processing results into the result queue according to their timestamps; as well as Based on the result queue, the N processing results are returned to the first database in batches.

6. A data processing apparatus, comprising: A parsing module is used to respond to receiving source data from a first database, parse the source data, and obtain parsed data, the parsed data including data to be processed; The determination module is used to determine M processing steps and M database interfaces that match the data to be processed, wherein there is a unique correspondence between the database interface and the processing step, and M is greater than or equal to 1; The calling module is used to call N second databases using the M database interfaces, so that the N second databases execute the M processing steps, where N is greater than or equal to 1; as well as The result return module is used to receive N processing results and return the N processing results to the first database, wherein the N processing results come from the N second databases; The determination of the M processing steps and M database interfaces that match the data to be processed includes: The execution path is determined based on the processing type of the data to be processed, whereby the processing type represents the type of operation performed on the data to be processed; multiple processing types correspond to multiple execution paths. If the execution path is determined to be a transmission path, a transmission step matching the data to be processed is determined; if the data to be processed is a file, the third configuration parameter and the fourth configuration parameter of the data to be processed are matched to determine a unique transmission step; and the transmission operation of the data to be processed is realized through the transmission interface corresponding to the transmission step; wherein, the database interface includes a transmission interface, the third configuration parameter is used to characterize the transmission-related parameters in the data to be processed, and the fourth configuration parameter characterizes the predetermined transmission-related parameters; If the execution path is determined to be a processing path, a first configuration parameter for the data to be processed is determined, wherein the first configuration parameter represents the processing logic of the data to be processed; the first configuration parameter is matched with a second configuration parameter to determine the M processing steps, wherein the second configuration parameter represents a pre-determined standard processing logic; and Based on the driving information of the data to be processed, determine the database interface corresponding to the processing step.

7. An electronic device, comprising: One or more processors; Storage device for storing one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors perform the method according to any one of claims 1 to 5.

8. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, cause the processor to perform the method according to any one of claims 1 to 5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 5.

Images