A cross-system data integration method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application disclose a cross-system data integration method and device, electronic equipment and a storage medium. The method comprises: in a development stage, determining integration configuration information of the development stage based on configuration parameters input by a user, integrating test data from a source system of a development environment to a target system of the development environment based on the integration configuration information of the development stage, and generating and exporting an integration configuration installation package based on the integration configuration information of the development stage; and in a production stage, importing and parsing the integration configuration installation package, determining integration configuration information of the production stage based on a parsing result of the integration configuration installation package, and integrating business data from a source system of a production environment to a target system of the production environment based on the integration configuration information of the production stage. The embodiments of the present application improve the production efficiency and production accuracy of a data integration scheme.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a cross-system data integration method, apparatus, electronic device, and storage medium. Background Technology

[0002] As enterprises accelerate their digital transformation, the demand for cross-system data flow is constantly increasing, making data integration technology a key measure to break the information silo dilemma and promote seamless data linkage across the entire domain. For example, in the data integration of payment systems, the batch data integration process is roughly as follows: first, the data is unloaded into file form, then sent to the target system through a dedicated data transmission channel, and finally, the file content is imported into the big data platform using a data loading tool. However, this model has low deployment efficiency. When the development environment and production environment are separated, data integration configuration needs to be repeated, and the deployment process is highly dependent on manual operation, which is not only time-consuming but also prone to missing key steps, affecting overall efficiency and accuracy. Summary of the Invention

[0003] This invention provides a cross-system data integration method, apparatus, electronic device, and storage medium, which can improve the efficiency and accuracy of data integration solutions in production.

[0004] In a first aspect, embodiments of the present invention provide a cross-system data integration method, comprising:

[0005] During the development phase, integration configuration information is determined based on user-inputted configuration parameters. Based on this integration configuration information, test data is integrated from the source system in the development environment to the target system. An integration configuration installation package is then generated and exported based on the integration configuration information.

[0006] During the production phase, the integrated configuration installation package is imported and parsed. Based on the parsing results of the integrated configuration installation package, the integrated configuration information for the production phase is determined. Based on the integrated configuration information for the production phase, business data is integrated from the source system in the production environment to the target system in the production environment.

[0007] Secondly, embodiments of the present invention provide a cross-system data integration device, comprising:

[0008] The development phase data integration module is used to determine the integration configuration information during the development phase based on user-input configuration parameters, integrate test data from the source system to the target system in the development environment based on the development phase integration configuration information, and generate and export the integration configuration installation package based on the development phase integration configuration information; and

[0009] The production stage data integration module is used to import and parse the integration configuration installation package during the production stage, determine the integration configuration information for the production stage based on the parsing results of the integration configuration installation package, and integrate business data from the source system in the production environment to the target system in the production environment based on the integration configuration information for the production stage.

[0010] Thirdly, embodiments of the present invention also provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the cross-system data integration method as described in any of the embodiments of the present invention.

[0011] Fourthly, embodiments of the present invention also provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the cross-system data integration method as described in any of the embodiments of the present invention.

[0012] Fifthly, embodiments of the present invention also provide a computer program product, including a computer program that, when executed by a processor, implements the cross-system data integration method as described in any of the embodiments of the present invention.

[0013] This invention provides a cross-system data integration method, apparatus, electronic device, and storage medium. During the development phase, integration configuration information is determined based on user-inputted configuration parameters. Based on this development phase integration configuration information, test data is integrated from the source system in the development environment to the target system. An integration configuration installation package is generated and exported based on the development phase integration configuration information. In the production phase, the integration configuration installation package is imported and parsed. Based on the parsing results of the integration configuration installation package, production phase integration configuration information is determined. Based on the production phase integration configuration information, business data is integrated from the source system in the production environment to the target system. This allows for cross-system reuse of integration configuration information during the production phase, enabling automated deployment of data integration tasks in production mode. It effectively avoids the problems of repeated deployment and high reliance on manual intervention caused by environmental differences, improving deployment efficiency and accuracy, thereby improving the production efficiency and accuracy of the data integration solution. Furthermore, this embodiment, through the reuse of process configurations and installation package management, significantly reduces operational complexity and makes system maintenance more convenient and efficient. Attached Figure Description

[0014] To more clearly illustrate the technical solution of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart illustrating a cross-system data integration method provided in an embodiment of the present invention;

[0016] Figure 2 This is another flowchart illustrating the cross-system data integration method provided in this embodiment of the invention;

[0017] Figure 3 This is another flowchart illustrating the cross-system data integration method provided in this embodiment of the invention;

[0018] Figure 4 This is another flowchart illustrating the cross-system data integration method provided in this embodiment of the invention;

[0019] Figure 5 This is a schematic diagram of a cross-system data integration device provided in an embodiment of the present invention;

[0020] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0021] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0022] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0023] Figure 1This is a flowchart illustrating a cross-system data integration method provided in an embodiment of the present invention. This method can be executed by a cross-system data integration device provided in this embodiment, which can be implemented in software and / or hardware. In a specific embodiment, the device can be integrated into an electronic device, such as a computer or server. The following embodiments will illustrate this using the integration of the device into an electronic device as an example. (Reference) Figure 1 The method may specifically include the following steps:

[0024] Step 101: During the development phase, the integration configuration information is determined based on the configuration parameters input by the user. Based on this information, test data is integrated from the source system in the development environment to the target system. Finally, the integration configuration installation package is generated and exported based on the development phase integration configuration information. This step facilitates the cross-system reuse of the integration configuration information during the production phase, enabling automated deployment of data integration tasks in production.

[0025] Specifically, the aforementioned source system and target system can be heterogeneous database systems, or business application systems that include heterogeneous databases.

[0026] Specifically, configuration parameters input by the user can be obtained through a guided configuration interface before determining the integration configuration information during the development phase based on the configuration parameters input by the user.

[0027] Specifically, the aforementioned guided configuration interface can cluster parameter configurations, with each category of parameter configuration information placed on the same page, and each page displayed via a "next" tab.

[0028] Specifically, the integrated configuration information can include: process information, data object information, exchange parameter information, and file transfer parameter information.

[0029] Specifically, for multiple data tables corresponding to a data integration process, the process information corresponding to each data table can be different, and the data object information corresponding to each data table can also be different; the exchange parameter information corresponding to each data table can be the same, and the file transfer parameter information corresponding to each data table can also be the same.

[0030] Specifically, the aforementioned process information can be understood as information used to accurately describe and configure the complete execution logic of data exchange tasks during the data integration process. It may include information such as the type of data entering the lake, triggering mechanism, execution steps, dependencies, exception handling strategies and / or logging rules. It is a set of metadata that enables the automation, observability and manageability of the data exchange process.

[0031] Specifically, the aforementioned data object information can be understood as a set of metadata used to uniquely define and describe the structure of the data itself being exchanged during the data integration process. This may include information such as table name, table mapping relationship, data type mapping relationship, field mapping relationship, primary key and incremental fields, data filtering conditions and / or business meaning comments.

[0032] Specifically, the aforementioned exchange parameter information can be understood as configuration metadata used to precisely control the runtime behavior of data exchange tasks during the data integration process. This may include information such as source system encoding, extraction conditions, submission batches, number of concurrent threads, and / or timeout thresholds.

[0033] Specifically, the aforementioned file transfer parameter information can be understood as core configuration information used to configure and control the "transfer status notification" behavior during the data integration process. It may include the password information of the file transfer server (e.g., FTP server), notification triggering conditions, notification content template, and / or retry strategy.

[0034] Specifically, the above process information can be stored in the database table data exchange definition table; the above data object information can be stored in the database table data object table; the above exchange parameter information can be stored in the database table data exchange parameter table; and the above file transfer parameter information can be stored in the database table file transfer notification parameter table. The above metadata mapping relationship can include table mapping relationship, field mapping relationship, and data type mapping relationship.

[0035] Optionally, the integrated configuration information includes the metadata mapping relationship between the source system and the target system, as well as the table creation statements of the target system.

[0036] Optionally, the process of determining integration configuration information during the development phase based on user-input configuration parameters includes:

[0037] The metadata mapping relationship between the source system and the target system is determined based on the configuration parameters input by the user; the metadata information of the source system is parsed and obtained; the metadata of the source system is replaced with the metadata of the target system based on the metadata mapping relationship; and the table creation statement of the target system is generated based on the metadata of the target system.

[0038] Specifically, by automatically constructing table creation statements for the target system, the conversion time for table creation statements in heterogeneous databases can be reduced from hours to minutes, which can significantly optimize configuration efficiency.

[0039] Specifically, the metadata of the aforementioned source system includes field names and data types.

[0040] Optionally, the above integrated configuration information includes verification statistics statements, source system database connection information, and field non-empty information.

[0041] Specifically, the aforementioned verification statistics statement information and source system database connection information are used to obtain the data volume of the corresponding data in the source database system based on the source system database connection information after data integration processing, and to compare the data volume of the corresponding data in the source database system with the data volume of the corresponding data in the target system database to verify whether the statistical data integration task has been completed.

[0042] Specifically, the aforementioned non-empty field information is used to automatically compare whether the corresponding field in the target system database is empty after data integration processing. If it is empty, an alarm message indicating that the data integration task has failed will be issued.

[0043] Specifically, the process of generating and exporting the integration configuration installation package based on the integration configuration information in the development phase can include: after determining the integration configuration information in the development phase based on the configuration parameters input by the user, automatically generating and exporting the integration configuration installation package based on the integration configuration information.

[0044] Specifically, the process of generating and exporting the integrated configuration installation package based on the integrated configuration information of the development phase may also include: after determining the integrated configuration information of the development phase based on the configuration parameters input by the user, generating and exporting the integrated configuration installation package based on the integrated configuration information of the development phase in response to the configuration information export request initiated by the user.

[0045] Specifically, the process of integrating test data from the source system in the development environment to the target system in the development environment based on the integration configuration information during the development phase includes:

[0046] Test data can be integrated from the source system of the development environment to the target system of the development environment by carrying data in the message based on the corresponding integration configuration information, or by simply notifying the user in the message based on the corresponding integration configuration information.

[0047] Step 102: In the production phase, the integration configuration installation package is imported and parsed. Based on the parsing results of the integration configuration installation package, the integration configuration information for the production phase is determined. Based on this production phase integration configuration information, business data is integrated from the source system in the production environment to the target system in the production environment. Building upon step 101, this step enables cross-system reuse of integration configuration information during the production phase, achieving automated deployment of data integration tasks in production mode. This effectively avoids the problems of repeated deployments and high reliance on manual intervention caused by environmental differences, improving deployment efficiency and accuracy, thereby increasing the production efficiency and accuracy of the data integration solution. Furthermore, this embodiment of the application, through the reuse of process configurations and installation package management, significantly reduces operational complexity and makes system maintenance more convenient and efficient.

[0048] Specifically, the process of importing and parsing the integrated configuration installation package, and determining the integrated configuration information for the production stage based on the parsing results of the integrated configuration installation package, includes:

[0049] After importing the integrated configuration installation package, a consistency check is performed on the integrated configuration installation package, and after the check passes, the integrated configuration installation package is parsed to obtain the integrated configuration information for the production stage.

[0050] Specifically, consistency checks can also be performed on the contents of the installation package after parsing.

[0051] Specifically, before importing and parsing the integration configuration installation package, users can be prompted to enter and obtain standardized registration information such as the business organization, business domain, source system, Chinese and English descriptions, and storage location of the business data, so as to form a standardized ledger record and ensure that the data entry process is more transparent and traceable.

[0052] Specifically, the process of integrating business data from the source system in the production environment to the target system in the production environment based on the integration configuration information during the production phase includes:

[0053] Based on the corresponding integration configuration information, business data is integrated from the source system in the production environment to the target system in the production environment by carrying data in the message, or based on the corresponding integration configuration information, business data is integrated from the source system in the production environment to the target system in the production environment by simply notifying via the message.

[0054] The cross-system data integration method provided by the embodiments of the present invention is further described below.

[0055] Optional, such as Figure 2 As shown, that is Figure 1 Step 101 may include the following steps:

[0056] Step 1011: In response to the configuration information export request initiated by the user, obtain the export scope and reuse type selected by the user; the export scope is the data integration process corresponding to the reused configuration information.

[0057] Specifically, the data integration process corresponding to the reused configuration information mentioned above can be one data integration process or multiple data integration processes.

[0058] Step 1012: Determine the types of configuration information included in the current reuse configuration information based on the current reuse type.

[0059] Optionally, the reuse type includes full reuse and incremental reuse.

[0060] Optionally, the process of determining the types of configuration information included in the current reuse configuration information based on the current reuse type includes:

[0061] When the reuse type is full reuse, the configuration information types included in the reused configuration information are defined as target system table creation statements, process information, data object information, exchange parameter information, and file transfer parameter information; when the reuse type is incremental reuse, the integrated configuration information types included in the reused configuration information are defined as target system table creation statements, process information, and data object information.

[0062] It is understandable that when a table is added or changed in the source system, the existing data integration method based on the ETL (Extract-Transform-Load) architecture needs to update the configuration in the Excel spreadsheet that maintains the connection parameters and mapping rules and the transmission channel respectively. The manual synchronization error rate is high. The embodiment of this application only needs to perform a configuration update once, so it can avoid human error and improve the sorting accuracy.

[0063] Step 1013: Obtain the table-level reuse configuration information of each data table corresponding to the data integration process based on the configuration information types included in the reuse configuration information.

[0064] Optionally, when the reuse type is full reuse, the process of obtaining the reuse configuration information based on the configuration information type included in the reuse configuration information includes: obtaining the table-level reuse configuration information of each data table corresponding to the data integration process from the database table data exchange definition table; obtaining the table-level data object information of each data table from the database table data object table; obtaining the table-level exchange parameters of each data table from the database table data exchange parameter table; and obtaining the table-level file transfer parameter information of each data table from the file transfer notification parameter table.

[0065] Optionally, when the reuse type is incremental reuse, the process of obtaining the reuse configuration information based on the configuration information type included in the reuse configuration information includes: obtaining the table-level reuse configuration information of each data table corresponding to the data integration process from the database table data exchange definition table; and obtaining the table-level data object information of each data table from the database table data object table.

[0066] It is understandable that when performing incremental reuse, configuration information types for data tables that contain the same information do not need to be configured repeatedly. Therefore, the embodiments of the present invention can further improve the configuration efficiency of data integration configuration in the production stage.

[0067] Step 1014: Generate a digital fingerprint for the current reuse configuration information of each table level, and generate a digital fingerprint for the current reuse configuration information based on the digital fingerprint of the current reuse configuration information of each table level.

[0068] Specifically, the hash value of the reused configuration information at each table level can be calculated using the MD5 algorithm to obtain the corresponding digital fingerprint.

[0069] Specifically, the hash value of the numerical fingerprint of each marker of the current reused configuration information can be calculated based on the MD5 algorithm to obtain the digital fingerprint of the current reused configuration information.

[0070] Step 1015: Generate an integrated configuration installation package based on the reused configuration information and the digital fingerprint of the reused configuration information.

[0071] Optionally, the process of generating an integrated configuration installation package based on the current reuse configuration information and the digital fingerprint of the current reuse configuration information includes: generating an integrated configuration installation package based on the current reuse configuration information, the digital fingerprint of the current reuse configuration information, and the current reuse type.

[0072] Optionally, the process of generating an integrated configuration installation package based on the reused configuration information and the digital fingerprint of the reused configuration information includes: encoding the file transfer server password information, which is included in the file transfer parameter information in the reused configuration information, using the SM4 algorithm to avoid plaintext display.

[0073] The embodiments of the present invention can help improve the reuse accuracy when reusing integrated configuration information in the production stage.

[0074] The cross-system data integration method provided by the embodiments of the present invention is further described below.

[0075] Optional, such as Figure 3 As shown, that is Figure 1 Step 102 may include the following steps:

[0076] Step 1021: In response to the user's request to import configuration information, obtain the integrated configuration installation package.

[0077] Step 1022: Parse and obtain the digital fingerprint of the reused configuration information and the digital fingerprint of the reused configuration information at each table level from the integrated configuration installation package.

[0078] Step 1023: Perform process-level consistency verification on the reused configuration information based on the digital fingerprint of the reused configuration information and the digital fingerprint of the reused configuration information at each table level.

[0079] Step 1024: If the process-level consistency check passes, parse the current reuse configuration information of each table level and determine the integrated configuration information for the production stage based on the current reuse configuration information of each table level.

[0080] Optionally, before determining the integrated configuration information for the production stage based on the reused configuration information at each table level, the reuse type is parsed from the integrated configuration installation package; the existence check results of each type of configuration information are obtained by checking whether the same process configuration information of each type of configuration information in the reused configuration information already exists in the target system of the production environment.

[0081] Optionally, the process of checking whether the corresponding configuration information exists in the same process as the configuration information of each type in the reused configuration information in the target system of the production environment includes: checking whether the corresponding configuration information of the same process exists in the four tables of the target system: database table data exchange definition table, database table data object table, database table data exchange parameter table, and file transfer notification parameter table.

[0082] Optionally, the process of determining the integrated configuration information for the production stage based on the current reuse configuration information of each table level includes: determining the integrated configuration information for the production stage based on the existence check results, the current reuse type, and the current reuse configuration information of each table level.

[0083] Optionally, the integration configuration information for the production stage determined based on the existence check results, the current reuse type, and the current reuse configuration information of each table level includes: when the current reuse type is full reuse, if the corresponding type of same process configuration information does not exist in the four tables of database table data exchange definition table, database table data object table, database table data exchange parameter table, and file transfer notification parameter table, then the integration configuration information for the production stage is obtained by acquiring the configuration information of each type of current reuse configuration information of each table level.

[0084] Optionally, if the same type of configuration information for the same process exists in any of the four tables, the import operation will be terminated, and a message indicating that the configuration information was reused.

[0085] Optionally, the process of determining the integrated configuration information for the production stage based on the existence check results, the current reuse type, and the current reuse configuration information of each table level includes: when the current reuse type is incremental reuse, for each of the four tables, if the current table does not have the corresponding type of same process configuration information, the corresponding type configuration information in the current reuse configuration information of each table level is obtained as the integrated configuration information for the production stage.

[0086] Optionally, the process of obtaining the various types of configuration information from the current reuse configuration information of each table level to obtain the integrated configuration information for the production stage, or obtaining the corresponding types of configuration information from the current reuse configuration information of each table level as the integrated configuration information for the production stage, includes: using an algorithm to decode the file transfer server password included in the file transfer parameter information in the current reuse configuration information to obtain the corresponding file transfer server password information.

[0087] Optionally, if the current table already contains configuration information of the same type for the same process, then the import of the corresponding type of configuration information from the current reuse configuration information of each table level will be terminated.

[0088] Understandably, determining the integration configuration information for the production stage based on the existence check results and the current reuse type can avoid configuration confusion caused by duplicate configurations and improve configuration accuracy.

[0089] Step 1025: If the process-level consistency check fails, parse and obtain the current reuse configuration information of each table level and perform table-level consistency check on the current reuse configuration information of each table level based on the corresponding digital fingerprint.

[0090] Step 1026: For each table-level reused configuration information, if the current table-level reused configuration information fails the consistency check, issue a prompt message to the user indicating that the corresponding table-level configuration consistency check has failed.

[0091] The embodiments of the present invention, through multi-level consistency verification and existence checks, can help improve the accuracy of configuration when reusing configuration information in the production stage.

[0092] The cross-system data integration method provided by the embodiments of the present invention is further described below.

[0093] like Figure 4 As shown, that is Figure 1 Step 101 or 102 includes the following steps;

[0094] Step 401: Monitor and obtain the ready file transfer notification message corresponding to the data to be integrated. The ready file transfer notification message is the file transfer notification message sent by the file transfer server after the ready file compressed file carrying the check code is uploaded to the file transfer server.

[0095] Specifically, the data to be integrated during the development phase can be understood as test data, while the data to be integrated during the production phase can be understood as business data.

[0096] Specifically, the aforementioned ready files can be files uploaded to a shared folder by the corresponding source system through data monitoring, showing the data to be integrated and uninstalled.

[0097] In a specific instance, after the source system completes the upload of the ready file, it sends a ready file notification to the switching system. Upon receiving the ready file notification, the switching system compresses the ready file to obtain a compressed ready file and generates a checksum for the compressed ready file based on the SM3 algorithm.

[0098] Step 402: Perform compliance verification on the ready file transfer notification message, and download the ready file compressed file from the file transfer server after the compliance verification is passed.

[0099] Specifically, it can also check whether the ready file transfer notification message is a duplicate message, and start the execution step 403 after the check passes.

[0100] Step 403: Perform integrity verification on the ready file archive based on the corresponding check code.

[0101] Step 404: After the ready file archive passes the integrity verification, the ready file archive is decompressed to obtain the data to be integrated, and the data to be integrated is loaded into the database of the corresponding target system.

[0102] Specifically, steps 404 and 405 are not executed if the ready file archive fails the integrity check.

[0103] Optionally, the loading results (including the number of successful records, the number of failed records, the reasons for errors, etc.) are recorded to the system log or the results database, and fed back to the corresponding source system through the data exchange system, so that the source system can decide whether to end the lake entry process or resend it if it fails based on the processing status.

[0104] The embodiments of the present invention can improve the efficiency and accuracy of data integration.

[0105] Figure 5 This is a structural diagram of a cross-system data integration apparatus provided in an embodiment of the present invention. This apparatus is suitable for executing the cross-system data integration method provided in an embodiment of the present invention. Figure 5 As shown, the device may specifically include:

[0106] The development phase data integration module 501 is used to determine the integration configuration information based on user-input configuration parameters during the development phase, integrate test data from the source system to the target system in the development environment based on the development phase integration configuration information, and generate and export the integration configuration installation package based on the development phase integration configuration information. This facilitates the cross-system reuse of integration configuration information in the production phase, enabling automated deployment of data integration tasks in production mode.

[0107] Optionally, the integrated configuration information includes the metadata mapping relationship between the source system and the target system, as well as the table creation statements of the target system.

[0108] Optionally, the data integration module 501 in the development phase is specifically used to: determine the metadata mapping relationship between the source system and the target system based on the configuration parameters input by the user; parse and obtain the metadata information of the source system; replace the metadata of the source system with the metadata of the target system based on the metadata mapping relationship; and generate the target system table creation statement based on the metadata of the target system.

[0109] Optionally, the data integration module 501 in the development phase is specifically used to: respond to a user-initiated configuration information export request to obtain the user-selected export scope and reuse type; the export scope is the data integration process corresponding to the reused configuration information; determine the types of configuration information included in the reused configuration information based on the reuse type; obtain the table-level reused configuration information of each data table corresponding to the data integration process based on the types of configuration information included in the reused configuration information to obtain the reused configuration information; generate digital fingerprints for each table-level reused configuration information, and generate digital fingerprints for the reused configuration information based on the digital fingerprints of each table-level reused configuration information; and generate an integrated configuration installation package based on the reused configuration information and the digital fingerprints of the reused configuration information.

[0110] Optionally, the reuse type includes full reuse and incremental reuse.

[0111] Optionally, the data integration module 501 in the development phase is specifically used to determine the types of configuration information included in the reuse configuration information as target system table creation statements, process information, data object information, exchange parameter information, and file transfer parameter information when the reuse type is full reuse; and to determine the types of integrated configuration information included in the reuse configuration information as target system table creation statements, process information, and data object information when the reuse type is incremental reuse.

[0112] Optionally, the aforementioned data integration module 501 in the development phase is specifically used to generate an integrated configuration installation package based on the current reuse configuration information, the digital fingerprint of the current reuse configuration information, and the current reuse type.

[0113] The production-stage data integration module 502 is used to import and parse the integration configuration installation package during the production stage. Based on the parsing results of the integration configuration installation package, it determines the integration configuration information for the production stage and integrates business data from the source system in the production environment to the target system in the production environment based on the integration configuration information for the production stage. Combined with module 501, the integration configuration information can be reused across systems during the production stage, enabling automated deployment of data integration tasks in production mode. This effectively avoids the problems of repeated deployment and high dependence on manual labor caused by environmental differences, improving deployment efficiency and accuracy, thereby improving the production efficiency and accuracy of the data integration solution. Furthermore, this embodiment of the application, through the reuse of process configurations and installation package management, can significantly reduce the complexity of operation and maintenance, and make system maintenance more convenient and efficient.

[0114] Optionally, the aforementioned production stage data integration module 502 can be specifically used to: respond to a user-initiated configuration information import request, obtain an integrated configuration installation package; parse and obtain the digital fingerprint of the current reused configuration information and the digital fingerprint of the current reused configuration information at each table level from the integrated configuration installation package; perform process-level consistency verification on the current reused configuration information based on the digital fingerprint of the current reused configuration information and the digital fingerprint of the current reused configuration information at each table level; if the process-level consistency verification passes, parse the current reused configuration information at each table level and determine the integrated configuration information for the production stage based on the current reused configuration information at each table level; if the process-level consistency verification fails, parse and obtain the current reused configuration information at each table level and perform table-level consistency verification on the current reused configuration information at each table level based on the corresponding digital fingerprint; and for each table level current reused configuration information, issue a corresponding table-level configuration consistency verification failure prompt message to the user when the current table level current reused configuration information fails the table-level consistency verification.

[0115] Optionally, the aforementioned production stage data integration module 502 can be specifically used to parse and obtain the reuse type from the integrated configuration installation package; check whether the same process configuration information of each type of configuration information in the reused configuration information already exists in the target system of the production environment to obtain the existence check results of each type of configuration information.

[0116] Optionally, the aforementioned production stage data integration module 502 can be specifically used to determine the integration configuration information for the production stage based on the existence check results, the current reuse type, and the current reuse configuration information of each table level.

[0117] Optionally, the aforementioned development phase data integration module 501 or production phase data integration module 502 can be specifically used to: monitor and acquire the ready file transfer notification message corresponding to the data to be integrated; the ready file transfer notification message is a file transfer notification message sent by the file transfer server after a ready file compressed file carrying a verification code is uploaded to the file transfer server; perform compliance verification on the ready file transfer notification message, and download the ready file compressed file from the file transfer server after the compliance verification passes; perform integrity verification on the ready file compressed file based on the corresponding verification code; and after the ready file compressed file passes the integrity verification, decompress the ready file compressed file to obtain the data to be integrated, and load the data to be integrated into the database of the corresponding target system.

[0118] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional modules is merely an example. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working process of the functional modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0119] This invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the cross-system data integration method provided in any of the above embodiments.

[0120] This invention also provides a computer-readable medium having a computer program stored thereon, which, when executed by a processor, implements the cross-system data integration method provided in any of the above embodiments.

[0121] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cross-system data integration method as described in any of the embodiments of this invention.

[0122] The following is for reference. Figure 6 It shows a schematic diagram of the structure of a computer system 600 suitable for implementing an electronic device according to embodiments of the present invention. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0123] like Figure 6 As shown, the computer system 600 includes a central processing unit (CPU) 601, which can perform various appropriate actions and processes based on programs stored in read-only memory (ROM) 602 or programs loaded from storage section 608 into random access memory (RAM) 603. The RAM 603 also stores various programs and data required for the operation of the system 600. The CPU 601, ROM 602, and RAM 603 are interconnected via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0124] The following components are connected to I / O interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to I / O interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on drive 610 as needed so that computer programs read from it can be installed into storage section 608 as needed.

[0125] In particular, according to the embodiments disclosed in this invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this invention include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by central processing unit (CPU) 601, it performs the functions defined above in the system of this invention.

[0126] It should be noted that the computer-readable medium shown in this invention can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this invention, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0127] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0128] The modules and / or units described in the embodiments of this invention can be implemented in software or hardware. The described modules and / or units can also be housed in a processor; for example, a processor may be described as including a gradient compression module, a gradient transfer module, a momentum determination module, and a model update module; or, a processor may be described as including an acquisition module, a query module, an identification module, and a determination module. The names of these modules do not necessarily constitute a limitation on the module itself.

[0129] In another aspect, the present invention also provides a computer-readable medium, which may be included in the device described in the above embodiments; or it may exist independently and not assembled into the device. The computer-readable medium carries one or more programs that, when executed by the device, cause the device to: in the development phase, determine integration configuration information based on user-input configuration parameters, integrate test data from the source system of the development environment to the target system of the development environment based on the integration configuration information of the development phase, and generate and export an integration configuration installation package based on the integration configuration information of the development phase; and in the production phase, import and parse the integration configuration installation package, determine integration configuration information for the production phase based on the parsing results of the integration configuration installation package, and integrate business data from the source system of the production environment to the target system of the production environment based on the integration configuration information of the production phase.

[0130] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can occur depending on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A cross-system data integration method, characterized in that, include: During the development phase, integration configuration information is determined based on user-inputted configuration parameters. Based on this integration configuration information, test data is integrated from the source system in the development environment to the target system. An integration configuration installation package is then generated and exported based on the integration configuration information. During the production phase, the integrated configuration installation package is imported and parsed. Based on the parsing results of the integrated configuration installation package, the integrated configuration information for the production phase is determined. Based on the integrated configuration information for the production phase, business data is integrated from the source system in the production environment to the target system in the production environment.

2. The cross-system data integration method according to claim 1, characterized in that, The integrated configuration information includes the metadata mapping relationship between the source system and the target system, as well as the table creation statements of the target system; the process of determining the integrated configuration information during the development phase based on user-input configuration parameters includes: The metadata mapping relationship between the source system and the target system is determined based on the configuration parameters input by the user. Parse and obtain metadata information from the source system; Based on the aforementioned metadata mapping relationship, the metadata of the source system is replaced with the metadata of the target system; and Generate table creation statements for the target system based on the target system's metadata.

3. The cross-system data integration method according to claim 1, characterized in that, The generation of the integration configuration installation package based on the integration configuration information of the development phase includes: In response to a user-initiated configuration information export request, the scope of this export and the reuse type selected by the user are obtained; the scope of this export is the data integration process corresponding to the configuration information reused this time. The types of configuration information included in the current reuse configuration information are determined based on the current reuse type. Based on the configuration information types included in the current reuse configuration information, the table-level current reuse configuration information of each data table corresponding to the data integration process is obtained to obtain the current reuse configuration information; A digital fingerprint is generated for the current reuse configuration information at each table level, and a digital fingerprint for the current reuse configuration information is generated based on the digital fingerprint of the current reuse configuration information at each table level; and The integrated configuration installation package is generated based on the reused configuration information and the digital fingerprint of the reused configuration information.

4. The cross-system data integration method according to claim 3, characterized in that, The reuse type mentioned above includes full reuse and incremental reuse; the configuration information types included in determining the reuse configuration information based on the reuse type include: When the reuse type is full reuse, the configuration information types included in the reuse configuration information are determined to be the target system table creation statement, process information, data object information, exchange parameter information and file transfer parameter information; When the reuse type is incremental reuse, the integrated configuration information type included in the reuse configuration information is determined to be the target system table creation statement, process information and data object information.

5. The cross-system data integration method according to claim 3, characterized in that, The process of importing and parsing the integrated configuration installation package, and determining the integrated configuration information for the production stage based on the parsing results of the integrated configuration installation package, includes: In response to a user-initiated configuration information import request, the integrated configuration installation package is obtained; The digital fingerprints of the reused configuration information and the digital fingerprints of the reused configuration information at each table level are obtained by parsing the integrated configuration installation package. Based on the digital fingerprint of the current reused configuration information and the digital fingerprint of the current reused configuration information of each table level, a process-level consistency check is performed on the current reused configuration information. If the process-level consistency check passes, the current reuse configuration information of each table level of the current reuse configuration information is parsed, and the integrated configuration information of the production stage is determined based on the current reuse configuration information of each table level; If the process-level consistency check fails, the current reuse configuration information of each table level is parsed and obtained, and the current reuse configuration information of each table level is checked for consistency based on the corresponding digital fingerprint. For each table level, if the current table level configuration information fails the table level consistency check, a corresponding table level configuration consistency check failure message will be sent to the user.

6. The cross-system data integration method according to claim 5, characterized in that, The process of generating the integrated configuration installation package based on the reused configuration information and its digital fingerprint includes: The integrated configuration installation package is generated based on the reused configuration information, the digital fingerprint of the reused configuration information, and the reuse type. Before determining the integrated configuration information for the production stage based on the current reuse configuration information of each table level, the method further includes: The reuse type is obtained by parsing the integrated configuration installation package; Check whether the same process configuration information of each type of configuration information in the reused configuration information already exists in the target system of the production environment to obtain the existence check results of each type of configuration information. The process of determining the integrated configuration information for the production stage based on the current reuse configuration information of each table level includes: The integration configuration information for the production stage is determined based on the existence check results, the current reuse type, and the current reuse configuration information for each table level.

7. The cross-system data integration method according to claim 1, characterized in that, The integration of test data from the source system in the development environment to the target system in the development environment based on the integration configuration information of the development phase, or the integration of business data from the source system in the production environment to the target system in the production environment based on the integration configuration information of the production phase, includes: The system monitors and acquires ready file transfer notification messages corresponding to the data to be integrated. These ready file transfer notification messages are file transfer notification messages sent by the file transfer server after a compressed ready file carrying a checksum is uploaded to the file transfer server. The ready file transfer notification message is subjected to compliance verification, and the ready file compressed file is downloaded from the file transfer server after the compliance verification is passed; The integrity of the ready file archive is verified based on the corresponding checksum; and After the ready file archive passes the integrity check, the ready file archive is decompressed to obtain the data to be integrated, and the data to be integrated is loaded into the database of the corresponding target system.

8. A cross-system data integration device, characterized in that, include: The development phase data integration module is used to determine the integration configuration information during the development phase based on user-input configuration parameters, integrate test data from the source system to the target system in the development environment based on the development phase integration configuration information, and generate and export the integration configuration installation package based on the development phase integration configuration information; and The production stage data integration module is used to import and parse the integration configuration installation package during the production stage, determine the integration configuration information for the production stage based on the parsing results of the integration configuration installation package, and integrate business data from the source system in the production environment to the target system in the production environment based on the integration configuration information for the production stage.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the cross-system data integration method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the cross-system data integration method as described in any one of claims 1 to 7.

Images