Data source configuration method, apparatus, device, medium, and program product

Abstract

The present disclosure provides a data source configuration method, which can be applied to the technical field of computers. The data source configuration method is applied to a data migration stage, the data migration stage includes an initial sub-stage and a first transition sub-stage, and the data source configuration method includes: in the first transition sub-stage, obtaining a first data table of a first system; updating a first lake table in a data lake according to the first data table; determining a second lake table matched with the first lake table from the data lake according to anti-corrosion configuration information, wherein the second lake table is correspondingly arranged with a second data table of a second system, and the first data table and the second data table are subjected to data double writing by a first application; taking the content of the updated first lake table as a reference to update the second lake table; and taking the updated second lake table as a data source of a second application, wherein the second application includes a downstream application of the first application. The present disclosure also provides a data source configuration device, equipment, medium and program product.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of computer, in particular, to a data source configuration method and device, electronic equipment, storage medium and program product. BACKGROUND

[0002] With the increasing demand for autonomous controllability in the financial industry, the financial system is gradually transformed from the traditional mainframe system to the distributed platform system, and the data originally stored in the mainframe system also needs to be migrated from the mainframe system to the distributed platform system.

[0003] Figure 1a And Figure 1b An illustrative diagram of a downstream application using a data source in an example is shown, in combination with Figure 1a And Figure 1b As shown in the data migration process, the current method is that the upstream application keeps the data double writing of the mainframe system and the distributed platform system, and the upstream application notifies the downstream application to use the data of the distributed platform system as the data source after the business pilot is passed.

[0004] However, due to the large and complex logic of the business system, the upstream application sometimes needs to be switched back to the mainframe system for running, at this time, only the data of the mainframe system is updated, and the data of the distributed platform system is not updated, thereby causing the data source used by the downstream application to be inaccurate. To deal with this situation, all downstream applications can be notified to switch to use the data of the mainframe system as the data source, but in most scenarios, there are a large number of downstream applications, and it is difficult and costly in a short time to notify all downstream applications to complete the switching of the data source. SUMMARY

[0005] In view of the above problems, the present disclosure provides a data source configuration method, device, electronic equipment, storage medium and program product.

[0006] According to a first aspect of the present disclosure, a data source configuration method is provided, applied to a data migration stage, the data migration stage including a first sub-stage in which a first application is switched from a first system to a second system, and a second sub-stage after the first sub-stage, wherein the data source configuration method comprises:

[0007] In the second sub-stage,

[0008] obtaining a first data table of the first system;

[0009] updating a first lake table in a data lake according to the first data table;

[0010] determine, according to the anti-corruption configuration information, a second lake table matching the first lake table from the data lake, wherein the second lake table is correspondingly arranged with a second data table of the second system, and the first data table and the second data table are double-written by the first application;

[0011] update the second lake table according to the content of the updated first lake table;

[0012] use the updated second lake table as a data source of a second application, and the second application includes a downstream application of the first application.

[0013] According to an embodiment of the present disclosure, the anti-corruption configuration information includes a table-to-table correspondence relationship based on a first primary key record;

[0014] The anti-corruption configuration information includes a table-to-table correspondence relationship based on a first primary key record;

[0015] According to the first primary key, determine the second lake table matching the first lake table from the data lake.

[0016] According to an embodiment of the present disclosure, the updating the first lake table in the data lake according to the first data table includes:

[0017] According to the first data table, perform data writing on a first target partition of the first lake table;

[0018] The updating the second lake table according to the content of the updated first lake table includes:

[0019] determine a second target partition in the second lake table matching the first target partition;

[0020] According to the anti-corruption configuration information, use the content in the first target partition as a reference to rewrite data in the second target partition to update the second lake table.

[0021] According to an embodiment of the present disclosure, the anti-corruption configuration information includes a field-to-field correspondence relationship based on a second primary key record;

[0022] According to the anti-corruption configuration information, use the content in the first target partition as a reference to rewrite data in the second target partition, including:

[0023] clear first target data in the second target partition;

[0024] According to the second primary key, insert second target data in the first target partition into the second target partition to rewrite data in the second target partition.

[0025] According to an embodiment of the present disclosure, the data source configuration method further comprises:

[0026] In the first sub-stage,

[0027] The first data table and the second data table are obtained, and the first data table and the second data table are subjected to data double writing by the first application;

[0028] The first lake table is updated according to the first data table;

[0029] The first lake table after the update is used as a data source of the second application.

[0030] According to an embodiment of the present disclosure, the data migration stage further comprises a second transition sub-stage, and the data source configuration method further comprises:

[0031] In the second transition sub-stage,

[0032] The first application stops writing data into the first data table;

[0033] The second lake table is updated according to the second data table;

[0034] It is judged whether data is appended to the second lake table after the update according to the first lake table;

[0035] The second lake table is used as a data source of the second application.

[0036] According to an embodiment of the present disclosure, the first data table and the second data table perform data interaction with the data lake through a message middleware.

[0037] A second aspect of the present disclosure provides a data source configuration device applied to a data migration stage, the data migration stage comprising a first sub-stage in which a first application is switched from a first system to a second system and a second sub-stage after the first sub-stage, wherein the data source configuration device comprises a first obtaining module, a first updating module, a second updating module and a first processing module.

[0038] In the second sub-stage,

[0039] The first obtaining module is configured to obtain a first data table of the first system;

[0040] The first updating module is configured to update a first lake table in a data lake according to the first data table;

[0041] The first processing module is configured to determine, according to the anti-corrosion configuration information, a second lake table matched with the first lake table from the data lake, wherein the second lake table is correspondingly arranged with a second data table of the second system, and the first data table and the second data table are double-written by the first application.

[0042] The second updating module is configured to update the second lake table with the content of the updated first lake table as a reference.

[0043] The second processing module is configured to use the updated second lake table as a data source of a second application, and the second application includes a downstream application of the first application.

[0044] A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors perform the data source configuration method described above.

[0045] A fourth aspect of the present disclosure further provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the data source configuration method described above.

[0046] A fifth aspect of the present disclosure further provides a computer program product comprising a computer program that, when executed by a processor, implements the data source configuration method described above.

[0047] The one or more embodiments described above have the following advantages or benefits:

[0048] In the embodiments of the present disclosure, by correcting the update result of the second lake table with the first lake table as a reference, the consistency of the second lake table and the first lake table data can be ensured, that is, a data anti-corrosion step is introduced into the update process of the second lake table. In this way, even if the second system data (specifically, the second data table) has no update due to the first application switching back to the first system, the second lake table can be updated based on the first lake table through the data anti-corrosion step, so that the second application can always maintain the second lake table as a data source without switching back, ensuring that the data used by the second application is stable and reliable. BRIEF DESCRIPTION OF DRAWINGS

[0049] The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments of the present disclosure, taken in conjunction with the accompanying drawings, in which:

[0050] Figure 1a and Figure 1b An example of a schematic diagram of a downstream application using a data source is schematically shown;

[0051] Figure 2 An application scenario diagram of the data source configuration method, apparatus, electronic device, storage medium and program product according to an embodiment of the present disclosure is schematically shown;

[0052] Figure 3a One of the flowcharts of the data source configuration method according to an embodiment of the present disclosure is schematically shown;

[0053] Figure 3b One of the schematic diagrams of the data lake scenario according to an embodiment of the present disclosure is schematically shown;

[0054] Figure 3c The second one of the schematic diagrams of the data lake scenario according to an embodiment of the present disclosure is schematically shown;

[0055] Figure 4 The flowchart of updating the second lake table according to an embodiment of the present disclosure is schematically shown;

[0056] Figure 5 The flowchart of data rewriting according to an embodiment of the present disclosure is schematically shown;

[0057] Figure 6a The second one of the flowcharts of the data source configuration method according to an embodiment of the present disclosure is schematically shown;

[0058] Figure 6b The third one of the schematic diagrams of the data lake scenario according to an embodiment of the present disclosure is schematically shown;

[0059] Figure 7a The third one of the flowcharts of the data source configuration method according to an embodiment of the present disclosure is schematically shown;

[0060] Figure 7b The fourth one of the schematic diagrams of the data lake scenario according to an embodiment of the present disclosure is schematically shown;

[0061] Figure 8 The structural block diagram of the data source configuration apparatus according to an embodiment of the present disclosure is schematically shown;

[0062] Figure 9 The block diagram of the electronic device suitable for implementing the data source configuration method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0063] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood, however, that the description is merely exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description of the embodiments, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. However, it would be apparent to one skilled in the art that the embodiments can be practiced without these specific details. In other instances, well-known structures and functions have been described in detail in order to avoid obscuring the concepts of the present disclosure.

[0064] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the present disclosure. As used herein, the term "includes" and tautological expressions thereof, such as "including," "includes," "include," "contains," "containing," and so on, mean the stated features, steps, operations and / or components are present, but do not preclude the presence or addition of one or more other features, steps, operations, components, and / or groups thereof.

[0065] All terms used herein, including technical and scientific terms, have the same meanings as those generally understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having meanings consistent with the context of the present specification, and should not be interpreted in an idealized or excessively formal manner.

[0066] In the case of using expressions similar to "at least one of A, B, and C, etc.", it should generally be interpreted to include at least one of the items enumerated, but not limited to the items enumerated (e.g., "a system having at least one of A, B, and C" should include a system having A alone, a system having B alone, a system having C alone, a system having A and B together, a system having A and C together, a system having B and C together, and / or a system having A, B, and C together, etc.).

[0067] It should be noted that the data source configuration method, apparatus, electronic device, storage medium, and program product provided by the embodiments of the present disclosure relate to the technical field of computers. The data source configuration method, apparatus, electronic device, storage medium, and program product provided by the embodiments of the present disclosure can be applied in the financial field or any field other than the financial field, for example, the data source configuration method, apparatus, electronic device, storage medium, and program product provided by the embodiments of the present disclosure can be applied in the data migration business in the financial field. The application field of the data source configuration method, apparatus, electronic device, storage medium, and program product is not limited by the present disclosure.

[0068] In the technical solutions of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure, and application of user personal information comply with relevant laws and regulations, necessary security measures are taken, and do not violate public order and good customs.

[0069] Embodiments of the present disclosure provide a data source configuration method, applied to a data migration stage, the data migration stage including a first sub-stage in which a first application is switched from a first system to a second system, and a second sub-stage following the first sub-stage, wherein the data source configuration method includes the following steps performed in the second sub-stage:

[0070] obtaining a first data table of the first system; updating a first lake table in the data lake according to the first data table; determining a second lake table matching the first lake table from the data lake according to the data preservation configuration information, wherein the second lake table is correspondingly set with a second data table of the second system, and the first data table and the second data table are subjected to data double writing by the first application; updating the second lake table with the content of the updated first lake table as a reference; and taking the updated second lake table as a data source of a second application, the second application including a downstream application of the first application.

[0071] In the embodiments of the present disclosure, the updating result of the second lake table is corrected with the first lake table as a reference, which can ensure the consistency of the second lake table and the first lake table data, that is, a data preservation step is introduced into the updating process of the second lake table. In this way, even if the data (specifically, the second data table) of the second system is not updated due to the first application switching back to the first system, the second lake table can be updated based on the first lake table through the data preservation step, so that the second application can always maintain the second lake table as a data source without switching back, ensuring that the data used by the second application is stable and reliable.

[0072] Figure 2 An application scenario diagram of the data source configuration method, apparatus, electronic device, storage medium and program product according to the embodiments of the present disclosure is schematically shown.

[0073] As shown in Figure 2 application scenario 200 according to the embodiments can include terminal devices 201, 202, 203, a network 204 and a server 205. The network 204 is used as a medium to provide a communication link between the terminal devices 201, 202, 203 and the server 205. The network 204 can include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0074] A user can use the terminal devices 201, 202, 203 to interact with the server 205 through the network 204 to receive or send messages, etc. Various communication client applications can be installed on the terminal devices 201, 202, 203, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only as examples).

[0075] The terminal devices 201, 202, and 203 can be various electronic devices with display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0076] The server 205 can be a server providing various services, such as a background management server supporting websites browsed by users using the terminal devices 201, 202, and 203 (only as an example). The background management server can perform analysis and the like on received user requests and the like, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests, and the like) to the terminal devices.

[0077] It should be noted that the data source configuration method provided by the embodiments of the present disclosure can generally be executed by the server 205. Accordingly, the data source configuration apparatus provided by the embodiments of the present disclosure can generally be arranged in the server 205. The data source configuration method provided by the embodiments of the present disclosure can also be executed by a server or a server cluster different from the server 205 and capable of communicating with the terminal devices 201, 202, and 203 and / or the server 205. Accordingly, the data source configuration apparatus provided by the embodiments of the present disclosure can also be arranged in a server or a server cluster different from the server 205 and capable of communicating with the terminal devices 201, 202, and 203 and / or the server 205.

[0078] It should be understood that Figure 2 The number of terminal devices, networks, and servers in the scenario described in the foregoing embodiments is only illustrative. According to implementation needs, there can be any number of terminal devices, networks, and servers.

[0079] The data source configuration method of the embodiments of the present disclosure will be described in detail below based on the scenario described in the foregoing embodiments. Figure 2 Figures 3a to 7b The data source configuration method of the embodiments of the present disclosure will be described in detail below based on the scenario described in the foregoing embodiments.

[0080] The embodiments of the present disclosure provide a data source configuration method, which is applied to a data migration stage. The data migration stage includes a first sub-stage in which a first application is switched from a first system to a second system, and a second sub-stage after the first sub-stage.

[0081] In the embodiments of the present disclosure, the data migration stage can include a process of migrating business data originally stored in the first system to the second system, where the first system refers to an old system, and the second system can refer to a new system. For example, the first system can include a large machine system, and the second system can include a distributed platform system.

[0082] ​The mainframe system can refer to a mainframe system, also known as a large mainframe, which is used to distinguish from microcomputers and personal computers. At present, the mainstream is a large mainframe produced by IBM Enterprise, which has high reliability, high availability, high serviceability and strong I / O processing capability, and is commonly used in commercial fields such as banking and communication industries.

[0083] The distributed platform system includes a cluster composed of multiple machines, which collectively complete data calculation, data storage and other functions originally completed by a single machine, and has the characteristics of high concurrency, high availability, high performance, easy expansion, flexibility and the like.

[0084] The core system design and architecture of the traditional financial industry are in the centralized architecture of the IBM mainframe era. With the development of the times, online transactions based on the Internet have become the mainstream of financial transactions. The traditional architecture has been difficult to support the rapidly developing market demand, and it is also unable to quickly respond to the market and flexibly launch new financial products. At the same time, in order to reduce the dependence on some key technologies, the transformation of independent controllability is imminent. Based on the above factors, the financial industry needs to transform to a new generation of distributed platform system, and data migration is the most important in this transformation process.

[0085] In the embodiment of the present disclosure, in the first sub-stage, the upstream application starts to switch from the first system to the second system. When the upstream application switches to the second system and is verified, it will enter the second sub-stage.

[0086] Figure 3a One of the flowcharts of the data source configuration method according to the embodiment of the present disclosure is schematically shown, Figure 3b One of the schematic diagrams of the data lake scenario according to the embodiment of the present disclosure is schematically shown, Figure 3c Two of the schematic diagrams of the data lake scenario according to the embodiment of the present disclosure are schematically shown, Figures 3a to 3c As shown, the data source configuration method of the embodiment includes steps S210-S250 performed in the second sub-stage.

[0087] It should be noted that although Figure 3a Each step in the figure is displayed in sequence according to the arrow, but these steps are not necessarily executed in the order indicated by the arrow. Unless otherwise stated herein, the execution of these steps has no strict order restriction, and they can be executed in other orders. Moreover, at least part of the steps in the figure can include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order is not necessarily sequential, but can be executed in rotation or alternation with at least part of other steps or sub-steps or stages of other steps.

[0088] In step S210, a first data table of a first system is acquired.

[0089] In an embodiment of the present disclosure, the first application can be referred to as an upstream application. In the second sub-stage, the first application can write data to the first data table, and in addition, the first application can also write data to a second data table of a second system. In other words, in the second sub-stage, the first application can perform double writing of data, that is, in a transaction occurring in the second sub-stage, the first application writes data to the first data table of the first system and the second data table of the second system at the same time. Of course, in the first sub-stage, the first application can also perform double writing of data, which will be described in detail below, and therefore, will not be described here.

[0090] In step S220, according to the first data table, a first lake table in the data lake is updated.

[0091] The data lake is an information storage method that can face multiple data sources. It is a gathering of raw data, and does not perform conversion, cleaning, processing and other processing on the data, and preserves the original characteristics of the data. It realizes the integration of the source layer data, and comprehensively and timely gathers the business data and external data in the system. The data storage format in the data lake is consistent with the data source, and provides data support for the data middle platform construction. For example, in an embodiment of the present disclosure, in the second sub-stage, the data lake provides the required data for the second application. The second application can be referred to as a downstream application of the first application.

[0092] In an embodiment of the present disclosure, the upstream application realizes the corresponding function by calling the downstream application. For example, for application A1 and application A2, application A1 calls application A2, and application A1 is the upstream application, and application A2 is the downstream application.

[0093] In an embodiment of the present disclosure, data can be entered into the lake according to a preset period. The data entering the lake in the embodiment of the present disclosure can refer to storing the data that needs to be entered into the lake into the first lake table and the second lake table, so as to update the first lake table and the second lake table. The preset period can be determined according to actual needs. For example, the preset period can be set to “every day”. For example, data can be entered into the lake in a relatively idle period of the day.

[0094] Optionally, for the first lake table, the first data table can be entered into the lake to update the first lake table. Wherein, the source entry into the lake can refer to storing the data in the first data table without modification into the first lake table.

[0095] For the second lake table, in the traditional scheme, the second data table can be pasted into the lake to update the second lake table, but as described above, when the first application switches back to the first system, the data of the second system (specifically, the second data table) is no longer updated, at this time, the second lake table cannot be normally updated, which will cause the data source of the second application to have an error.

[0096] Therefore, in the embodiments of the present disclosure, the second lake table is updated through steps S230 and S240. Through steps S230 and S240, a data preservation step can be introduced on the basis of keeping the original update step unchanged, so as to ensure that the data of the second lake table is accurate and effective.

[0097] In step S230, the second lake table matched with the first lake table is determined from the data lake according to the preservation configuration information, wherein the second lake table is correspondingly set with the second data table of the second system, and the first data table and the second data table are written by the first application.

[0098] In the embodiments of the present disclosure, the first lake table can be one-to-one set with the first data table, and the second lake table can be one-to-one set with the second data table. For the same transaction, the first system can record through one first data table, and the second system can record through multiple second data tables (vice versa). At this time, the first application writes the transaction into one first data table and multiple second data tables when performing data double writing. In the first transition phase, when data is entered into the lake, the first data table can write data into the corresponding first lake table to update it, and for the second lake table, the second data table can first write data into the corresponding second lake table in the original manner, and then perform step S240 to preserve the second lake table to complete the update.

[0099] In the embodiments of the present disclosure, in the preservation configuration information, the first data table, the second data table, the first lake table and the second lake table used to record the same transaction are associated, and these associated tables can be called “matched”.

[0100] In step S240, the second lake table is updated with the content of the updated first lake table as a reference.

[0101] In the embodiments of the present disclosure, since the data of the first lake table is from the first data table of the first system, the data reliability is higher, and therefore, the first lake table is used as the reference for updating the second lake table. Alternatively, the original update step can be kept without modification, that is, the second lake table is updated based on the second data table first. Then, the data anti-corrosion step is introduced, that is, the second lake table is corrected based on the content of the first lake table, for example, the data of the updated part of the second lake table is rewritten, so as to complete the update of the second lake table.

[0102] In the embodiments of the present disclosure, the data rewriting can be performed according to the following principles:

[0103] 1) For data D1, if the first lake table has data D1 and the second lake table does not have data D1, the data D1 is loaded into the second target partition.

[0104] 2) For data D2, if the first lake table does not have data D2 and the second lake table has data D2, the data D2 is discarded.

[0105] 3) For data D3, if the first lake table and the second lake table both have data D3, the data D3 in the first lake table is loaded.

[0106] In step S250, the updated second lake table is used as the data source of the second application, that is, in the second sub-stage, the second lake table provides data support for the second application.

[0107] In the embodiments of the present disclosure, the update of the second lake table based on the first lake table in the second sub-stage can ensure that the data of the second lake table is consistent with the data of the first lake table, that is, the data anti-corrosion step is introduced for the second lake table. In this way, even if the data of the second system is not updated (specifically, the second data table is not updated) due to the first application switching back to the first system, the second lake table can be updated based on the first lake table through the data anti-corrosion step, so that the second application can always use the second lake table as the data source without switching back, and the data used by the second application is stable and reliable.

[0108] The data source configuration method of the embodiments of the present disclosure will be further described below. Figures 3a to 7b The data source configuration method of the embodiments of the present disclosure will be further described below.

[0109] In some specific embodiments, the anti-corrosion configuration information includes a table-to-table correspondence relationship based on a first primary key, and the first primary key includes a table name of the first data table and a table name of the second data table.

[0110] In some specific embodiments, step S230 includes step S231.

[0111] In step S231, the second lake table matched with the first lake table is determined from the data lake according to the first primary key.

[0112] For example, the inter-table correspondence relationship is shown in Table 1, in which there are three types of inter-table correspondence relationship.

[0113] The first inter-table correspondence relationship is a one-to-one correspondence relationship between the first data table and the second data table, that is, one first data table corresponds to one second data table, and the first application writes one transaction into the first data table and the second data table corresponding thereto at the same time when data is written twice. Correspondingly, the first lake table corresponding to the first data table and the second lake table corresponding to the second data table are also set one-to-one, and when data is entered into the lake, the data of the first data table is written into the first lake table corresponding thereto, and the data of the second data table is written into the second lake table corresponding thereto.

[0114] It should be noted that for the second lake table, after the data of the second data table is written into the second lake table corresponding thereto in the second sub-stage, the data rewriting can be performed based on the content of the first lake table through the data preservation step in the foregoing, so as to ensure that the second lake table is consistent with the first lake table, thereby completing the update. After it is determined that there is no possibility of the first application returning to the first system (for example, after the second sub-stage is completed, the connection between the first application and the first system is cut off), the data preservation step can be removed, and after the data of the second data table is written into the second lake table corresponding thereto, the completion of the update is directly confirmed.

[0115] As shown in Table 1, in the first inter-table correspondence relationship, the first data table a and the second data table A are set one-to-one, and the first lake table a' and the second lake table A' are also set one-to-one. The first application writes one transaction into the first data table a and the second data table A at the same time when data is written twice, and exemplarily, the first data table a and the second data table A satisfy: a=A. When data is entered into the lake, the data of the first data table a is written into the first lake table a', and the data of the second data table A is written into the second lake table A', and then the first lake table a' and the second lake table A' are written into the same transaction, and exemplarily, the first lake table a' and the second lake table A' satisfy: a'=A'. Therefore, the first data table a, the second data table A, the first lake table a', and the second lake table A' are divided together, the first primary key is the table name of the first data table and the second data table, and then the second lake table used for the same transaction as the first lake table a' record can be found according to the inter-table correspondence relationship, that is, the second lake table A' is found.

[0116] The second inter-table correspondence relationship is a one-to-many correspondence relationship between the first data table and the second data table, that is, one first data table corresponds to multiple second data tables, and correspondingly, the first lake table corresponding to the first data table and the second lake table (multiple) corresponding to the second data table (multiple) are also set in a "one-to-many" manner. As shown in Table 1, one first data table b is set to correspond to second data tables B1, B2 and B3, and one first lake table b' is set to correspond to second lake tables B1', B2' and B3'. When the first application writes data, a transaction is written into the first data table b, and at the same time, the transaction is split into multiple parts and written into the second data table B1, the second data table B2 and the second data table B3. Exemplarily, the first data table b and the multiple second data tables satisfy: b = B1 + B2 + B3. When the data is written into the lake, the data of the first data table b is written into the first lake table b', the data of the second data table B1 is written into the second lake table B1', the data of the second data table B2 is written into the second lake table B2', and the data of the second data table B3 is written into the second lake table B3'. Further, the first lake table b' and the multiple second lake tables (that is, the second lake table B1', the second lake table B2' and the second lake table B3') are written into the same transaction. Exemplarily, the first lake table b' and the multiple second lake tables satisfy: b' = B1' + B2' + B3'. Therefore, the first data table b, the second data table B1, the second data table B2, the second data table B3, the first lake table b', the second lake table B1', the second lake table B2' and the second lake table B3' are divided together, the first primary key is the table name of the first data table and the second data table, and further, the second lake table for recording the same transaction as the first lake table b' can be found according to the inter-table correspondence relationship, that is, the second lake table B1', the second lake table B2' and the second lake table B3' are found.

[0117] The third inter-table correspondence relationship is a many-to-one correspondence relationship between the first data table and the second data table, that is, a plurality of first data tables correspond to one second data table, and correspondingly, the first lake table (plurality) corresponding to the first data table (plurality) and the second lake table corresponding to the second data table are also set in a many-to-one manner. As in Table 1, the first data table c1, the first data table c2 and the first data table c3 are set to correspond to one second data table C, and the plurality of first lake tables c1', the first lake table c2' and the first lake table c3' are set to correspond to one second lake table C'. When the first application writes data into the second data table C, the transaction is split into multiple parts and written into the first data table c1, the first data table c2 and the first data table c3, and the plurality of first data tables and the second data table C satisfy: c1+c2+c3=C. When the data is written into the lake, the data of the first data table c1 is written into the first lake table c1', the data of the first data table c2 is written into the first lake table c2', the data of the first data table c3 is written into the first lake table c3', and the data of the second data table C is written into the second lake table C'. In turn, the plurality of first lake tables (i.e., the first lake table c1', the first lake table c2' and the first lake table c3') and the second lake table C' are written into the same transaction, and the plurality of first lake tables and the second lake table C' satisfy: c1'+c2'+c3'=C'. Therefore, the first data table c1, the first data table c2, the first data table c3, the second data table C, the first lake table c1', the first lake table c2', the first lake table c3' and the second lake table C' are divided together, the first primary key is the table name of the first data table and the second data table, and then the second lake table for recording the same transaction as the first lake table c1', the first lake table c2' and the first lake table c3' can be found according to the inter-table correspondence relationship, that is, the second lake table C' is found.

[0118] Table 1

[0119]

[0120] In some embodiments, step S220 includes step S221.

[0121] In step S221, data is written into the first target partition of the first lake table according to the first data table.

[0122] In the embodiments of the present disclosure, the first lake table can be divided into multiple partitions, and the data in the first data table that needs to be imported into the lake can be written into the partition in the first lake table that matches the "this time" import operation. For example, the import operation can be performed in a "day" cycle, and on the nth day, the data of the (n-1)th day in the first data table can be imported into the lake. At this time, the data of the (n-1)th day in the first data table can be written into the partition Xn-1 corresponding to the (n-1)th day in the first lake table, and the partition Xn-1 is the first target partition.

[0123] Figure 4 The flowchart of updating the second lake table according to the embodiments of the present disclosure is schematically shown as Figure 4 As shown, in some embodiments, step S240 includes step S241 and step S242.

[0124] In step S241, the second target partition in the second lake table that matches the first target partition is determined.

[0125] Similar to the first lake table, the second lake table can also be divided into multiple partitions, and the data in the second data table that needs to be imported into the lake can be written into the partition in the second lake table that matches the "this time" import operation. For example, the import operation can be performed in a "day" cycle, and on the nth day, the data of the (n-1)th day in the second data table can be imported into the lake. At this time, the data of the (n-1)th day in the second data table can be written into the partition Yn-1 corresponding to the (n-1)th day in the second lake table, and the partition Yn-1 is the second target partition.

[0126] In step S242, according to the anti-corrosion configuration information, the content in the first target partition is taken as a reference to rewrite the data in the second target partition, so as to update the second lake table.

[0127] In some embodiments, the anti-corrosion configuration information includes a field-to-field correspondence relationship based on a second primary key record, and the second primary key includes the table name of the first data table and the field of the first data table.

[0128] Figure 5 The flowchart of data rewriting according to the embodiments of the present disclosure is schematically shown as Figure 5 As shown, step S242 includes step S2421 and step S2422.

[0129] In step S2421, the first target data in the second target partition is cleared.

[0130] In step S2422, according to the second primary key, the second target data in the first target partition is inserted into the second target partition to rewrite the data in the second target partition.

[0131] Taking the inter-table correspondence relationship as an example, the inter-field correspondence relationship is shown in Table 2. The fields of the first data table include region number, point number, card number, customer name, certificate type, and certificate number, etc. After the architecture transformation reconstruction, in the second system, the region number and point number fields are constructed in the second data table B1, the card number field is constructed in the second data table B2, and the customer name, certificate type, and certificate number fields are constructed in the second data table B3.

[0132] Table 2

[0133]

[0134] In the embodiment of the present disclosure, the first target data in the second target partition can be cleared by using the SQL function, and the second target data can be inserted into the second target partition, and the like. The first target data may, for example, be the assignment corresponding to each field in Table 2 in the second target partition, and the second target data may, for example, be the assignment corresponding to each field in Table 2 in the first target partition. At this time, the data in the second lake table is the data in the first data table. Even if the second data table is not updated, since the data preservation step is performed, the second lake table will not be affected, and the stable and reliable use of data by the second application is ensured.

[0135] Figure 6a Fig. 2 schematically shows a flowchart of a data source configuration method according to an embodiment of the present disclosure, Figure 6b Fig. 3 schematically shows a schematic diagram of a data lake scenario according to an embodiment of the present disclosure, Figure 6a and Figure 6b In some specific embodiments, the data source configuration method further includes steps S310 to S330 performed in the first sub-stage.

[0136] In step S310, the first data table and the second data table are obtained, and the first data table and the second data table are written by the first application.

[0137] In the embodiment of the present disclosure, in the first sub-stage, the first application also adopts the data double writing of the first system and the second system of the distributed platform, that is, the same transaction is written into the first data table of the first system and the second data table of the second system at the same time.

[0138] In step S320, the first lake table is updated according to the first data table.

[0139] In step S330, the updated first lake table is used as the data source of the second application.

[0140] In the embodiment of the present disclosure, since in the first sub-stage, the first application starts to switch from the first system to the second system, in order to ensure the accuracy of the data, in this sub-stage, the first lake table is used as the data source of the second application.

[0141] In some specific embodiments, the data source configuration method further includes steps S340 to S370 performed in the first sub-stage.

[0142] In step S340, according to the second data table, the second lake table is updated and backed up to obtain a backup table.

[0143] In step S350, the second lake table is updated based on the content of the updated first lake table, that is, the data rewriting of the second target partition of the second lake table is performed according to the data preservation step described above.

[0144] In step S360, when the business verification of the first sub-stage passes, the data consistency verification is performed according to the updated first lake table, the preserved second lake table and the backup table.

[0145] In step S370, when the data consistency verification passes, the second sub-stage is entered, otherwise, a corresponding prompt can be generated to inform the person concerned to handle.

[0146] In the embodiment of the present disclosure, the data consistency verification can include data quantity consistency verification and field consistency verification, etc. After the verification passes, the second application can be informed to switch the data source from the first lake table to the second lake table to enter the second sub-stage.

[0147] Figure 7a Fig. 3 schematically shows a flowchart of a data source configuration method according to an embodiment of the present disclosure, Figure 7b Fig. 4 schematically shows a schematic diagram of a data entry lake scenario according to an embodiment of the present disclosure, in combination with Figure 7a and Figure 7b In some specific embodiments, the data migration stage further includes a second transition sub-stage, and the data source configuration method further includes steps S410 to S440 performed in the second transition sub-stage.

[0148] In step S410, the first application stops writing data into the first data table.

[0149] In the embodiment of the present disclosure, the first application is disconnected from the first system, so as to stop writing data into the first data table, and the first application only writes data into the second data table of the second system.

[0150] In step S420, the second lake table is updated according to the second data table.

[0151] At step S430, it is judged whether to append data to the updated second lake table according to the first lake table.

[0152] In the embodiments of the present disclosure, data can be written into the second target partition of the second lake table according to the second data table first, at this time, the first data table is theoretically no longer updated, but in order to ensure data accuracy, the first lake table can be monitored, and it is judged whether to append data to the updated second lake table according to the monitoring result.

[0153] For example, whether to append data can be determined according to the following principles.

[0154] 1) For data E1, if the first lake table has data E1 and the second lake table does not have data E1, data E1 is appended to the second lake table.

[0155] 2) For data E2, if the first lake table does not have data E2 and the second lake table has data E2, data E2 is retained.

[0156] 3) For data E3, if the first lake table and the second lake table both have data E3, data E3 is retained or data E3 in the first lake table is loaded.

[0157] That is, once the first lake table has data that does not exist in the second lake table, the part of data can be written into the second lake table to perform data appending.

[0158] In the embodiments of the present disclosure, SQL functions can be used to implement data appending, for example, the second target partition of the updated second lake table obtained in step S420 can be emptied first, then data that only appears in the second data table is obtained by left joining the primary key, and is written into the second target partition. Then, all data in the first target partition of the first lake table is obtained, and data that does not appear in the second target partition is inserted into the second target partition to complete data appending.

[0159] At step S440, the second lake table is used as a data source of the second application. At this time, the second lake table is full-amount data, so that the data of the second lake table is stable and reliable.

[0160] In some specific embodiments, the first data table and the second data table perform data interaction with the data lake through a message middleware. For example, the middleware can include a distributed publish / subscribe message system Kafka, the first data table and the second data table can push data to Kafka in a data replication manner, and the data lake completes data entry into the lake after subscribing and consuming data from Kafka.

[0161] In the first and second sub-stages of the data migration stage, the first application exists the situation of switching back to the first system, for the first sub-stage, the second application uses the first lake table as the data source, therefore, the switching back of the first application does not affect the data source of the second application. For the second sub-stage, the second application switches to use the second lake table as the data source, since the second lake table has undergone the data preservation step, it is always consistent with the data of the first lake table, therefore, the switching back of the first application also does not affect the data source of the second application. Therefore, in the entire data migration stage, the problem of the data source of the second application being wrong caused by the first application switching back to the first system is overcome, and the reliability and stability of the data source of the second application are improved.

[0162] In the second transition sub-stage, the first application is disconnected with the first system, and the first data table is theoretically no longer updated, therefore, the second lake table is updated according to the second data table, but in order to ensure the accuracy of the data, the first lake table can be monitored, and data can be added to the second lake table according to the first lake table, so that the data in the second lake table is full data, and further, the accuracy of the data source of the second application is ensured in the second transition sub-stage, so that the data source of the second application is always stable and reliable in the entire data migration stage.

[0163] Therefore, in the entire data migration stage, the second application only needs to perform the switching of the data source once, and does not need to care about the emergency switching back of the first application and the like during the period, thereby effectively reducing the influence of the first application running unstably and the like on the second application.

[0164] Based on the above data source configuration method, the present disclosure further provides a data source configuration device, which is applied to a data migration stage, the data migration stage including a first sub-stage in which a first application is switched from a first system to a second system, and a second sub-stage after the first sub-stage. The following will be described in detail in combination with Figure 8 The device is described in detail.

[0165] Figure 8 The structure block diagram of the data source configuration device according to the embodiment of the present disclosure is schematically shown, as Figure 8 The data source configuration device 800 of the embodiment includes a first acquisition module 810, a first update module 820, a first processing module 830, a second update module 840 and a second processing module 850. In the second sub-stage, the first acquisition module 810, the first update module 820, the first processing module 830, the second update module 840 and the second processing module 850 have the following functions:

[0166] The first acquisition module 810 is configured to acquire the first data table of the first system. In an embodiment, the first acquisition module 810 can be configured to perform the step S210 described above, which will not be described here again.

[0167] The first updating module 820 is configured to update the first lake table in the data lake according to the first data table. In an embodiment, the first updating module 820 can be configured to perform the step S220 described above, and details are not repeated here.

[0168] The first processing module 830 is configured to determine, according to the anti-corrosion configuration information, a second lake table matched with the first lake table from the data lake, where the second lake table is set to correspond to a second data table of a second system, and the first data table and the second data table are double-written by the first application. In an embodiment, the first processing module 830 can be configured to perform the step S230 described above, and details are not repeated here.

[0169] The second updating module 840 is configured to update the second lake table with the content of the updated first lake table as a reference. In an embodiment, the second updating module 840 can be configured to perform the step S240 described above, and details are not repeated here.

[0170] The second processing module 850 is configured to use the updated second lake table as a data source of a second application, where the second application includes a downstream application of the first application. In an embodiment, the second processing module 850 can be configured to perform the step S250 described above, and details are not repeated here.

[0171] In the embodiments of the present disclosure, the updating of the second lake table with the first lake table as a reference in the second sub-stage can ensure that the data of the second lake table is consistent with that of the first lake table, that is, the data anti-corrosion step is introduced to the second lake table. In this way, even if the data of the second system is not updated (specifically, the second data table is not updated) due to the first application being switched back to the first system, the second lake table can be updated based on the first lake table through the data anti-corrosion step, so that the second application can always use the second lake table as a data source without switching back, and the data used by the second application is stable and reliable.

[0172] According to an embodiment of the present disclosure, any of the first obtaining module 810, the first updating module 820, the first processing module 830, the second updating module 840 and the second processing module 850 can be combined in one module, or any of them can be split into multiple modules. Alternatively, at least part of the function of one or more of these modules can be combined with at least part of the function of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the first obtaining module 810, the first updating module 820, the first processing module 830, the second updating module 840 and the second processing module 850 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on chip, a system on substrate, a system on package, an application specific integrated circuit (ASIC), or any other reasonable way of hardware or firmware that can be integrated or packaged with a circuit, or any one of software, hardware and firmware or any appropriate combination of any of them. Alternatively, at least one of the first obtaining module 810, the first updating module 820, the first processing module 830, the second updating module 840 and the second processing module 850 can be at least partially implemented as a computer program module that can perform corresponding functions when executed.

[0173] In some embodiments, the anti-corruption configuration information includes a table-to-table correspondence relationship recorded based on the first primary key.

[0174] The first processing module 830 is specifically configured to perform the following steps:

[0175] According to the first primary key, determine a second lake table matching the first lake table from the data lake.

[0176] In some embodiments, the first updating module 820 is specifically configured to perform the following steps:

[0177] According to the first data table, perform data writing on a first target partition of the first lake table.

[0178] The second updating module 840 is specifically configured to perform the following steps:

[0179] Determine a second target partition in the second lake table matching the first target partition.

[0180] According to the anti-corruption configuration information, rewrite the second target partition based on the content in the first target partition to update the second lake table.

[0181] In some embodiments, the anti-corruption configuration information includes a field-to-field correspondence relationship recorded based on the second primary key.

[0182] The second updating module 840 is specifically configured to perform the following steps:

[0183] The first target data in the second target partition is cleared.

[0184] According to the second primary key, the second target data in the first target partition is inserted into the second target partition to rewrite the data of the second target partition.

[0185] In some specific embodiments, the data source configuration apparatus further comprises a third processing module, and in the first sub-stage, the third processing module is configured to perform the following steps:

[0186] The first data table and the second data table are obtained, and the first data table and the second data table are subjected to data double writing by the first application.

[0187] The first lake table is updated according to the first data table.

[0188] The updated first lake table is used as a data source of the second application.

[0189] In some specific embodiments, the data migration stage further comprises a second transition sub-stage, and the data source configuration apparatus further comprises a fourth processing module, and in the second transition sub-stage, the fourth processing module is configured to perform the following steps:

[0190] The first application stops writing data into the first data table.

[0191] The second lake table is updated according to the second data table.

[0192] According to the first lake table, it is determined whether to append data to the updated second lake table.

[0193] The second lake table is used as a data source of the second application.

[0194] In some specific embodiments, the first data table and the second data table perform data interaction with the data lake through a message middleware.

[0195] Figure 9 A block diagram of an electronic device suitable for implementing the data source configuration method according to an embodiment of the present disclosure is schematically shown.

[0196] As Figure 9As shown, the electronic device 900 according to the embodiments of the present disclosure includes a processor 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded into a random access memory (RAM) 903 from a storage section 908. The processor 901 can include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor, and / or a related chipset, and / or a dedicated microprocessor (e.g., an application specific integrated circuit (ASIC)), and the like. The processor 901 can also include an on-board memory for cache use. The processor 901 can include a single processing unit or a plurality of processing units for performing different actions of the method processes according to the embodiments of the present disclosure.

[0197] In the RAM 903, various programs and data required for the operation of the electronic device 900 are stored. The processor 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. The processor 901 performs various operations of the method processes according to the embodiments of the present disclosure by executing the programs in the ROM 902 and / or the RAM 903. Note that the programs can also be stored in one or more memories other than the ROM 902 and the RAM 903. The processor 901 can also perform various operations of the method processes according to the embodiments of the present disclosure by executing the programs stored in the one or more memories.

[0198] According to the embodiments of the present disclosure, the electronic device 900 can further include an input / output (I / O) interface 905, which is also connected to the bus 904. The electronic device 900 can further include one or more of the following components connected to the I / O interface 905: an input section 906 including a keyboard, a mouse, etc.; an output section 907 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 908 including a hard disk, etc.; and a communication section 909 including a network interface card such as a LAN card, a modem, etc. The communication section 909 performs communication processing via a network such as the Internet. A drive 910 is also connected to the I / O interface 905 as necessary. A removable recording medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 910 as necessary, so that a computer program read therefrom is installed into the storage section 908 as necessary.

[0199] The present disclosure also provides a computer readable storage medium, which can be included in the device / apparatus / system described in the above embodiments, or can exist separately without being assembled into the device / apparatus / system. The above computer readable storage medium carries one or more programs, which, when executed, implement the data source configuration method according to the embodiments of the present disclosure.

[0200] According to an embodiment of the present disclosure, the computer readable storage medium can be a nonvolatile computer readable storage medium, for example, can include but not limited to: a portable computer diskette, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), a portable compact disc read only memory (CD-ROM), an optical storage device, a magnetic storage device, or any appropriate combination thereof. In the present disclosure, the computer readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, the computer readable storage medium can include one or more memories such as the ROM 902 and / or the RAM 903 described above and / or one or more memory other than the ROM 902 and the RAM 903.

[0201] Embodiments of the present disclosure also include a computer program product, which includes a computer program containing program codes for executing the methods shown in the flowcharts. When the computer program product is run in a computer system, the program codes are used to make the computer system implement the data source configuration method provided by the embodiments of the present disclosure.

[0202] The above-described functions defined in the system / apparatus / module / unit of the embodiments of the present disclosure are performed when the computer program is executed by the processor 901. According to an embodiment of the present disclosure, the above-described system, apparatus, module, unit, etc. can be implemented by computer program modules.

[0203] In one embodiment, the computer program can rely on a tangible storage medium such as an optical storage device, a magnetic storage device, etc. In another embodiment, the computer program can also be transmitted, distributed, and downloaded in the form of a signal on a network medium and be downloaded and installed through the communication part 909 and / or installed from the detachable medium 911. The program codes contained in the computer program can be transmitted by any appropriate network medium, including but not limited to wireless, wired, etc., or any appropriate combination thereof.

[0204] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 909 and / or installed from the detachable medium 911. When the computer program is executed by the processor 901, the above-described functions defined in the system of the embodiments of the present disclosure are performed. According to an embodiment of the present disclosure, the above-described system, apparatus, device, module, unit, etc. can be implemented by computer program modules.

[0205] According to embodiments of the present disclosure, program code of the computer program for performing the methods provided by the embodiments of the present disclosure can be written in any combination of one or more programming languages, and can be implemented in a computer program product. Specifically, the computer program can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. The programming language includes, but is not limited to, Java, C++, python, “C” language, or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, and partly on a remote computing device, or entirely on the remote computing device or server. In the latter scenario, the remote computing device can be connected to the user's computing device through any kind of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computing device, such as through the Internet using an Internet Service Provider (ISP).

[0206] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a procedure, or a part of a code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks depicted in succession can in fact be executed substantially concurrently or the blocks can sometimes be executed in the reverse order, depending upon the functionality involved. It is also noted that each block of the block diagrams or flow diagrams and combinations of blocks in the block diagrams or flow diagrams can be implemented by dedicated hardware-based systems that perform the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0207] Those skilled in the art can understand that the features described in various embodiments of the present disclosure and / or claims can be combined or / and integrated, even if such combinations or integrations are not explicitly described in the present disclosure. In particular, the features described in various embodiments of the present disclosure and / or claims can be combined and / or integrated in various combinations, without departing from the spirit and teachings of the present disclosure. All these combinations and / or integrations fall within the scope of the present disclosure.

[0208] The above describes embodiments of the present disclosure. However, these embodiments are merely for illustrative purposes, and are not intended to limit the scope of the present disclosure. Although each embodiment is described above separately, this does not mean that the measures in each embodiment cannot be used advantageously in combination. The scope of the present disclosure is defined by the appended claims and their equivalents. Those skilled in the art can make various substitutions and modifications without departing from the scope of the present disclosure, and these substitutions and modifications should all fall within the scope of the present disclosure.

Claims

1. A data source configuration method, applied to a data migration stage, the data migration stage comprising: The first sub-stage of switching the first application from the first system to the second system and the second sub-stage after the first sub-stage, characterized in that the data source configuration method comprises: In the second sub-stage, obtaining a first data table of the first system; updating a first lake table in the data lake according to the first data table; determining a second lake table matched with the first lake table from the data lake according to the anti-corrosion configuration information, wherein the second lake table is correspondingly arranged with a second data table of the second system, the first data table and the second data table are data-doubly written by the first application, and the anti-corrosion configuration information comprises a table-to-table correspondence relationship based on a first primary key record; and updating the second lake table with the content of the updated first lake table as a reference; using the updated second lake table as a data source of a second application, wherein the second application comprises a downstream application of the first application; The determining of the second lake table matched with the first lake table from the data lake according to the anti-corrosion configuration information comprises: establishing an association relationship among the first data table, the second data table, the first lake table and the second lake table for recording the same transaction according to the first primary key; and determining the second lake table matched with the first lake table from the data lake according to the association relationship.

2. The data source configuration method of claim 1, wherein, The updating of the first lake table according to the first data table comprises: performing data writing on a first target partition of the first lake table according to the first data table. The updating of the second lake table with the content of the updated first lake table as a reference comprises: determining a second target partition matched with the first target partition in the second lake table; rewriting data in the second target partition according to the anti-corrosion configuration information, with the content in the first target partition as a reference, to update the second lake table.

3. The data source configuration method of claim 2, wherein, The anti-corrosion configuration information comprises a field-to-field correspondence relationship based on a second primary key record. The rewriting of data in the second target partition according to the anti-corrosion configuration information, with the content in the first target partition as a reference, comprises: clearing first target data in the second target partition; and inserting second target data in the first target partition into the second target partition according to the second primary key, to rewrite data in the second target partition.

4. The data source configuration method of claim 1, wherein, The data source configuration method further comprises: In the first sub-stage, obtaining the first data table and the second data table, wherein the first data table and the second data table are data-doubly written by the first application; updating the first lake table according to the first data table; using the updated first lake table as a data source of the second application.

5. The data source configuration method of claim 1, wherein, The data migration stage further comprises a second transition sub-stage, and the data source configuration method further comprises: In the second transition sub-stage, the first application stops writing data into the first data table; updating the second lake table according to the second data table; determining whether to append data to the updated second lake table according to the first lake table; and The second lake table is taken as a data source of the second application.

6. The data source configuration method of claim 1, wherein, The first data table and the second data table perform data interaction with the data lake through message middleware.

7. A data source configuration apparatus, applied to a data migration stage, the data migration stage comprising: The first application is switched from the first system to the second system in a first sub-stage and a second sub-stage after the first sub-stage, and the data source configuration device comprises a first acquisition module, a first update module, a second update module, a first processing module and a second processing module. In the second sub-stage, The first acquisition module is configured to acquire a first data table of the first system. The first update module is configured to update a first lake table in the data lake according to the first data table. The first processing module is configured to determine a second lake table matched with the first lake table from the data lake according to anti-corrosion configuration information, wherein the second lake table is correspondingly arranged with a second data table of the second system, the first data table and the second data table are double-written by the first application, and the anti-corrosion configuration information comprises a table inter-correlation relationship based on a first primary key record; the determination of the second lake table matched with the first lake table from the data lake according to the anti-corrosion configuration information comprises: establishing an association relationship among the first data table, the second data table, the first lake table and the second lake table for recording a same transaction according to the first primary key; and determining the second lake table matched with the first lake table from the data lake according to the association relationship. The second update module is configured to update the second lake table with the content of the updated first lake table as a reference. The second processing module is configured to take the updated second lake table as a data source of a second application, and the second application comprises a downstream application of the first application.

8. An electronic device, comprising: comprise: one or more processors; a storage device for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data source configuration method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the data source configuration method according to any one of claims 1-6.

10. A computer program product, characterised in that, comprise a computer program that, when executed by a processor, implements the data source configuration method according to any one of claims 1-6.

Images