Data migration processing method and system for logistics platform

Abstract

The invention discloses a data migration processing method and system for a logistics platform. The method comprises the following steps: acquiring business data in different management platforms and business platform databases; performing data processing and data conversion on the business data to obtain converted data, and backing up the converted data; migrating the converted data by adopting a migration tool, and carrying out data synchronization, data double writing and data consistency verification on the migrated data; and the data passing the verification is transmitted to the constructed operable data storage and data warehouse. According to the method, the service data of different platforms can be migrated, and the accuracy and consistency of the data can be ensured in the data migration process.

Description

Technical Field

[0001] The present invention relates to the technical field of data migration, and in particular to a data migration processing method and system for a logistics platform. Background Art

[0002] Logistics enterprises have multiple scattered business systems, such as warehouse management systems, transportation management systems, and financial management systems. Through data migration, these scattered data can be integrated into a unified logistics platform to achieve centralized data management and sharing. However, the business data migration process of the logistics platform has the following problems: 1. The problem of large and complex data volume. Because it involves a large amount of business data, including order information, cargo tracking data, customer information, transportation routes, inventory status, etc. These data usually have complex structures and relationships, such as the association between orders and goods, and the matching of transportation routes and vehicles. 2. Data compatibility issues. Because different logistics platforms may use different data formats, database systems, and technical architectures. Therefore, during the data migration process, data compatibility issues need to be resolved. Summary of the invention

[0003] In view of the defects in the prior art, the embodiments of the present invention provide a data migration processing method and system for a logistics platform, which can migrate business data of different platforms and ensure the accuracy and consistency of the data during the data migration process.

[0004] In a first aspect, an embodiment of the present invention provides a data migration processing method for a logistics platform, comprising:

[0005] Obtain business data from different management platforms and business platform databases;

[0006] Process and convert the business data to obtain converted data, and back up the converted data;

[0007] Use migration tools to migrate the converted data, and perform data synchronization, data double writing, and data consistency verification on the migrated data;

[0008] The validated data is then transferred to the constructed operational data storage and data warehouse.

[0009] In a second aspect, an embodiment of the present invention provides a data migration processing system for a logistics platform, comprising: a data acquisition module, a data processing module, a data migration module and a storage module;

[0010] The data acquisition module is used to acquire business data from different management platforms and business platform databases;

[0011] The data processing module is used to process and convert the business data to obtain the converted data and back up the converted data;

[0012] The data migration module is used to migrate the converted data using a migration tool, and perform data synchronization, data double writing and data consistency verification on the migrated data;

[0013] The storage module is used to transmit the verified data to the constructed operational data storage and data warehouse.

[0014] Beneficial effects of the present invention:

[0015] The embodiments of the present invention provide a data migration processing method and system for a logistics platform, which can migrate business data of different platforms and ensure the accuracy and consistency of the data during the data migration process. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0017] Figure 1 A flow chart showing a data migration processing method for a logistics platform provided by a first embodiment of the present invention;

[0018] Figure 2 A business data migration deployment architecture diagram is shown;

[0019] Figure 3 A schematic diagram of business data migration is shown;

[0020] Figure 4 A business data migration flow chart is shown;

[0021] Figure 5 A structural block diagram of a data migration processing system for a logistics platform provided by another embodiment of the present invention is shown. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0024] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0025] It should be further understood that the term "and / or" used in the present specification and the appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0026] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0027] It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should have the common meanings understood by those skilled in the art to which the invention belongs.

[0028] like Figures 1 to 4 As shown, a data migration processing method for a logistics platform provided by a first embodiment of the present invention includes the following steps:

[0029] Obtain business data from different management platforms and business platform databases;

[0030] Process and convert the business data to obtain converted data, and back up the converted data;

[0031] Use migration tools to migrate the converted data, and perform data synchronization, data double writing, and data consistency verification on the migrated data;

[0032] The validated data is then transferred to the constructed operational data storage and data warehouse.

[0033] In this embodiment, business data in different management platforms and business platform databases are obtained, and the business data include: dictionary data, driver data, vehicle data, order data, waybill data, dispatch order and weighing order. In practical applications, the management platform includes an order management system (OMS), a transportation management system (TMS), a master data management (MDM), a database (DB), etc. Among them, the OMS ERP enterprise management information system is an enterprise management software covering procurement management, sales management, inventory management, e-commerce, financial management, customer management, trading partners and other links. The order management system (OMS) is a part of the logistics management system. By managing and tracking the orders placed by customers, the progress and completion of orders are dynamically grasped, and the operation efficiency in the logistics process is improved, thereby saving operation time and operation costs and improving the market competitiveness of logistics companies. The main function of the order management system is to provide users with integrated one-stop supply chain services through unified orders. Order management and order tracking management can enable users to obtain full satisfaction of logistics services. The order management system is an indispensable part of the logistics management chain. Through the management and allocation of orders, the warehousing management and transportation management are organically combined, and each link in the logistics management is fully utilized stably and effectively, so that warehousing, transportation and orders become an organic whole, meeting the needs of logistics system informatization. The logistics and transportation management system (TMS) is suitable for transportation companies, transportation teams under various enterprises, etc. It mainly includes order management, scheduling and allocation, driving management, GPS vehicle positioning system, vehicle management, personnel management, data reporting, basic information maintenance, system management and other modules. The system conducts comprehensive and detailed statistical assessments on vehicles, drivers, routes, etc., which can greatly improve operational efficiency, reduce transportation costs, and put you in a leading position in the fierce market competition. Master data management (MDM) is a connected, flexible and scalable platform that can effectively manage master data and provide a single point of fact for business-critical data. It supports business plans by confirming, linking and integrating information such as products, customers, stores / locations, employees, suppliers, digital assets, etc. After data processing, migration tools are used to migrate the converted data, and data synchronization, data double writing and data consistency verification are performed on the migrated data; the data that passes the verification is transmitted to the constructed operational data storage and data warehouse. The construction of a data warehouse can store the data in various distributed platforms in a dedicated database, reducing the consumption of computer resources caused by the analysis and statistics of logistics data.

[0034] Migrate the data in these systems. When collecting data from the order management system, transportation management system, and master data management, the migration tool canal is used. Based on the incremental log analysis of the database, incremental data subscription and consumption are provided. Currently, MySQL (also supports mariaDB). Businesses supported based on log incremental subscription and consumption: 1. Database mirroring; 2. Real-time database backup; 3. Multi-level indexing (sellers and buyers have separate database indexes); 4. Search build; 5. Business cache refresh; 6. Important business messages such as price changes. When migrating data from the database, the ETL tool uses kettle, which can run on multiple platforms and supports a variety of data sources (including: xml, json, EDI, txt, word, excel, hbase, elasticsearch, Mysql, Postgresql, oracle, sqlserver, etc. The data formats of the above systems are different, so the data needs to be processed.

[0035] Data processing includes cleaning and converting data. When cleaning data, the data in the source system is cleaned and sorted, duplicate data, erroneous data and unnecessary data are removed, and missing data is supplemented. Data conversion is to convert and map data according to the requirements of the target system, and convert the source data into a format acceptable to the target system.

[0036] Specifically, the specific methods for processing business data include:

[0037] Delete duplicate data, erroneous data and unnecessary data in business data;

[0038] Data with missing information are filled using global variables, mean values ​​or medians.

[0039] Specifically, the specific methods for converting business data include:

[0040] Dictionary data, driver data and vehicle data are mapped and converted based on the management platform;

[0041] Unify the numbering of business document data in the management platform and business platform, and generate new numbers for the order data, waybill data, vehicle dispatch orders and weighing orders in the business platform.

[0042] In this embodiment, the specific method of performing data double writing on the migration data includes:

[0043] Configure the main database and backup database, use the operational data storage as the main database and the data warehouse as the backup database. The main database is responsible for all read and write operations, and the backup database is responsible for receiving the written data of the main database and backing it up;

[0044] When writing data, the data is first written to the primary database, which records the write operation in the log and returns a successful response to the application.

[0045] The primary database waits for confirmation from the backup database and then writes the data to the backup database.

[0046] In this embodiment, data double writing is used to process the migration data. Database double writing is a mechanism for data protection and data consistency. It writes data to two or more different physical locations at the same time to ensure that data can be restored and consistency is maintained in the event of a failure or error. The operational data store is used as the main database and the data warehouse is used as the backup database. The main database is responsible for all read and write operations, and the backup database is responsible for receiving the write data of the main database and backing it up. When writing data, the data is first written to the operational data store, and the operational data store records the write operation in the log and returns a successful response to the application. The operational data store waits for the confirmation of the data warehouse before writing the data to the data warehouse. This can prevent data loss or damage during data migration, improve the reliability of data migration, and improve the integrity and consistency of data.

[0047] After the data is double-written, the consistency of the data is checked. First, the consistency of the read interface is checked, including consistency check, automated testing, business testing, stress testing, and performance optimization. After the check passes, the read interface is switched and grayscale is performed according to the traffic. If an exception occurs, the grayscale is closed and the read interface consistency check is returned. After the read interface switching is completed, the write interface is switched, and the write interface is grayscaled, grayscaled in batches, and rollback is supported. After the data is repaired when an exception occurs, the read interface switching operation is returned. After the write interface switching is completed, the double writing is stopped.

[0048] A data migration processing method for a logistics platform provided in an embodiment of the present invention can migrate business data of different platforms and ensure the accuracy and consistency of the data during the data migration process.

[0049] In the above first embodiment, a data migration processing method for a logistics platform is provided. Correspondingly, the present application also provides a data migration processing system for a logistics platform. Figure 5, which is a structural block diagram of a data migration processing system for a logistics platform provided by the second embodiment of the present invention. Since the device embodiment is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described below is only illustrative.

[0050] like Figure 5 As shown, another embodiment of the present invention provides a data migration processing system for a logistics platform, including: a data acquisition module, a data processing module, a data migration module and a storage module; the data acquisition module is used to acquire business data from different management platforms and business platform databases; the data processing module is used to process and convert the business data to obtain the converted data, and back up the converted data; the data migration module is used to migrate the converted data using a migration tool, and perform data synchronization, data double writing and data consistency verification on the migrated data; the storage module is used to transmit the verified data to the constructed operational data storage and data warehouse.

[0051] Business data includes: dictionary data, driver data, vehicle data, order data, waybill data, vehicle dispatch order and weighing order.

[0052] The data processing module includes a processing unit, which is used to delete duplicate data, erroneous data and unnecessary data in the business data, and fill in the missing data with global variables, average values ​​or medians. The data processing module includes a conversion unit, which is used to map and convert dictionary data, driver data and vehicle data based on the management platform, unify the business document data in the management platform and the business platform, and generate new numbers for the order data, waybill data, vehicle dispatch order and weighing order in the business platform.

[0053] The data migration module includes a data double-write unit, which is used to configure the main database and the backup database, store the operational data as the main database, and use the data warehouse as the backup database. The main database is responsible for all read and write operations, and the backup database is responsible for receiving the write data of the main database and backing it up; when writing data, the data is first written to the main database, the main database records the write operation in the log, and returns a successful response to the application; the main database waits for the confirmation of the backup database, and then writes the data to the backup database.

[0054] A data migration processing system for a logistics platform provided by an embodiment of the present invention can migrate business data of different platforms and ensure the accuracy and consistency of the data during the data migration process.

[0055] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A data migration processing method for a logistics platform, characterized in that: include: Obtain business data from different management platforms and business platform databases; Process and convert the business data to obtain converted data, and back up the converted data; Use migration tools to migrate the converted data, and perform data synchronization, data double writing, and data consistency verification on the migrated data; The validated data is then transferred to the constructed operational data storage and data warehouse.

2. The data migration processing method of the logistics platform according to claim 1, characterized in that: The business data includes: dictionary data, driver data, vehicle data, order data, waybill data, vehicle dispatch order and weighing order.

3. The data migration processing method of the logistics platform according to claim 2, characterized in that: The specific method of processing the business data includes: Delete duplicate data, erroneous data and unnecessary data in business data; Data with missing information are filled using global variables, mean values ​​or medians.

4. The data migration processing method of the logistics platform according to claim 3, characterized in that: The specific method of converting the business data includes: Dictionary data, driver data and vehicle data are mapped and converted based on the management platform; Unify the numbering of business document data in the management platform and business platform, and generate new numbers for the order data, waybill data, vehicle dispatch orders and weighing orders in the business platform.

5. The data migration processing method of the logistics platform according to claim 1, characterized in that: The specific method of performing data double writing on the migration data includes: Configure the main database and backup database, use the operational data storage as the main database and the data warehouse as the backup database. The main database is responsible for all read and write operations, and the backup database is responsible for receiving and backing up the written data of the main database; When writing data, the data is first written to the primary database, which records the write operation in the log and returns a successful response to the application. The primary database waits for confirmation from the backup database and then writes the data to the backup database.

6. A data migration processing system for a logistics platform, characterized in that: include: Data acquisition module, data processing module, data migration module and storage module; The data acquisition module is used to acquire business data from different management platforms and business platform databases; The data processing module is used to process and convert the business data to obtain the converted data and back up the converted data; The data migration module is used to migrate the converted data using a migration tool, and perform data synchronization, data double writing and data consistency verification on the migrated data; The storage module is used to transmit the verified data to the constructed operational data storage and data warehouse.

7. The system according to claim 6, characterized in that The business data includes: dictionary data, driver data, vehicle data, order data, waybill data, vehicle dispatch order and weighing order.

8. The system according to claim 7, characterized in that The data processing module includes a processing unit, which is used to delete duplicate data, erroneous data and unnecessary data in the business data, and fill in the data with missing information using global variables, average values ​​or medians.

9. The system according to claim 8, characterized in that The data processing module includes a conversion unit, which is used to map and convert dictionary data, driver data and vehicle data based on the management platform, unify the numbering of business document data in the management platform and the business platform, and generate new numbers for order data, waybill data, vehicle dispatch orders and weighing orders in the business platform.

10. The system according to claim 6, characterized in that The data migration module includes a data double-write unit, which is used to configure a main database and a backup database, and uses the operational data storage as the main database and the data warehouse as the backup database. The main database is responsible for all read and write operations, and the backup database is responsible for receiving and backing up the written data of the main database; When writing data, the data is first written to the primary database, which records the write operation in the log and returns a successful response to the application. The primary database waits for confirmation from the backup database and then writes the data to the backup database.

Images