Travel mass data storage method

Abstract

The invention discloses a travel mass data storage method. The method comprises the following steps: S1, performing library division according to a travel user ID (Identity) mode; s2, a unique data storage method of a driver and passengers; s3, sharing a data storage method by the driver and the passenger; s4, an external heterogeneous data storage method; and S5, a program JOB data processing and storage method. According to the method, the data storage problem of the travel industry is solved, the storage scale can be increased according to needs, cost input does not need to be carried out in advance, and the data storage problem of user scale increase can be solved.

Description

Technical Field

[0001] The present invention relates to the field of travel such as online car-hailing and taxis, and in particular to a method for storing massive travel data. Background Art

[0002] The amount of data in the travel industry is huge. As the user scale grows, the number of orders per day will reach hundreds of thousands to tens of millions. However, a single database cannot accommodate such a large storage scale. There are several problems with storing data in a single database: (1) The disk cannot support the expansion of extremely large data volumes; (2) The query and modification performance of a single table will decline severely as the data scale increases.

[0003] Based on this, the present invention provides a method for storing massive travel data. The method combines the characteristics of travel services and can support massive data expansion without performance degradation. Summary of the invention

[0004] The present invention is aimed at the deficiencies in the prior art and provides a method for storing massive travel data to solve the data storage problem in the travel industry. The method can increase the storage scale as needed without the need for advance cost investment, and can solve the data storage problem as the user scale grows.

[0005] To solve the above problems, the technical solutions adopted by the present invention are as follows:

[0006] A method for storing mass travel data comprises the following steps:

[0007] S1. Database division by travel user ID

[0008] As the scale of travel users grows, the number of users will increase;

[0009] If the newly added users are divided into a new database, there are the following benefits:

[0010] (1) The existing database can be left unaffected. New users can access the new database, while old users can continue to access the old database.

[0011] (2) It can support the expansion of data scale as needed. When the new database cannot support new users, other databases can be added.

[0012] (3) The cost increases with the number of users, so there is no need to waste costs by building a new database in advance;

[0013] The specific rules for user ID database division are as follows:

[0014] (3) User ID is stored in string form;

[0015] (4) The last 4 digits of the user ID uniquely represent a database, which serves as a sub-database dependency;

[0016] For example, if the user ID is dkajdje4idadn787uudayefd762d0000, the last 4 digits 0000 indicate that the data is in database No. 0. For example, when the number of registered users is less than 20 million, the last 4 digits of the newly registered user ID generated are 0000, and the data is in database No. 0. When the number of registered users is between 20 million and 40 million, the last 4 digits of the newly registered user ID generated are 0001, and the data is in database No. 1, and so on.

[0017] For the travel industry, the user ID of the driver and the passenger is the same, that is, the driver ID and passenger ID of the same person are the same;

[0018] S2. Driver and passenger unique data storage method

[0019] The unique data of drivers and passengers is that the driver ID and passenger ID will not exist at the same time in the data table, and only one driver and passenger will appear; taking rewards as an example in the travel industry, the key information in the reward table includes: reward number, driver ID; the reward number is stored in a passenger self-contained manner, and the generated reward number is generated in the following rules: the last 4 digits of the reward number are the same as the last 4 digits of the driver ID; this is a self-contained method, so that querying / updating rewards by reward number, querying / updating rewards by driver, or operating other driver information can be kept in the same database, so that there is no need to consider cross-database issues in transactions that operate user data and reward data;

[0020] S3, driver and passenger shared data storage method

[0021] The information shared by the driver and the passenger is that both the driver and the passenger appear in the information. For example, in the travel industry, a passenger calls a driver to complete an order. The key information in the order includes: travel order number, passenger ID, and driver ID. The order is stored in a passenger-self-contained manner, and the rule for generating the travel order number is: the last 4 digits of the order number are the same as the last 4 digits of the passenger ID. This is a self-contained manner, so that querying / updating by travel order number, querying / updating orders by passengers, or operating passenger information are kept in one database, so that there is no need to consider cross-database transaction issues when operating user data and order data.

[0022] The method for processing drivers in the shared information is to use the event log table method. For orders, passengers can query order information in the database corresponding to the passenger ID according to the order, or they can query order information in the database corresponding to the driver ID according to the driver;

[0023] Specifically:

[0024] (1) When updating order information, an event log is inserted into the event log table at the same time. The format is: event number, travel order number, passenger ID, driver ID, and other order information;

[0025] The last 4 digits of the event number are the same as the last 4 digits of the passenger ID;

[0026] (2) The software processes the newly added data in the event log table periodically, usually in a processing cycle of 1-3 seconds. The processing method is to create or update the corresponding database record as the order data in the event log table in the database corresponding to the driver ID;

[0027] (3) After the scheduled event log is successfully processed, the record is deleted;

[0028] S4. External heterogeneous data storage method

[0029] The data format of external cooperative merchants cannot be adjusted by the enterprise, so a heterogeneous data storage method should be adopted, specifically, to establish heterogeneous tables to store the data of external cooperative merchants;

[0030] For example, the key order data of an external cooperative merchant include: merchant number, merchant order number. The storage steps are as follows:

[0031] (1) Generate an internal travel order number based on the merchant order number;

[0032] (2) Store in separate databases according to merchant numbers, such as merchants No. 1-100 are stored in database No. 0, and merchants No. 101-200 are stored in database No. 1. This method is the same as the method in step S1, and can be expanded as needed, without the need to purchase a database in advance and waste costs, and is infinitely expandable;

[0033] (3) Establish heterogeneous tables to store: merchant number, merchant order number, and travel order number;

[0034] S5. Program JOB data processing and storage method

[0035] Program JOB is generally a global scheduled task. In single database mode, only the data of this database needs to be processed. The method for processing multiple databases by user is to read a configuration file. The configuration file defines the names of all related databases. The JOB processes all related databases in a loop according to the read database names.

[0036] Establish a separate database to store the running results of the job. Every time a job processes the data in its related database, record a processing result in the separate database. The specific record format is: job name, processing time, processing result, and database name list. In this way, you can audit whether the data results of the database job in multiple databases are correct, which is convenient for locating and troubleshooting the cause of the fault.

[0037] Compared with the prior art, the implementation effects of the present invention are as follows:

[0038] The present invention provides a method for storing massive travel data to solve the data storage problem of travel companies when the user scale grows. The method can effectively solve the storage problem by formulating processing rules for sub-libraries, unique data, shared data, heterogeneous data, and JOB data.

[0039] The advantage of this method is that it can predict that there will be a large number of users in the future, but without increasing the cost of additional data storage, it can solve the problem of massive data storage through an expansion mechanism. This method can effectively improve the scalability of data storage, and there is no bottleneck limit on the system's response time and system pressure. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 It is a schematic diagram of the process of the present invention. DETAILED DESCRIPTION

[0041] The content of the present invention will be described below with reference to specific embodiments.

[0042] like Figure 1 As shown, the invention provides a method for storing massive travel data to solve the data storage problem in the travel industry. The method can increase the storage scale as needed without the need for advance cost investment, and can solve the data storage problem as the user scale grows.

[0043] A method for storing massive travel data comprises the following steps:

[0044] S1. Database division by travel user ID

[0045] As the scale of travel users grows, the number of users will increase;

[0046] If the newly added users are divided into a new database, there are the following benefits:

[0047] (1) The existing database can be left unaffected. New users can access the new database, while old users can continue to access the old database.

[0048] (2) It can support the expansion of data scale as needed. When the new database cannot support new users, other databases can be added.

[0049] (3) The cost increases with the number of users, so there is no need to waste costs by building a new database in advance;

[0050] The specific rules for user ID database division are as follows:

[0051] (5) User ID is stored in string form;

[0052] (6) The last 4 digits of the user ID uniquely represent a database, which serves as a sub-database dependency;

[0053] For example, if the user ID is dkajdje4idadn787uudayefd762d0000, the last 4 digits 0000 indicate that the data is in database No. 0. For example, when the number of registered users is less than 20 million, the last 4 digits of the newly registered user ID generated are 0000, and the data is in database No. 0. When the number of registered users is between 20 million and 40 million, the last 4 digits of the newly registered user ID generated are 0001, and the data is in database No. 1, and so on.

[0054] For the travel industry, the user ID of the driver and the passenger is the same, that is, the driver ID and passenger ID of the same person are the same;

[0055] S2. Driver and passenger unique data storage method

[0056] The unique data of drivers and passengers is that the driver ID and passenger ID will not exist at the same time in the data table, and only one driver and passenger will appear; taking rewards as an example in the travel industry, the key information in the reward table includes: reward number, driver ID; the reward number is stored in a passenger self-contained manner, and the generated reward number is generated in the following rules: the last 4 digits of the reward number are the same as the last 4 digits of the driver ID; this is a self-contained method, so that querying / updating rewards by reward number, querying / updating rewards by driver, or operating other driver information can be kept in the same database, so that there is no need to consider cross-database issues in transactions that operate user data and reward data;

[0057] S3, driver and passenger shared data storage method

[0058] The information shared by the driver and the passenger is that both the driver and the passenger appear in the information. For example, in the travel industry, a passenger calls a driver to complete an order. The key information in the order includes: travel order number, passenger ID, and driver ID. The order is stored in a passenger-self-contained manner, and the rule for generating the travel order number is: the last 4 digits of the order number are the same as the last 4 digits of the passenger ID. This is a self-contained manner, so that querying / updating by travel order number, querying / updating orders by passengers, or operating passenger information are kept in one database, so that there is no need to consider cross-database transaction issues when operating user data and order data.

[0059] The method for processing drivers in the shared information is to use the event log table method. For orders, passengers can query order information in the database corresponding to the passenger ID according to the order, or they can query order information in the database corresponding to the driver ID according to the driver;

[0060] Specifically:

[0061] (1) When updating order information, an event log is inserted into the event log table at the same time. The format is: event number, travel order number, passenger ID, driver ID, and other order information;

[0062] The last 4 digits of the event number are the same as the last 4 digits of the passenger ID;

[0063] (2) The software processes the newly added data in the event log table periodically, usually in a processing cycle of 1-3 seconds. The processing method is to create or update the corresponding database record as the order data in the event log table in the database corresponding to the driver ID;

[0064] (3) After the scheduled event log is successfully processed, the record is deleted;

[0065] S4. External heterogeneous data storage method

[0066] The data format of external cooperative merchants cannot be adjusted by the enterprise, so a heterogeneous data storage method should be adopted, specifically, to establish heterogeneous tables to store the data of external cooperative merchants;

[0067] For example, the key order data of an external cooperative merchant include: merchant number, merchant order number. The storage steps are as follows:

[0068] (2) Generate an internal travel order number for the enterprise based on the merchant order number;

[0069] (2) Store in separate databases according to merchant numbers, such as merchants No. 1-100 are stored in database No. 0, and merchants No. 101-200 are stored in database No. 1. This method is the same as the method in step S1, and can be expanded as needed, without the need to purchase a database in advance and waste costs, and is infinitely expandable;

[0070] (3) Establish heterogeneous tables to store: merchant number, merchant order number, and travel order number;

[0071] S5. Program JOB data processing and storage method

[0072] Program JOB is generally a global scheduled task. In single database mode, only the data of this database needs to be processed. The method for processing multiple databases by user is to read a configuration file. The configuration file defines the names of all related databases. The JOB processes all related databases in a loop according to the read database names.

[0073] Establish a separate database to store the running results of the job. Every time a job processes the data in its related database, record a processing result in the separate database. The specific record format is: job name, processing time, processing result, and database name list. In this way, you can audit whether the data results of the database job in multiple databases are correct, which is convenient for locating and troubleshooting the cause of the fault.

[0074] The present invention provides a method for storing massive travel data to solve the data storage problem of travel companies when the user scale grows. The method can effectively solve the storage problem by formulating processing rules for sub-libraries, unique data, shared data, heterogeneous data, and JOB data.

[0075] The advantage of this method is that it can predict that there will be a large number of users in the future, but without increasing the cost of additional data storage, it can solve the problem of massive data storage through an expansion mechanism. This method can effectively improve the scalability of data storage, and there is no bottleneck limit on the system's response time and system pressure.

[0076] In summary, compared with the traditional single-database travel data storage or the method of dividing the number of sub-databases from the beginning, it has the advantages of adjusting the number of sub-databases on demand, dynamically increasing sub-databases, and avoiding the complexity of distributed transactions. It improves the continuous scalability of massive travel data storage, so that travel companies do not need to consider business failures or interruptions caused by major technical architecture adjustments or upgrades due to data growth.

[0077] The above content is a detailed description of the present invention in combination with specific embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For those skilled in the art of the present invention, without departing from the concept of the present invention, several simple deductions or substitutions can be made, which should be regarded as falling within the scope of protection of the present invention.

Claims

1. A method for storing massive travel data, characterized in that: The following steps are involved: S1. Database division by travel user ID The specific rules for user ID database division are as follows: (1) User ID is stored in string form; (2) The last 4 digits of the user ID uniquely represent a database, which serves as a sub-database dependency; For example, if the user ID is dkajdje4idadn787uudayefd762d0000, the last 4 digits 0000 indicate that the data is in database No.

0. For example, when the number of registered users is less than 20 million, the last 4 digits of the newly registered user ID generated are 0000, and the data is in database No.

0. When the number of registered users is between 20 million and 40 million, the last 4 digits of the newly registered user ID generated are 0001, and the data is in database No. 1, and so on. For the travel industry, the user ID of the driver and the passenger is the same, that is, the driver ID and passenger ID of the same person are the same; S2. Driver and passenger unique data storage method The unique data of drivers and passengers is that the driver ID and passenger ID will not exist at the same time in the data table. There will only be one driver and one passenger. Taking the travel industry as an example, the key information in the reward table is: reward number, driver ID; the reward number is stored in a passenger self-contained manner, and the generated reward number is generated in the following rules: the last 4 digits of the reward number are the same as the last 4 digits of the driver ID; This is a self-contained approach that allows querying / updating rewards by reward number, querying / updating rewards by driver, or operating other driver information to be kept in the same database, so that transactions operating user data and reward data do not need to consider cross-database issues; S3, driver and passenger shared data storage method The information shared by the driver and the passenger is that both the driver and the passenger appear in the information. For example, in the travel industry, a passenger calls a driver to complete an order. The key information in the order includes: travel order number, passenger ID, and driver ID. The order is stored in a passenger-contained manner. The rule for generating the travel order number is: the last 4 digits of the order number are the same as the last 4 digits of the passenger ID. This is a self-contained way to query / update by travel order number, query / update by passenger, or operate passenger information in one database, so that there is no need to consider cross-database transaction issues when operating user data and order data; The method for processing drivers in the shared information is to use the event log table method. For orders, passengers can query order information in the database corresponding to the passenger ID according to the order, or they can query order information in the database corresponding to the driver ID according to the driver; Specifically: (1) When updating order information, an event log is inserted into the event log table at the same time. The format is: event number, travel order number, passenger ID, driver ID, and other order information; The last 4 digits of the event number are the same as the last 4 digits of the passenger ID; (2) The software processes the newly added data in the event log table periodically, usually in a processing cycle of 1-3 seconds. The processing method is to create or update the corresponding database record as the order data in the event log table in the database corresponding to the driver ID; (3) After the scheduled event log is successfully processed, the record is deleted; S4. External heterogeneous data storage method The data format of external cooperative merchants cannot be adjusted by the enterprise, so a heterogeneous data storage method should be adopted, specifically, to establish heterogeneous tables to store the data of external cooperative merchants; S5. Program JOB data processing and storage method Program JOB is generally a global scheduled task. In single database mode, only the data of this database needs to be processed. The method for processing multiple databases by user is to read a configuration file. The configuration file defines the names of all related databases. The JOB processes all related databases in a loop according to the read database names. Establish a separate database to store the running results of the job. Every time a job processes the data in its related database, record a processing result in the separate database. The specific record format is: job name, processing time, processing result, and database name list. In this way, you can audit whether the data results of the database job in multiple databases are correct, which is convenient for locating and troubleshooting the cause of the fault.

Images