Mass data query method and system suitable for logistics platform

Abstract

The invention provides a mass data query method and system suitable for a logistics platform. The method comprises the steps that a production library is set; the production library stores hot data, and the hot data comprises business data of a current month and previous N months; setting a history library; a hot partition of the historical library backs up part of hot data; the cold partition of the historical library backs up the cold data; the cold data is business data before the first N months; when a data query request of a business party is received, querying the production library according to the data query request; if the query result returned by the production library is empty, querying a hot partition in a historical library according to the data query request; if the query result returned by the historical library is not null, returning the query result to the business party; and if the query result returned by the production library is not null, returning the query result to the business party. According to the method, cold and hot partition processing is carried out on the mass data, the amount of data searched during data query is reduced, the mass data query speed is increased, and resources occupied by mass data query are reduced.

Description

Technical Field

[0001] The present invention belongs to the technical field of databases, and in particular relates to a mass data query method and system applicable to a logistics platform. Background Art

[0002] When the amount of data is large, the database needs to search and filter in a large amount of data, resulting in slower query speeds. For example, in a small database, querying data under specific conditions may only take a few milliseconds, but when the amount of data grows to millions or even billions of records, the same query may take several seconds, tens of seconds or even longer. In addition, queries with large amounts of data also require more memory to store temporary result sets, indexes, and query execution plans, more CPU computing resources, and more storage space. Summary of the invention

[0003] In view of the defects in the prior art, the present invention provides a mass data query method and system suitable for a logistics platform, which improves the mass data query speed and reduces the resources occupied by the mass data query.

[0004] In a first aspect, a massive data query method applicable to a logistics platform includes:

[0005] Set up a production library; the production library is used to store hot data, and the hot data includes business data of the current month and the previous N months, where N is an integer;

[0006] Set up a history database; the history database includes hot partitions and cold partitions; the hot partition is used to back up some hot data; the cold partition is used to back up cold data; cold data is business data before the previous N months;

[0007] When receiving a data query request from the business party, query the production library according to the data query request; if the query result returned by the production library is empty, query the hot partitions in the history library according to the data query request; if the query result returned by the history library is not empty, return the query result to the business party; if the query result returned by the production library is not empty, return the query result to the business party.

[0008] Furthermore, after setting up the history library, it also includes:

[0009] If the backup time of the current month arrives, the business data of the previous month in the production database will be backed up to the hot partition of the historical database, and the business data of the earliest month in the hot partition will be backed up to the cold partition of the historical database.

[0010] Furthermore, after backing up the earliest month's business data in the hot partition to the cold partition of the history library, the following steps are also included:

[0011] Delete the data partition corresponding to the earliest month's business data in the hot partition.

[0012] Furthermore, after setting up the history library, it also includes:

[0013] When receiving a cold data hot conversion request from the business party, forward the cold data hot conversion request to the administrator for review; the cold data hot conversion request includes part of the cold data in the cold partition of the historical database;

[0014] When the cold data to hot request is approved, the cold data in the cold data to hot request is defined as hot data, and the cold data is backed up to the production database.

[0015] Furthermore, after setting up the history library, it also includes:

[0016] When changes are detected in the business data in the production database, a synchronization request containing a data identifier and the changed business data is generated; the data identifier is used to uniquely identify the changed business data;

[0017] When the history database receives a synchronization request and the business data corresponding to the data identifier exists in the history database, the business data in the history database is updated to the changed business data.

[0018] In the second aspect, a massive data query system suitable for a logistics platform includes:

[0019] Production database: used to store hot data, including business data of the current month and the previous N months, where N is an integer;

[0020] History database: includes hot partitions and cold partitions. Hot partitions are used to back up some hot data. Cold partitions are used to back up cold data. Cold data refers to business data from the previous N months.

[0021] Query unit: used to query the production library according to the data query request when receiving the data query request from the business party; if the query result returned by the production library is empty, query the hot partitions in the history library according to the data query request; if the query result returned by the history library is not empty, return the query result to the business party; if the query result returned by the production library is not empty, return the query result to the business party.

[0022] Furthermore, it also includes:

[0023] Data backup unit: used to back up the business data of the previous month in the production database to the hot partition of the historical database when the backup time of the current month arrives, and to back up the business data of the earliest month in the hot partition to the cold partition of the historical database.

[0024] Furthermore, after backing up the business data of the earliest month in the hot partition to the cold partition of the history library, the data backup unit further includes:

[0025] Delete the data partition corresponding to the earliest month's business data in the hot partition.

[0026] Furthermore, it also includes:

[0027] Data hot conversion unit: when receiving a cold data hot conversion request from the business party, forward the cold data hot conversion request to the administrator for review; the cold data hot conversion request includes part of the cold data in the cold partition of the historical database; when the cold data hot conversion request is approved, define the cold data in the cold data hot conversion request as hot data, and back up the cold data to the production database.

[0028] Furthermore, it also includes:

[0029] Synchronization unit: used to generate a synchronization request containing a data identifier and the changed business data when it detects changes in the business data in the production library; the data identifier is used to uniquely identify the changed business data; when the historical library receives the synchronization request and the historical library contains business data corresponding to the data identifier, the business data in the historical library is updated to the changed business data.

[0030] It can be seen from the above technical scheme that the massive data query method and system suitable for the logistics platform provided by the present invention perform cold and hot partition processing on massive data, reduce the amount of data searched during data query, improve the massive data query speed, and reduce the resources occupied by massive data query. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following is a brief introduction to the drawings required for the specific embodiments or the description of the prior art. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn according to the actual scale.

[0032] Figure 1 The present invention is a flowchart of a data query method provided in an embodiment.

[0033] Figure 2 A flow chart of a method for converting cold data to hot data provided in an embodiment.

[0034] Figure 3 A software interface diagram for a business party to initiate a request to convert cold data into hot data provided in an embodiment.

[0035] Figure 4 A diagram of a software interface for an administrator to review cold data to hot data provided in an embodiment.

[0036] Figure 5 A software interface diagram for an administrator to execute a request to convert cold data to hot data provided in an embodiment.

[0037] Figure 6 A module block diagram of a data query system provided in an embodiment. DETAILED DESCRIPTION

[0038] The embodiments of the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and are therefore only used as examples, and the protection scope of the present invention cannot be limited thereto. It should be noted that, unless otherwise specified, the technical terms or scientific terms used in this application should be the common meanings understood by those skilled in the art to which the present invention belongs.

[0039] It should be understood that when used in this specification and the appended claims, the terms "include" and "comprises" indicate the presence of described features, integers, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0040] It should also be understood that the terms used in this specification of the present invention are only for the purpose of describing specific embodiments and are not intended to limit the present invention. As used in the specification of the present invention and the appended claims, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are intended to include plural forms.

[0041] As used in this specification and the appended claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0042] Example:

[0043] A massive data query method suitable for logistics platforms, see Figure 1 ,include:

[0044] Set up a production library; the production library is used to store hot data, and the hot data includes business data in the current month and the previous N months, where N is an integer;

[0045] Set up a history database; the history database includes hot partitions and cold partitions; the hot partition is used to back up some hot data; the cold partition is used to back up cold data; cold data is business data before the previous N months;

[0046] When receiving a data query request from the business party, query the production library according to the data query request; if the query result returned by the production library is empty, query the hot partitions in the history library according to the data query request; if the query result returned by the history library is not empty, return the query result to the business party; if the query result returned by the production library is not empty, return the query result to the business party.

[0047] In this embodiment, since the business data in the current month and the previous N months are frequently used, queried or operated, the business data with a higher frequency of use is defined as hot data. For historical data that is not frequently queried, that is, business data before the previous N months, it is defined as cold data, which ensures the integrity of the data and also reduces the amount of data to search for hot data during data query. For example, assuming that the current month is June, N=3, then the business data for March, April, May, and June are hot data, and the business data for January and February are cold data. The production library only stores hot data, and the historical library is provided with multiple data partitions, of which the hot partition is used to back up some hot data, and the cold partition is used to back up cold data. The business data in the production library can be queried and operated by the business party, and the business data in the hot partition in the historical library can be queried by the business party, and the business data in the cold partition does not provide query or operation functions for the time being.

[0048] In this embodiment, when the business party (for example, a cargo owner in the transportation industry) wants to query data, a data query request is initiated. At this time, the production library is queried according to the data query request. If the production library does not find relevant data, the historical library is queried according to the data query request, and the queried data is returned to the business party.

[0049] The massive data query method applicable to the logistics platform performs hot and cold partition processing on the massive data, reduces the amount of data to be searched during data query, improves the massive data query speed, and reduces the resources occupied by the massive data query.

[0050] Furthermore, in some embodiments, after setting the history library, the method further includes:

[0051] If the backup time of the current month arrives, the business data of the previous month in the production database will be backed up to the hot partition of the historical database, and the business data of the earliest month in the hot partition will be backed up to the cold partition of the historical database.

[0052] In this embodiment, the method can set the backup time to a certain day of each month, for example, it can be set to 2:30 am on the 27th of each month. Before 2:30 am on June 27th, the business data of March, April, May, and June are stored in the production library, the hot partition in the history library stores the business data of February, March, and April, and the cold partition of the history library stores the business data of January. When 2:30 am on June 27th, the business data of May in the production library is backed up to the hot partition of the history library, and the business data of February in the hot partition is backed up to the cold partition of the history library. Then the business data of March, April, May, and June are still stored in the production library, the hot partition in the history library stores the business data of March, April, and May, and the cold partition of the history library stores the business data of January and February. The production library can delete expired business data regularly. For example, when July arrives, the business data of March is converted from hot data to cold data, and the production library will delete the business data of March.

[0053] Furthermore, in some embodiments, after backing up the business data of the earliest month in the hot partition to the cold partition of the history library, the method further includes:

[0054] Delete the data partition corresponding to the earliest month's business data in the hot partition.

[0055] In this embodiment, normal business may involve the operation of deleting certain business data in the hot partition. Therefore, in order to perform the operation of deleting data for normal business in the hot partition and the operation of backing up cold data, this method backs up the business data of the earliest month in the hot partition to the cold partition of the historical library, and then deletes the data partition corresponding to the business data.

[0056] Further, in some embodiments, after setting the history library, see Figure 2 , also includes:

[0057] When receiving a cold data hot conversion request from the business party, forward the cold data hot conversion request to the administrator for review; the cold data hot conversion request includes part of the cold data in the cold partition of the historical database;

[0058] When the cold data to hot request is approved, the cold data in the cold data to hot request is defined as hot data, and the cold data is backed up to the production database.

[0059] In this embodiment, the method performs cold processing on business data before the last N months. However, in actual business processing, some business parties with longer account periods may need to reconcile and settle business data before the last N months. Therefore, in order to meet the reconciliation needs of the business parties, the business parties can initiate a cold data hot conversion request for certain cold data. When the administrator approves it, the corresponding cold data will be converted to hot data and synchronized to the production database. For example Figure 3This is the query interface for the business side. The business side selects the cold data that they want to convert to hot data in the query interface, and clicks the Apply button and the Submit button to submit a request to convert the cold data to hot data. Figure 4 This is the administrator's review interface. In the review interface, the administrator selects a cold data for which a cold data hot conversion request is initiated, and clicks the Review button, the Agree button, or the Reject button to agree or reject the cold data hot conversion request. If the administrator agrees to the cold data hot conversion request, enter Figure 5 In the review interface, the administrator selects a cold data that has passed the review, clicks the Execute button and the Confirm button to convert the cold data to hot.

[0060] Furthermore, in some embodiments, after setting the history library, the method further includes:

[0061] When changes are detected in the business data in the production database, a synchronization request containing a data identifier and the changed business data is generated; the data identifier is used to uniquely identify the changed business data;

[0062] When the history database receives a synchronization request and the business data corresponding to the data identifier exists in the history database, the business data in the history database is updated to the changed business data.

[0063] In this embodiment, when the business data in the production library changes, the data is updated synchronously with the historical library to ensure the data uniformity of the production library and the historical library. When the business data in the production library changes, a synchronization request containing the data identifier and the changed business data is generated; after the historical library parses the data identifier, it determines whether there is business data corresponding to the data identifier. If so, the corresponding business data is updated to the changed business data. When this method detects that the production library table structure has changed, it also needs to synchronize with the historical library for update.

[0064] A massive data query system suitable for logistics platforms, see Figure 6 ,include:

[0065] Production database: used to store hot data, including business data of the current month and the previous N months, where N is an integer;

[0066] History database: includes hot partitions and cold partitions. Hot partitions are used to back up some hot data. Cold partitions are used to back up cold data. Cold data refers to business data from the previous N months.

[0067] Query unit: used to query the production library according to the data query request when receiving the data query request from the business party; if the query result returned by the production library is empty, query the hot partitions in the history library according to the data query request; if the query result returned by the history library is not empty, return the query result to the business party; if the query result returned by the production library is not empty, return the query result to the business party.

[0068] Furthermore, in some embodiments, it also includes:

[0069] Data backup unit: used to back up the business data of the previous month in the production database to the hot partition of the historical database when the backup time of the current month arrives, and to back up the business data of the earliest month in the hot partition to the cold partition of the historical database.

[0070] Furthermore, in some embodiments, after backing up the business data of the earliest month in the hot partition to the cold partition of the history library, the data backup unit further includes:

[0071] Delete the data partition corresponding to the earliest month's business data in the hot partition.

[0072] Furthermore, in some embodiments, it also includes:

[0073] Data hot conversion unit: when receiving a cold data hot conversion request from the business party, forward the cold data hot conversion request to the administrator for review; the cold data hot conversion request includes part of the cold data in the cold partition of the historical database; when the cold data hot conversion request is approved, define the cold data in the cold data hot conversion request as hot data, and back up the cold data to the production database.

[0074] Furthermore, in some embodiments, it also includes:

[0075] Synchronization unit: used to generate a synchronization request containing a data identifier and the changed business data when it detects changes in the business data in the production library; the data identifier is used to uniquely identify the changed business data; when the historical library receives the synchronization request and the historical library contains business data corresponding to the data identifier, the business data in the historical library is updated to the changed business data.

[0076] The system provided in the embodiment of the present invention is briefly described. For matters not mentioned in the embodiment part, reference may be made to the corresponding contents in the aforementioned embodiment.

[0077] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein by equivalents. These modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be included in the scope of the claims and specification of the present invention.

Claims

1. A method for querying massive data suitable for a logistics platform, characterized in that: include: Set up a production library; The production library is used to store hot data, which includes business data of the current month and the previous N months, where N is an integer; Setting a history library; the history library includes a hot partition and a cold partition; the hot partition is used to back up some hot data; The cold partition is used to back up cold data; Cold data refers to business data before the previous N months; When receiving a data query request from a business party, querying the production database according to the data query request; If the query result returned by the production database is empty, query the hot partitions in the history database according to the data query request; If the query result returned by the history database is not empty, the query result is returned to the business party; If the query result returned by the production database is not empty, the query result is returned to the business party.

2. The method for querying massive data applicable to a logistics platform according to claim 1, characterized in that: After setting up the history repository, also include: If the backup time of the current month arrives, the business data of the previous month in the production library is backed up to the hot partition of the historical library, and the business data of the earliest month in the hot partition is backed up to the cold partition of the historical library.

3. The method for querying massive data applicable to a logistics platform according to claim 2 is characterized in that: After backing up the earliest month's business data in the hot partition to the cold partition of the history library, the method further includes: Delete the data partition corresponding to the earliest month's business data in the hot partition.

4. The method for querying massive data applicable to a logistics platform according to claim 1, characterized in that: After setting up the history repository, also include: When a cold data hot partition request is received from a business party, the cold data hot partition request is forwarded to an administrator for review; the cold data hot partition request includes part of the cold data in the cold partition of the history library; When the cold data-to-hot request is approved, the cold data in the cold data-to-hot request is defined as hot data, and the cold data is backed up to the production database.

5. The method for querying massive data applicable to a logistics platform according to claim 1, characterized in that: After setting up the history repository, also include: When a change in the business data in the production database is detected, a synchronization request including a data identifier and the changed business data is generated; the data identifier is used to uniquely identify the changed business data; When the history database receives the synchronization request and the business data corresponding to the data identifier exists in the history database, the business data in the history database is updated to the changed business data.

6. A massive data query system suitable for logistics platform, characterized in that: include: Production database: used to store hot data, including business data of the current month and the previous N months, where N is an integer; History library: including hot partition and cold partition; the hot partition is used to back up some hot data; The cold partition is used to back up cold data; Cold data refers to business data before the previous N months; Query unit: configured to query the production database according to the data query request when receiving a data query request from the business party; If the query result returned by the production database is empty, query the hot partitions in the history database according to the data query request; If the query result returned by the history database is not empty, the query result is returned to the business party; If the query result returned by the production database is not empty, the query result is returned to the business party.

7. The massive data query system applicable to the logistics platform according to claim 6 is characterized in that: Also includes: Data backup unit: used to back up the business data of the previous month in the production library to the hot partition of the historical library if the backup time of the current month arrives, and to back up the business data of the earliest month in the hot partition to the cold partition of the historical library.

8. The massive data query system applicable to the logistics platform according to claim 7 is characterized in that: After backing up the business data of the earliest month in the hot partition to the cold partition of the history library, the data backup unit further includes: Delete the data partition corresponding to the earliest month's business data in the hot partition.

9. The massive data query system applicable to the logistics platform according to claim 6 is characterized in that: Also includes: Data hot conversion unit: used for forwarding the cold data hot conversion request to the administrator for review when receiving the cold data hot conversion request from the business party; the cold data hot conversion request includes part of the cold data in the cold partition of the historical library; when the cold data hot conversion request is approved, the cold data in the cold data hot conversion request is defined as hot data, and the cold data is backed up to the production library.

10. The massive data query system applicable to the logistics platform according to claim 6, characterized in that: Also includes: Synchronization unit: used to generate a synchronization request including a data identifier and the changed business data when a change in the business data in the production library is monitored; the data identifier is used to uniquely identify the changed business data; when the historical library receives the synchronization request and the business data corresponding to the data identifier exists in the historical library, the business data in the historical library is updated to the changed business data.

Images