Data query method and device and related product

Abstract

The invention relates to the field of data processing and financial science and technology, and discloses a data query method and device and a related product. The method comprises the following steps: processing a query instruction to obtain range information of to-be-queried data; based on a pre-constructed metadatabase, determining a target storage space and a corresponding storage address according to the range information; if the target storage space comprises a hot storage space or a warm storage space, querying in the hot storage space or the warm storage space by calling a corresponding interface; if the target storage space comprises the cold storage space, migrating the to-be-migrated data into the hot storage space based on the storage address of the to-be-migrated data; and querying the to-be-migrated data in the hot storage space. The problems that in the prior art, the query speed is low, the accuracy is poor, and the storage cost is high are effectively solved, and the data query speed and accuracy are improved, and the storage cost is optimized.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of data processing and financial technology, and in particular, to a data query method and device and related products. BACKGROUND

[0002] In reality, various business scenarios have data query needs. These needs run through work promotion, business operation and other links, and whether they can be responded to efficiently directly affects work efficiency and progress. In existing technologies, first, the storage mode of data is generally to store data in an online database, and to archive data generated a long time ago to low-cost storage on a regular basis (such as every quarter, every year). Correspondingly, when data query is performed, a user generally inputs a query instruction through a client, and a server processes the query instruction and performs data query in the online database to obtain a query result. If some data is not in the online database, manual query is performed in the low-cost storage address.

[0003] For example, in the field of financial technology, a financial personnel of a property insurance company may query vehicle insurance premium income details for 2022-2024 through a financial system client. According to the existing query process, the server first processes the instruction and queries in the online database. The database is archived according to a fixed node at the end of each year, and only stores data for 2023-2024, so the server can return the results. The premium data for 2022 has been archived to a tape library or other low-cost storage, and the server does not find it. Therefore, the financial or operation personnel need to manually locate the corresponding tape library storage address according to the archiving record of “vehicle insurance premium for 2022”, retrieve the data file and extract the details, integrate the manually obtained data for 2022 with the online query result, and finally feed back to the query personnel.

[0004] It can be seen that the data query method of the existing technology at least has the following problems: 1. Data is long-term concentrated in high-performance online storage media, and the huge data will slow down the query and analysis speed of the online database, and also make the overall storage cost high; 2. Since part of the data is archived to low-cost storage, it is often necessary to manually check each storage medium to query the corresponding part of the data, which not only greatly slows down the query response speed, but also is prone to errors due to the complicated operation process.

[0005] In summary, the existing data query method has poor query speed and accuracy. SUMMARY

[0006] In view of the above problems, the embodiments of the present disclosure provide a data query method, device and related products, aiming to solve the technical problem of how to improve the data query speed and accuracy.

[0007] In a first aspect, the embodiments of the present disclosure provide a data query method, comprising: in response to a query instruction sent by a user through a client, processing the query instruction to obtain range information of data to be queried; wherein the range information comprises a table name and a time range; based on a pre-constructed metadata database, determining one or more target storage spaces and corresponding storage addresses according to the range information; wherein the target storage spaces comprise a hot storage space, a warm storage space, and a cold storage space; the metadata database stores distribution information and storage addresses of table data in the hot storage space, the warm storage space, and the cold storage space; if the one or more target storage spaces comprise the hot storage space or the warm storage space, querying in the hot storage space or the warm storage space by calling a corresponding interface to obtain a first query result; if the one or more target storage spaces comprise the cold storage space, migrating data to be migrated to the hot storage space based on a storage address of the data to be migrated; wherein the data to be migrated is part of the data to be queried that is stored in the cold storage space; querying the data to be migrated in the hot storage space to obtain a second query result; sending the first query result and / or the second query result to the client.

[0008] In a second aspect, the embodiments of the present disclosure further provide a data query device, and the device comprises: a processing module configured to, in response to a query instruction sent by a user through a client, process the query instruction to obtain range information of data to be queried; wherein the range information comprises a table name and a time range; an address determination module configured to, based on a pre-constructed metadata database, determine one or more target storage spaces and corresponding storage addresses according to the range information; wherein the target storage spaces comprise a hot storage space, a warm storage space, and a cold storage space; the metadata database stores distribution information and storage addresses of table data in the hot storage space, the warm storage space, and the cold storage space; a first query module configured to, if the one or more target storage spaces comprise the hot storage space or the warm storage space, query in the hot storage space or the warm storage space by calling a corresponding interface to obtain a first query result; a second query module configured to, if the one or more target storage spaces comprise the cold storage space, migrate data to be migrated to the hot storage space based on a storage address of the data to be migrated; wherein the data to be migrated is part of the data to be queried that is stored in the cold storage space; query the data to be migrated in the hot storage space to obtain a second query result; The sending module is configured to send the first query result and / or the second query result to the client.

[0009] In a third aspect, the embodiments of the present disclosure further provide an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, which, when executed, cause the processor to perform the steps of the data query method.

[0010] In a fourth aspect, the embodiments of the present disclosure further provide a computer readable storage medium, which stores one or more programs, which, when executed by an electronic device comprising a plurality of application programs, cause the electronic device to perform the steps of the data query method.

[0011] By means of the above technical solutions, the data query method, device and related products provided by the embodiments of the present disclosure first parse the query instruction to obtain the table name and time range of the data to be queried, and then rely on the pre-constructed metadata database to accurately locate the hot, warm, cold or any or multiple target storage spaces and corresponding addresses where the data is located, thereby avoiding the cumbersome operation of manually checking the storage medium in the prior art, greatly reducing query errors and improving response speed. At the same time, for hot and warm storage spaces, an interface is directly called for query, and for cold storage spaces, the data to be migrated is automatically migrated to a hot storage space for query, which not only reduces the resource occupation of high-performance online storage through hierarchical storage, reduces the overall storage cost, but also replaces manual operation through an automatic migration and query process, further ensuring the query efficiency, and finally effectively solves the problems of slow query speed, poor accuracy and high storage cost in the prior art, and realizes the improvement of data query speed and accuracy and the optimization of storage cost.

[0012] The above description is only a summary of the technical solutions of the present disclosure. In order to enable a clearer understanding of the technical means of the present disclosure, the contents of the specification can be implemented, and in order to enable the above and other purposes, features and advantages of the present disclosure to be more apparent and easy to understand, the specific implementation manner of the present disclosure is described below. BRIEF DESCRIPTION OF DRAWINGS

[0013] The drawings described herein are used to provide further understanding of the present disclosure, and form a part of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions serve to explain the present disclosure, and do not constitute an improper limitation on the present disclosure. In the drawings: Figure 1 An application environment schematic diagram of the data query method provided by the embodiments of the present disclosure is shown; Figure 2 A flow schematic diagram of the data query method provided by the embodiments of the present disclosure is shown; Figure 3 A structure schematic diagram of the data query device provided by the embodiments of the present disclosure is shown; Figure 4 A structural schematic diagram of an electronic device is shown. DETAILED DESCRIPTION

[0014] For the purposes of the present disclosure, technical solutions and advantages, the following will combine specific embodiments of the present disclosure and corresponding drawings to clearly and completely describe the technical solutions of the present disclosure. Obviously, the described embodiments are only a part of the embodiments of the present disclosure, not all. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present disclosure.

[0015] It should be noted that: similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0016] It should be noted that the terms "first", "second", and the like in the specification and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such use can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the term "comprising" and its variants are to be interpreted as meaning "including but not limited to" an open term.

[0017] As introduced before, in the prior art, first, the data storage mode is generally to store data in an online database, and to archive data generated a long time ago to a low-cost storage periodically (such as every quarter, every year); Correspondingly, when performing data query, generally, the user inputs query instructions through the client, and the server processes these query instructions to perform data query in the online database to obtain query results; If some data is not in the online database, manually query the low-cost storage address. It can be seen that the data query method of the prior art at least has the following problems: 1. The data is long-term centralized storage in high-performance online storage medium, and the huge data will slow down the query and analysis speed of the online database, and also make the overall storage cost higher; 2. Because a part of the data is archived to the low-cost storage, it is often necessary to manually check the storage medium one by one to query the corresponding part of the data, which not only greatly slows down the query response speed, but also is easy to make an error query due to the tedious operation process. Based on this, the present application proposes a data query method, device and related products, and the present disclosure will be described in detail through specific embodiments.

[0018] To facilitate understanding of this embodiment, the data query method disclosed in this disclosure will first be described in detail. The data query method provided by this embodiment can be applied to, for example... Figure 1 In this application environment, the client communicates with the server via a network. The server can respond to a query command sent by the client, process the query command, and obtain the range information of the data to be queried; wherein, the range information includes the table name and time range; based on a pre-built metadata database, one or more target storage spaces and corresponding storage addresses are determined according to the range information; wherein, the target storage spaces include hot storage space, warm storage space, and cold storage space; the metadata database stores the distribution information and storage addresses of each table's data in the hot storage space, the warm storage space, and the cold storage space; if the one or more target storage spaces include the hot storage space or the warm storage space, a query is performed in the hot storage space or the warm storage space by calling the corresponding interface to obtain a first query result; if the one or more target storage spaces include the cold storage space, the data to be migrated is migrated to the hot storage space based on the storage address of the data to be migrated; wherein, the data to be migrated is the part of the data to be queried that is stored in the cold storage space; a query is performed on the data to be migrated in the hot storage space to obtain a second query result; the first query result and / or the second query result are sent to the client. This invention first parses the query command to obtain the table name and time range of the data to be queried. Then, relying on a pre-built metadata database, it accurately locates one or more target storage spaces (hot, warm, and cold) and their corresponding addresses where the data is located. This avoids the tedious manual checking of storage media in existing technologies, significantly reducing query errors and improving response speed. Simultaneously, it directly calls the interface to query hot and warm storage spaces, and automatically migrates the data to be migrated to hot storage spaces before querying for cold storage spaces. This not only reduces the resource consumption of high-performance online storage through layered storage, lowering the overall storage cost, but also replaces manual operation with automated migration and query processes, further ensuring query efficiency. Ultimately, it effectively solves the problems of slow query speed, poor accuracy, and high storage cost in existing technologies, achieving improved data query speed and accuracy while optimizing storage costs. The client can be, but is not limited to, various personal computers, laptops, smartphones, tablets, etc. The server can be a standalone server or a server cluster composed of multiple servers. In some possible implementations, this method can be implemented by the processor calling computer-readable instructions stored in memory. The invention will be described in detail below through specific embodiments.

[0019] Figure 2 This illustration shows a flowchart of a data query method provided in an embodiment of the present disclosure. Figure 2It can be seen that the embodiments of this disclosure include at least steps S201-S205: S201: In response to a query command sent by a user through a client, process the query command to obtain the range information of the data to be queried; wherein, the range information includes the table name and the time range; S202: Based on the pre-built metadata database, determine one or more target storage spaces and corresponding storage addresses according to the range information; wherein, the target storage space includes hot storage space, warm storage space and cold storage space; the metadata database stores information on the distribution of data of each table in the hot storage space, the warm storage space and the cold storage space, and storage addresses; S203: If the one or more target storage spaces include the hot storage space or the warm storage space, then by calling the corresponding interface, a query is performed in the hot storage space or the warm storage space to obtain the first query result; S204: If the one or more target storage spaces include the cold storage space, then based on the storage address of the data to be migrated, the data to be migrated is migrated to the hot storage space; wherein, the data to be migrated is the portion of the data to be queried that is stored in the cold storage space; the data to be migrated in the hot storage space is queried to obtain a second query result; S205: Send the first query result and / or the second query result to the client.

[0020] In this embodiment, the metadata database is a pre-built database that centrally stores and manages the metadata of all data. Here, the metadata includes: the distribution of data in each table in the hot storage space, warm storage space, and cold storage space, as well as the specific storage addresses (such as the database address corresponding to the hot storage space, the object storage path corresponding to the warm storage space, and the tape library number corresponding to the cold storage space).

[0021] Hot storage space is a storage tier used for high-frequency access to data and supports high-performance access, such as online production databases; warm storage space is a storage tier used for data with medium access frequency, usually implemented by high-performance object storage or distributed file systems; cold storage space is a tier used for low-frequency access and long-term storage of data, such as low-cost object storage, tape libraries, Blu-ray storage, etc.

[0022] During implementation, the metadata database may also include at least one of the following information: the archiving time of data stored in a certain storage space, the retention period (for example, in the scenario of property insurance financial data query, the retention period of data after entering the cold storage space, which is used to meet the long-term compliance storage requirements of property data, and insurance industry regulations usually require financial and business data to be stored for many years), etc.

[0023] In practice, when a user sends a query command through the client, the execution entity of this method (hereinafter referred to as the system) responds to the command, parses and processes the command, clarifies which table(s) the data to be queried belongs to and the corresponding time interval, thereby obtaining the range information of the data to be queried; based on the pre-built metadata database, the system queries the metadata database to find which storage spaces (hot, warm, cold) the data of the table in the corresponding time interval is distributed in, and their specific storage addresses, thereby determining one or more target storage spaces and their corresponding storage addresses; if the determined target storage space includes hot storage space or warm storage space, the system calls the underlying interfaces corresponding to these two types of storage spaces (SQL statement interaction interface corresponding to hot storage space, API interface corresponding to warm storage space, such as RESTful API of object storage, etc.), performs the query operation in the corresponding storage space, and obtains the first query result; if the target storage space includes cold storage space, the system will migrate the part of the data to be queried in the cold storage space (i.e., the data to be migrated) to the hot storage space; query the data to be migrated that has just been migrated in the hot storage space to obtain the second query result, and finally send the first query result and / or the second query result to the user's client.

[0024] For example, in a property insurance financial data query scenario, the data stored in the hot storage space, the warm storage space, and the cold storage space are property insurance financial data. In implementation, the property insurance company's financial personnel send a query command through the financial system client, requesting a query for the second quarter of 2024 auto insurance premium income and a comparison with the same period in 2023. The backend system responds to the command and processes it, extracting the table name (auto insurance premium income table) and the time range (April-June 2023, April-June 2024). Then, based on the pre-built metadata database, it is found that the data for May-June 2024 is stored in the hot storage space, the data for April 2024 is stored in the warm storage space, and the data for April-June 2023 is stored in the cold storage space. This determines that the target storage space includes three categories: hot, warm, and cold, and their corresponding specific storage addresses. The system then calls the database query interface for the hot storage space and the API interface for the warm storage space to query the auto insurance premium income data for May-June 2024 and April 2024, respectively, obtaining the first query result. Simultaneously, based on the storage address of the same period data from 2023 in the cold storage space, this portion of the data to be migrated is moved to the hot storage space. Finally, the migrated 2023 data is queried in the hot storage space to obtain the second query result. The first query result for 2024 and the second query result for 2023 are then integrated and sent to the financial system client for the finance personnel.

[0025] As can be seen, this embodiment first parses the query command to obtain the table name and query time range of the data to be queried. Then, relying on the pre-built metadata database, it accurately locates one or more target storage spaces and their corresponding addresses in hot, warm, and cold storage. This avoids the tedious manual inspection of storage media in the prior art, significantly reducing query errors and improving response speed. At the same time, it directly calls the interface to query hot and warm storage spaces, and automatically migrates the data to be migrated to the hot storage space before querying for cold storage spaces. This not only reduces the resource consumption of high-performance online storage through layered storage and lowers the overall storage cost, but also replaces manual operation with automated migration and query processes, further ensuring query efficiency. Ultimately, it effectively solves the problems of slow query speed, poor accuracy, and high storage cost in the prior art, and achieves improved data query speed and accuracy as well as optimized storage costs.

[0026] Furthermore, in order to better illustrate the process of the above data query method, as a refinement and extension of the above embodiments, the present invention provides several embodiments, but is not limited thereto, as follows.

[0027] In some embodiments, migrating the data to be migrated to the hot storage space based on the storage address of the data to be migrated includes: Send the data to be migrated from the storage address of the data to be migrated to the terminal where the hot storage space is located; The integrity of the data to be migrated on the terminal where the hot storage space is located is verified, and the verification result is obtained. If the verification result indicates that the data is complete, the data format of the data to be migrated will be adapted to a type compatible with the hot storage space, and then imported into the hot storage space.

[0028] In this embodiment, in order to migrate the data to be migrated to the hot storage space, the storage address of the data to be migrated can be determined first. For example, the data to be migrated can be determined to be stored in a specific tape of a tape library or in a cold storage bucket of a cloud platform. Then, a dedicated transmission tool (such as Hadoop's DistCp, i.e., a distributed copy tool) or a self-developed program can be used to send the data to be migrated to the terminal where the hot storage space is located.

[0029] After the data transmission is completed, in order to avoid data loss or damage during the transmission process, the integrity of the data to be migrated on the terminal where the hot storage space is located can be verified by checking the data size, calculating and verifying the hash value of the data (such as MD5, SHA256), etc., to confirm that the data is not lost or damaged, and then obtain the verification result.

[0030] If the verification result indicates that the data is complete, the compatible data formats supported by the hot storage space (such as database structured formats, specific file formats, etc.) are obtained. The data to be migrated is converted from the existing format to the compatible type using a format conversion tool or program. Then, the converted and adapted data is imported into the hot storage space in batches or incrementally through the import interface corresponding to the hot storage space (such as the database import interface), thus completing the migration of the data to be migrated.

[0031] This embodiment standardizes the data migration process from cold storage to hot storage. First, it completes data transmission and integrity verification to ensure that the migrated data is not lost or damaged. Then, it uses format adaptation to ensure that the data is compatible with hot storage. This not only improves the reliability and accuracy of data migration, but also allows the migrated data to quickly adapt to the access requirements of hot storage, ensuring efficient and smooth subsequent query operations.

[0032] In some embodiments, the method further includes: Acquire the data generation time and historical access data of the data to be processed; calculate the access probability prediction data within a certain period in the future based on the historical access data; Based on the data generation time and the access probability prediction data, the data to be processed is stored in the hot storage space, the warm storage space, or the cold storage space; The data generation time of the data to be processed, and the storage address in the hot storage space, the warm storage space, or the cold storage space, are recorded in the metadata database.

[0033] In this embodiment, the data generation time of the data to be processed is obtained from the data source system (such as the source system of property insurance financial data, including the business underwriting system, core financial system, etc.), and is the timestamp automatically recorded by the system when the data is generated (such as the policy creation time, claims payment time, etc.). Historical access data refers to the record data of historical queries and update operations corresponding to the data to be processed within a past period (such as the most recent 30 days or 90 days), including information such as the number of accesses and the access time. Historical access data can also be stored in a metadata database.

[0034] In implementation, for the step of calculating the predicted access probability data for a certain period of time based on the historical access data of the data to be processed, for example, the number of visits in the historical access data can be extracted first, and then a simple statistical method can be used, such as counting the number of visits in the most recent N days (e.g., 30 days), and the predicted access probability data can be determined based on the number of visits (e.g., the access probability in the most recent 30 days is 80%). Alternatively, machine learning models (such as logistic regression and random forest) can be used to train the model by using historical access frequency, access time distribution, and the true value of future access probability as features and labels, and then inputting relevant data to obtain the predicted access probability data for a certain period of time.

[0035] Finally, based on the data generation time and the access probability prediction data, the property insurance financial data to be stored is stored in the hot storage space, the warm storage space, or the cold storage space; and the data generation time of the data to be processed, as well as the storage address in the hot storage space, the warm storage space, or the cold storage space, is recorded in the metadata database so that when a user queries, the system can quickly locate one or more target storage spaces and their corresponding storage addresses based on the data distribution information in the metadata database.

[0036] Specifically, in some embodiments, the metadata database further includes set hot storage time thresholds, cold storage time thresholds, hot storage access probability thresholds, and cold storage access probability thresholds; the step of storing the data to be processed in the hot storage space, the warm storage space, or the cold storage space based on the data generation time and the access probability prediction data includes: If the data generation time is less than the hot storage time threshold and the access probability prediction data is greater than the hot storage access probability threshold, then the data to be processed is stored in the hot storage space. If the data generation time is greater than or equal to the hot storage time threshold and less than the cold storage time threshold, or if the data generation time is less than the hot storage time threshold but the access probability prediction data is less than or equal to the hot storage access probability threshold, then the data to be processed is stored in the warm storage space. If the data generation time is greater than or equal to the cold storage time threshold, or the access probability prediction data is less than or equal to the cold storage access probability threshold, then the data to be processed is stored in the cold storage space.

[0037] In this embodiment, the hot storage time threshold is a time limit set in the metadata database to determine whether data is relatively new and may require frequent access, for example, it can be set to 30 days; the cold storage time threshold is a time limit to determine whether data is relatively old and may be accessed less frequently, for example, it can be set to 365 days, that is, data generated more than 365 days ago may meet the time conditions for cold storage; the hot storage access probability threshold is a probability limit to determine whether the probability of future access to data is high enough to be stored in hot storage, for example, it can be set to 80%; the cold storage access probability threshold is a probability limit to determine whether the probability of future access to data is low enough to be stored in cold storage, for example, it can be set to 20%, that is, data with a future access probability of less than 20% may enter cold storage.

[0038] For example, the hot storage time threshold is set to 30 days, the cold storage time threshold is set to 365 days, the hot storage access probability threshold is set to 80%, and the cold storage access probability threshold is set to 20%. If a certain property insurance financial data to be stored was generated 20 days ago (less than 30 days) and has a future access probability of 85% (greater than 80%), it will be stored in the hot storage space. If a certain data was generated 60 days ago (greater than 30 days and less than 365 days), or was generated 20 days ago but has a future access probability of 70% (less than 80%), it will be stored in the warm storage space. If a certain data was generated 400 days ago (greater than 365 days), or was generated 100 days ago but has a future access probability of 15% (less than 20%), it will be stored in the cold storage space.

[0039] This embodiment combines the data generation time of the data to be processed with the predicted future access probability data calculated based on historical access patterns to implement hot, warm, and cold tiered storage of the data. This allows frequently accessed recent data to be stored in high-performance hot storage to ensure efficient response of core operations; moderately accessed or short-term low-access probability data to be placed in warm storage to balance efficiency and cost; and long-term or extremely low-access probability data to be assigned to low-cost cold storage to significantly reduce overall storage overhead. Furthermore, standardized threshold judgment rules are used to automate and standardize storage allocation, reduce human intervention errors, ensure that data storage complies with compliance requirements, and allow financial data of different values ​​to obtain appropriate storage resources. Ultimately, this achieves reasonable allocation of storage resources, efficient and low-cost data storage, and refined management to facilitate subsequent data query processes.

[0040] In some embodiments, storing the data to be processed in the cold storage space includes: The data to be processed is processed using a preset compression algorithm and / or encryption algorithm to obtain processed data to be stored. The processed data to be stored is stored in the cold storage space; and according to the compression algorithm and / or encryption algorithm, the compression identifier data and / or encryption identifier data of the data to be stored are recorded in the source database; Before the step of adapting the data format of the processed data to a type compatible with the hot storage space and importing it into the hot storage space, the method further includes: Based on the compressed identifier data and / or encoded identifier data corresponding to the data to be migrated, determine whether the data to be migrated has been compressed or encoded; if the data to be migrated has been compressed or encoded, then decompress or decode the data to be migrated to obtain the processed data. The processed data is then used as the new data to be migrated.

[0041] In this embodiment, the preset compression algorithm can be, for example, Snappy compression, Gzip compression, LZ4 compression, BZIP2 compression, etc., and this embodiment is not limited to this. The encryption algorithm can be, for example, AES-128 (Advanced Encryption Standard 128-bit), AES-256 (Advanced Encryption Standard 256-bit), the Chinese national cryptographic algorithm SM4, asymmetric encryption algorithms (such as RSA asymmetric encryption algorithm), etc., and this embodiment is not limited to this.

[0042] Compression identification data should include at least: a status indicator of whether the data has been compressed (such as a Boolean value "yes / no"), the name of the compression algorithm used, key parameters in the compression process (such as compression level and data block size), and the data size before and after compression (used to verify the compression effect).

[0043] The encryption identification data includes at least: a status indicator of whether the data has been encrypted (such as a Boolean value "yes / no"), the name of the encryption algorithm used, and key association information related to decryption (such as the unique ID of the key in the key management system, without directly storing the key itself). If some fields are encrypted, it will also include an encryption range indicator (such as "sensitive field encryption" or "entire file encryption").

[0044] Regarding the step of recording compressed and / or encrypted identifier data of the data to be stored in the metadata database, in specific implementation, after the data to be processed is processed by a preset compression algorithm and / or encryption algorithm, the corresponding identifier information (such as compression algorithm "Gzip" and encryption key ID "KMS-20240601-003") can be extracted from the processing flow. This identifier information can be associated and matched with the unique identifier of the data to be stored, and then the associated identifier data can be entered into the corresponding field of the metadata database table through database write operations (such as SQL INSERT or UPDATE statements).

[0045] Based on this, before adapting the data format of the processed data to a type compatible with the hot storage space and importing it into the hot storage space, the processing status can be determined based on the compression identifier data and / or encoding identifier data corresponding to the data to be migrated, and decompression or decoding can be performed. Specifically, the compression identifier data and encryption identifier data corresponding to the data to be migrated can be queried from the metadata database first; then, the status fields of "whether compressed" and "whether encrypted" in the identifier data are checked to determine whether the data has been processed accordingly; if the status is compressed, the compression algorithm and parameters used are obtained from the compression identifier data, and the corresponding decompression tool or function is called to decompress and restore it to the uncompressed format; if the status is encrypted, the encryption algorithm and key association information (such as key ID) are obtained from the encryption identifier data, and after obtaining the corresponding key through the key management system, the matching decryption tool or function is called to decrypt and obtain the original data before encryption; after decompression or decryption is completed, the original format data is used as the new data to be migrated.

[0046] This embodiment processes the data to be stored using a preset compression algorithm, which reduces the space occupied by cold storage and lowers the cost of cold storage. Encryption algorithms ensure the security of the data during the cold storage phase, meeting data confidentiality and compliance requirements. Recording compressed and / or encrypted identification data in the metadata database provides a clear processing basis for subsequent data migration. Decompression or decoding is performed based on the identification data before migration, ensuring that the data format is compatible with the hot storage space and guaranteeing normal use after importing into hot storage. Overall, this achieves efficient, low-cost, and secure cold storage while seamlessly connecting the data migration process from cold to hot storage.

[0047] In some embodiments, the method further includes: Obtain the historical data growth rate of univariate data and the historical data growth rate of multivariate data; among them, univariate data and multivariate data belong to the preset core data; The historical data growth rate of the univariate data is input into a pre-trained univariate linear regression model to obtain the predicted data of the first data growth rate within a preset future period. The historical data growth rate of the multivariate data is input into a pre-trained multiple linear regression model to obtain the predicted data of the second data growth rate within a future preset period. Based on the first data volume growth prediction data, the second data volume growth prediction data, and the remaining storage space size of the hot storage space and the warm storage space, the data in the hot storage space and the warm storage space are migrated to the cold storage space.

[0048] In this embodiment, the decision to migrate data is also determined by predicting the data growth rate. Research has shown that in real-world applications, the growth of some data is driven linearly by a single factor, such as the growth of "electronic office expense reimbursement vouchers." This growth is linearly related to only one independent variable: "the number of reimbursement vouchers generated each month" (the size of a single voucher is fixed; the more vouchers generated, the more stable the data growth). There are no other key influencing factors. Therefore, training a model using historical data with a single variable of "time - number of vouchers generated" can accurately predict future growth. The growth of other data is driven linearly by multiple factors, such as the "total growth of core financial vouchers." This growth is simultaneously affected by three independent variables: "the number of premium payment vouchers generated," "the number of claims payment vouchers generated," and "the number of commission settlement vouchers generated" (the increase in the number of each voucher type contributes to the total data growth, and each is linearly correlated with the total growth). A multivariate model is needed to learn the contribution weight of each independent variable in order to accurately predict the total growth.

[0049] Based on this, in this embodiment, the historical data growth of univariate data is the storage growth value of a single core data in the past period, and the historical data growth of multivariate data is the set of storage growth values ​​of multiple related core data in the past period. The first data growth prediction data is the storage growth prediction value of the single core data in the future preset period obtained by inputting the historical data growth of univariate data into a pre-trained univariate linear regression model. The second data growth prediction data is the growth prediction value of the total core data storage corresponding to multiple related core data in the future preset period obtained by inputting the historical data growth of multivariate data into a pre-trained multiple linear regression model.

[0050] For example, a property insurance company presupposes standardized electronic office expense reimbursement vouchers as a single core financial data point. The growth in this data volume is linearly related only to the monthly voucher generation quantity. The historical data volume growth for this type of data from January to December 2023 was 2GB, 2.1GB, 1.9GB, 2GB, 2.2GB, 2GB, 2.1GB, 1.9GB, 2GB, 2.1GB, 2GB, and 2.2GB, respectively. Inputting this data into a pre-trained univariate linear regression model yields a predicted first-order data volume growth of 2.05GB for the electronic office expense reimbursement vouchers in January 2024. The company also presupposes multiple related core financial data points: premium payment vouchers, claims payment vouchers, and commission settlement vouchers. The growth in the storage volume of these three types of data jointly drives the growth in the total core data storage volume, and all three are linearly correlated. The historical data volume growth for these three types of data from January to December 2023 was 3GB, 2.5GB, and 2.5GB, respectively. The data sets are: (1.5GB), (3.2GB, 2.6GB, 1.6GB), (3.1GB, 2.4GB, 1.4GB), (3.3GB, 2.7GB, 1.5GB), (3.2GB, 2.5GB, 1.6GB), (3.4GB, 2.8GB, 1.7GB), (3.3GB, 2.6GB, 1.6GB), (3.5GB, 2.7GB, 1.7GB), (3.4GB, 2.8GB, 1.6GB), (3.6GB, 2.9GB, 1.8GB), (3.5GB, 2.7GB, 1.7GB), (3.7GB, 3GB, 1.9GB). These 12 sets of data are input into a pre-trained multiple linear regression model. The model automatically learns the contribution weights of the growth of the three types of data storage volume to the growth of the total voucher data volume, and finally obtains the predicted data of the second data volume growth of the core total voucher data storage volume in January 2024 as 8.3GB.

[0051] The data involved in this embodiment are all core data. For example, core data refers to core financial data, specifically such as insurance policies, premiums, claims, commissions, and reinsurance recovery. The volume of this core data grows steadily with business, exhibiting a linear growth characteristic. During implementation, the type of core data can be selected according to actual needs; this embodiment does not impose any limitations on this.

[0052] In some embodiments, before inputting the historical data growth rate of the univariate data into a pre-trained univariate linear regression model to obtain the predicted data of the first data growth rate within a preset future period; and before inputting the historical data growth rate of the multivariate data into a pre-trained multiple linear regression model to obtain the predicted data of the second data growth rate within a preset future period, the method further includes: Obtain a first dataset and a second dataset; wherein, the sample characteristics of the samples in the first dataset are the historical data growth rate of univariate data, and the sample labels are the future data growth rate of univariate data; the sample characteristics of the samples in the second dataset are the historical data growth rate of multivariate data, and the sample labels are the future data growth rate of multivariate data; wherein, the univariate data and multivariate data belong to the core data. The first dataset is fitted to obtain the univariate linear regression model; The second dataset is fitted to obtain the multiple linear regression model.

[0053] In this embodiment, the features and labels of the samples in the first dataset and the second dataset are similar to those of the aforementioned historical data growth of univariate data, historical data growth of multivariate data, predicted data of the first data growth, and predicted data of the second data growth, and will not be repeated here.

[0054] When fitting the first dataset, we can use the historical data growth rate of the univariate data as the independent variable and the future data growth rate as the dependent variable. We can use the least squares method to solve the coefficients a and constant term b of the univariate linear regression equation (y=ax+b) to minimize the sum of squared errors between the predicted value and the actual label, and thus obtain the univariate linear regression model.

[0055] When fitting the second dataset, the historical data growth of multiple related indicators can be used as independent variables and the future data growth of the target indicator can be used as dependent variables. Similarly, the coefficients and constants of the multiple linear regression equation (y=a1x1+a2x2+…+anxn+b) can be solved by the least squares method, and the sum of squared prediction errors can be minimized to obtain the multiple linear regression model.

[0056] Understandably, since the preset core data includes multiple types of univariate data (such as office rent, fixed salary expenses, etc.) and multiple types of multivariate data (such as sales profit, monthly funding gap, etc.), and the historical data growth patterns of different types of univariate data are different, as are the correlation indicators and data growth logic of different types of multivariate data, it is necessary to obtain a dedicated univariate linear regression model by fitting the corresponding first dataset for each type of univariate core data, and a dedicated multiple linear regression model by fitting the corresponding second dataset for each type of multivariate core data. This results in multiple univariate and multiple linear regression models to adapt to the data growth prediction needs of different types of core data.

[0057] Finally, when migrating data, first calculate the current remaining capacity of hot and warm storage spaces to determine the upper limit of new data that can be accommodated. Then, combine univariate and multivariate forecast data to calculate the total scale of new data in the future preset period. Compare the new data scale with the sum of the remaining capacity of hot and warm storage to determine whether migration is necessary and the total amount to be migrated. If migration is necessary, the hot storage space can be scanned periodically (e.g., during the early morning off-peak period each day). Data whose generation time is greater than or equal to the hot storage time threshold and whose future access probability is lower than the hot storage access probability threshold can be migrated to the warm storage space. The warm storage space can be scanned periodically, and data whose generation time is greater than or equal to the cold storage time threshold can be migrated to the cold storage space until the remaining capacity of hot and warm storage can meet the future new data storage needs.

[0058] In addition, during the actual implementation, data can be migrated in small batches and incrementally to avoid impacting the performance of online systems. Before the migration batch begins, relevant data snapshots can be locked or transaction logs can be used to ensure the consistency of the migrated data.

[0059] This embodiment predicts the future data volume growth of univariate and multivariate data based on preset core data using adapted pre-trained regression models. By combining the remaining capacity of hot and warm storage space with data migration planning, the prediction results can not only match the growth characteristics of different types of data and ensure prediction accuracy, but also anticipate storage needs in advance and rationally migrate data that does not require high-frequency access from hot and warm storage to cold storage, optimize storage resource allocation, avoid hot and warm storage resource shortages, reduce overall storage costs, and improve the access efficiency of core data. This achieves intelligent and efficient data storage management and ensures stable system operation.

[0060] Those skilled in the art will understand that in the above-described method of the specific embodiments, the order in which the steps are written does not imply a strict execution order, but constitutes no limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0061] It should be noted that in practical applications, all the above possible implementation methods can be arbitrarily combined to form possible embodiments of this disclosure, which will not be elaborated upon here. The information (including but not limited to device information, user information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the code snippets in the above steps are only examples illustrating the specific implementation process of page navigation and are not intended to limit this embodiment. The software tools or components appearing in the embodiments of this disclosure are merely illustrative and do not represent actual use.

[0062] Based on the same concept, this disclosure also provides a data query device, which corresponds one-to-one with the data query method in the above embodiments. Figure 3 A schematic diagram of the structure of the data query device provided in an embodiment of this disclosure is shown. See also: Figure 3 As shown, the data query device 300 provided in this embodiment includes: The processing module 301 is used to respond to a query command sent by a user through a client, process the query command, and obtain the range information of the data to be queried; wherein, the range information includes the table name and the time range; Address determination module 302 is used to determine one or more target storage spaces and corresponding storage addresses based on a pre-built metadata database and the range information; wherein, the target storage space includes hot storage space, warm storage space and cold storage space; the metadata database stores information on the distribution of data of each table in the hot storage space, the warm storage space and the cold storage space; The first query module 303 is used to query the hot storage space or the warm storage space by calling the corresponding interface if the one or more target storage spaces include the hot storage space or the warm storage space, and obtain the first query result. The second query module 304 is used to migrate the data to be migrated to the hot storage space based on the storage address of the data to be migrated if the one or more target storage spaces include the cold storage space; wherein the data to be migrated is the part of the data to be queried that is stored in the cold storage space; and to query the data to be migrated in the hot storage space to obtain a second query result. The sending module 305 is used to send the first query result and / or the second query result to the client.

[0063] In some embodiments, in the above-described apparatus, the second query module 304, when migrating the data to be migrated to the hot storage space based on the storage address of the data to be migrated, is used to: Send the data to be migrated from the storage address of the data to be migrated to the terminal where the hot storage space is located; The integrity of the data to be migrated on the terminal where the hot storage space is located is verified, and the verification result is obtained. If the verification result indicates that the data is complete, the data format of the data to be migrated will be adapted to a type compatible with the hot storage space, and then imported into the hot storage space.

[0064] In some embodiments, the device further includes a storage module for: Obtain the data generation time and historical access data of the data to be processed; Based on the historical access data, calculate the predicted access probability data for a certain period in the future; Based on the data generation time and the access probability prediction data, the data to be processed is stored in the hot storage space, the warm storage space, or the cold storage space; The data generation time of the data to be processed, and the storage address in the hot storage space, the warm storage space, or the cold storage space, are recorded in the metadata database.

[0065] In some embodiments, the metadata database also includes set hot storage time threshold, cold storage time threshold, hot storage access probability threshold, and cold storage access probability threshold; When the storage module stores the data to be processed in the hot storage space, the warm storage space, or the cold storage space based on the data generation time and the access probability prediction data, it is used for: If the data generation time is less than the hot storage time threshold and the access probability prediction data is greater than the hot storage access probability threshold, then the data to be processed is stored in the hot storage space. If the data generation time is greater than or equal to the hot storage time threshold and less than the cold storage time threshold, or if the data generation time is less than the hot storage time threshold but the access probability prediction data is less than or equal to the hot storage access probability threshold, then the data to be processed is stored in the warm storage space. If the data generation time is greater than or equal to the cold storage time threshold, or the access probability prediction data is less than or equal to the cold storage access probability threshold, then the data to be processed is stored in the cold storage space.

[0066] In some embodiments, in the above-described apparatus, the storage module, when storing the data to be processed in the cold storage space, is used to: The data to be processed is processed using a preset compression algorithm and / or encryption algorithm to obtain processed data to be stored. The processed data to be stored is stored in the cold storage space; and according to the compression algorithm and / or encryption algorithm, the compression identifier data and / or encryption identifier data of the data to be stored are recorded in the source database; The device further includes a decompression / decryption module, which, before the step of adapting the data format of the processed data to a type compatible with the hot storage space and importing it into the hot storage space, is used for: Based on the compressed identifier data and / or encoded identifier data corresponding to the data to be migrated, determine whether the data to be migrated has been compressed or encoded; if the data to be migrated has been compressed or encoded, then decompress or decode the data to be migrated to obtain the processed data. The processed data is then used as the new data to be migrated.

[0067] In some embodiments, the apparatus further includes a migration module for: Obtain the historical data growth rate of univariate data and the historical data growth rate of multivariate data; among them, univariate data and multivariate data belong to the preset core data; The historical data growth rate of the univariate data is input into a pre-trained univariate linear regression model to obtain the predicted data of the first data growth rate within a preset future period. The historical data growth rate of the multivariate data is input into a pre-trained multiple linear regression model to obtain the predicted data of the second data growth rate within a future preset period. Based on the first data volume growth prediction data, the second data volume growth prediction data, and the remaining storage space size of the hot storage space and the warm storage space, the data in the hot storage space and the warm storage space are migrated to the cold storage space.

[0068] In some embodiments, the apparatus further includes a fitting module for: Obtain a first dataset and a second dataset; wherein, the sample characteristics of the samples in the first dataset are the historical data growth rate of univariate data, and the sample labels are the future data growth rate of univariate data; the sample characteristics of the samples in the second dataset are the historical data growth rate of multivariate data, and the sample labels are the future data growth rate of multivariate data; wherein, the univariate data and multivariate data belong to the core data. The first dataset is fitted to obtain the univariate linear regression model; The second dataset is fitted to obtain the multiple linear regression model.

[0069] This invention provides a data query device that first parses the query command to obtain the table name and time range of the data to be queried. Then, relying on a pre-built metadata database, it accurately locates one or more target storage spaces (hot, warm, and cold) and their corresponding addresses where the data is located. This avoids the tedious manual inspection of storage media in existing technologies, significantly reducing query errors and improving response speed. Simultaneously, it directly calls the interface to query hot and warm storage spaces, and automatically migrates the data to be migrated to the hot storage space before querying for cold storage spaces. This not only reduces the resource consumption of high-performance online storage through layered storage, lowering the overall storage cost, but also replaces manual operation with an automated migration and query process, further ensuring query efficiency. Ultimately, it effectively solves the problems of slow query speed, poor accuracy, and high storage cost in existing technologies, achieving improved data query speed and accuracy while optimizing storage costs.

[0070] For specific limitations regarding the data query device, please refer to the limitations on the data query method above, which will not be repeated here. Each module in the aforementioned data query device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0071] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure is shown. Figure 4 As shown, at the hardware level, this electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or it may include non-volatile memory, such as at least one disk drive. Of course, this electronic device may also include other hardware required for other business operations.

[0072] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 4 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.

[0073] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.

[0074] The processor reads the corresponding computer program from non-volatile memory into main memory and then runs it, forming a data query device at the logical level. The processor executes the program stored in memory and specifically performs the aforementioned methods.

[0075] The processor may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above methods can be completed by integrated logic circuits in the processor's hardware or by software instructions. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this disclosure. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this disclosure can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in the memory, and the processor reads the information in the memory and, in conjunction with its hardware, completes the steps of the above method.

[0076] The electronic device can execute the data query methods provided in several embodiments of this disclosure and be implemented as a data query device. Figure 3 The functions of the embodiments shown are not described in detail here.

[0077] This disclosure also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform the data query methods provided in various embodiments of this disclosure.

[0078] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0079] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0080] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0081] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0082] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0083] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0084] Computer-readable media include both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0085] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0086] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0087] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A data query method, characterized in that, The method includes: In response to a query command sent by a user through a client, the query command is processed to obtain the range information of the data to be queried; wherein, the range information includes the table name and the time range; Based on the pre-built metadata database, one or more target storage spaces and corresponding storage addresses are determined according to the range information; wherein, the target storage space includes hot storage space, warm storage space and cold storage space; the metadata database stores information on the distribution of data of each table in the hot storage space, the warm storage space and the cold storage space, and their storage addresses; If the one or more target storage spaces include the hot storage space or the warm storage space, then by calling the corresponding interface, a query is performed in the hot storage space or the warm storage space to obtain the first query result; If the one or more target storage spaces include the cold storage space, then based on the storage address of the data to be migrated, the data to be migrated is migrated to the hot storage space; wherein, the data to be migrated is the portion of the data to be queried that is stored in the cold storage space; the data to be migrated in the hot storage space is queried to obtain a second query result; The first query result and / or the second query result are sent to the client.

2. The method according to claim 1, characterized in that, The step of migrating the data to be migrated to the hot storage space based on the storage address of the data to be migrated includes: Send the data to be migrated from the storage address of the data to be migrated to the terminal where the hot storage space is located; The integrity of the data to be migrated on the terminal where the hot storage space is located is verified, and the verification result is obtained. If the verification result indicates that the data is complete, the data format of the data to be migrated will be adapted to a type compatible with the hot storage space, and then imported into the hot storage space.

3. The method according to claim 2, characterized in that, The method further includes: Obtain the data generation time and historical access data of the data to be processed; Based on the historical access data, calculate the predicted access probability data for a certain period in the future; Based on the data generation time and the access probability prediction data, the data to be processed is stored in the hot storage space, the warm storage space, or the cold storage space; The data generation time of the data to be processed, and the storage address in the hot storage space, the warm storage space, or the cold storage space, are recorded in the metadata database.

4. The method according to claim 3, characterized in that, The metadata database also includes set hot storage time thresholds, cold storage time thresholds, hot storage access probability thresholds, and cold storage access probability thresholds; the step of storing the data to be processed in the hot storage space, the warm storage space, or the cold storage space based on the data generation time and the access probability prediction data includes: If the data generation time is less than the hot storage time threshold and the access probability prediction data is greater than the hot storage access probability threshold, then the data to be processed is stored in the hot storage space. If the data generation time is greater than or equal to the hot storage time threshold and less than the cold storage time threshold, or if the data generation time is less than the hot storage time threshold but the access probability prediction data is less than or equal to the hot storage access probability threshold, then the data to be processed is stored in the warm storage space. If the data generation time is greater than or equal to the cold storage time threshold, or the access probability prediction data is less than or equal to the cold storage access probability threshold, then the data to be processed is stored in the cold storage space.

5. The method according to claim 4, characterized in that, The step of storing the data to be processed in the cold storage space includes: The data to be processed is processed using a preset compression algorithm and / or encryption algorithm to obtain processed data to be stored. The processed data to be stored is stored in the cold storage space; and according to the compression algorithm and / or encryption algorithm, the compression identifier data and / or encryption identifier data of the data to be stored are recorded in the source database; Before the step of adapting the data format of the processed data to a type compatible with the hot storage space and importing it into the hot storage space, the method further includes: Based on the compressed identifier data and / or encoded identifier data corresponding to the data to be migrated, determine whether the data to be migrated has been compressed or encoded; if the data to be migrated has been compressed or encoded, then decompress or decode the data to be migrated to obtain the processed data. The processed data is then used as the new data to be migrated.

6. The method according to any one of claims 2-5, characterized in that, The method further includes: Obtain the historical data growth rate of univariate data and the historical data growth rate of multivariate data; among them, univariate data and multivariate data belong to the preset core data; The historical data growth rate of the univariate data is input into a pre-trained univariate linear regression model to obtain the predicted data of the first data growth rate within a preset future period. The historical data growth rate of the multivariate data is input into a pre-trained multiple linear regression model to obtain the predicted data of the second data growth rate within a future preset period. Based on the first data volume growth prediction data, the second data volume growth prediction data, and the remaining storage space size of the hot storage space and the warm storage space, the data in the hot storage space and the warm storage space are migrated to the cold storage space.

7. The method according to claim 6, characterized in that, Before inputting the historical data growth rate of the univariate data into a pre-trained univariate linear regression model to obtain the predicted data for the first data growth rate within a preset future period; and before inputting the historical data growth rate of the multivariate data into a pre-trained multiple linear regression model to obtain the predicted data for the second data growth rate within a preset future period, the method further includes: Obtain a first dataset and a second dataset; wherein, the sample characteristics of the samples in the first dataset are the historical data growth rate of univariate data, and the sample labels are the future data growth rate of univariate data; the sample characteristics of the samples in the second dataset are the historical data growth rate of multivariate data, and the sample labels are the future data growth rate of multivariate data; wherein, the univariate data and multivariate data belong to the core data. The first dataset is fitted to obtain the univariate linear regression model; The second dataset is fitted to obtain the multiple linear regression model.

8. A data query device, characterized in that, The device includes: The processing module is used to respond to a query command sent by a user through a client, process the query command, and obtain the range information of the data to be queried; wherein, the range information includes the table name and the time range; The address determination module is used to determine one or more target storage spaces and their corresponding storage addresses based on a pre-built metadata database and the range information; wherein, the target storage spaces include hot storage spaces, warm storage spaces, and cold storage spaces; the metadata database stores information on the distribution of data in each table in the hot storage spaces, the warm storage spaces, and the cold storage spaces, as well as their storage addresses; The first query module is used to query the hot storage space or the warm storage space by calling the corresponding interface if the one or more target storage spaces include the hot storage space or the warm storage space, and obtain the first query result. The second query module is used to migrate the data to be migrated to the hot storage space based on the storage address of the data to be migrated if the one or more target storage spaces include the cold storage space; wherein the data to be migrated is the portion of the data to be queried that is stored in the cold storage space; and to query the data to be migrated in the hot storage space to obtain a second query result. The sending module is used to send the first query result and / or the second query result to the client.

9. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, characterized in that, when executed, the executable instructions cause the processor to perform the steps of the data query method as described in any one of claims 1-7.

10. A computer-readable storage medium storing one or more programs, characterized in that, When the one or more programs are executed by an electronic device including multiple applications, the electronic device performs the steps of the data query method as described in any one of claims 1-7.

Images