Data storage method and system, electronic equipment and storage medium

Abstract

The invention discloses a data storage method and system, electronic equipment and a storage medium. The method comprises the following steps: acquiring business data which needs to be stored currently and business characteristics of the business data; based on the service features, a data storage directory and a storage block sequence number are constructed, the data storage directory is used for representing a directory used when the service data is stored, and the storage block sequence number is used for representing identification information of a storage data block to be stored of the service data; based on the data storage directory and the storage block serial number, storing the service data into a storage data block; and in response to successful storage of the business data in the storage data block, compressing the storage data block to obtain a compressed data block. According to the method and the device, the technical problem that the effect of storing data with relatively large data volume is relatively poor in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of civil aviation services, and in particular to a data storage method and system, electronic equipment, and a storage medium. Background Art

[0002] Driven by the rapid development of the global civil aviation industry and the trend of economic globalization, the number of airlines has increased dramatically, and with it a denser route network and a large user base. This has not only promoted competition among airlines, but also spawned a demand for faster and more accurate query of flight fare information. Especially in recent years, with the expansion of the online travel market, fare search services have become one of the core functions of major airlines, online travel platforms and flight price comparison websites. These services need to process a large amount of fare search data, including flight time, fare, flight status and other information, in order to provide real-time and accurate fare quotes to users.

[0003] However, as the frequency and amount of freight rate searches continue to grow, storing these business data poses a severe challenge to traditional databases and file systems. Traditional storage methods have a low effective utilization rate of storage space, resulting in excessively high storage costs. In addition, facing massive data query requests, storing data based on traditional storage methods will lead to low efficiency of data query, resulting in long response times. In summary, traditional data storage methods have poor storage effects on large amounts of data, which in turn leads to poor user experience.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] Embodiments of the present invention provide a data storage method and system, an electronic device, and a storage medium, so as to at least solve the technical problem in the related art that the storage effect of a large amount of data is poor.

[0006] According to one aspect of an embodiment of the present invention, a data storage method is provided, including: obtaining business data that currently needs to be stored, and business characteristics of the business data; based on the business characteristics, constructing a data storage directory and a storage block serial number, wherein the data storage directory is used to represent the directory used when storing business data, and the storage block serial number is used to represent identification information of a storage data block where the business data is to be stored; based on the data storage directory and the storage block serial number, storing the business data in a storage data block; in response to the business data being successfully stored in the storage data block, compressing the storage data block to obtain a compressed data block.

[0007] Furthermore, based on the business features, a data storage directory is constructed, including: obtaining feature priorities corresponding to the business features; and constructing the data storage directory based on the business features and the feature priorities.

[0008] Furthermore, the business characteristics include: a time offset of the business data and a data volume of the business data, and constructing a storage block serial number based on the business characteristics includes: obtaining a data block size of a storage data block and a data storage volume per second of the storage data block; obtaining a first value based on the product of the data block size and a preset value, and obtaining a second value based on the product of the data storage volume per second and the data volume of the business data; obtaining a third value based on a quotient of the first value and the second value; and obtaining a storage block serial number based on a quotient of the time offset and the third value.

[0009] Further, compressing the storage data block to obtain a compressed data block includes: obtaining the last modification time of the storage data block; determining the data block type of the storage data block based on the last modification time, wherein the data block type is used to reflect whether the storage data block is marked; in response to the data block type being that the storage data block is marked, compressing the storage data block to obtain a compressed data block.

[0010] Further, based on the last modification time, determining the data block type of the storage data block includes: determining the time span corresponding to the storage data block based on the data block size of the storage data block and the amount of data stored per second, wherein the time span is used to characterize the time required for the storage data block to be filled; obtaining the time difference between the last modification time and the current time; and determining the data block type of the storage data block based on the shown time difference and time span.

[0011] Furthermore, the business data currently required to be stored and the business features of the business data are obtained, including: obtaining the initial service data output by the business engine; performing feature extraction on the initial service data to obtain business features; and aggregating the initial service data based on the business features to obtain business data.

[0012] According to another aspect of an embodiment of the present invention, a data storage system is also provided, including: a data acquisition module, which is used to acquire initial service data output by a business engine, and perform feature extraction on the initial service data to obtain business features of the initial service data, and summarize the initial service data based on the business features to obtain business data; a data transmission module, which is connected to the data acquisition module and the data compression module, and is used to transmit business data from the data acquisition module to the data compression module; a data compression module, which is used to construct a data storage directory and a storage sequence number based on the business features of the business data, and based on the storage directory and the storage sequence number, store the business data in a corresponding storage data block, and compress the storage data block in the case of successful storage to obtain a compressed data block, wherein the data storage directory is used to represent the directory used when storing different business data, and the data storage target includes multiple storage data blocks, and different storage data blocks correspond to different storage block numbers.

[0013] Furthermore, the data transmission module is also used to: read business data from the data acquisition module and generate a business message queue based on the business data; obtain the network status of the data transmission network; in response to the normal network status, transmit the business message queue to the data compression module; in response to the abnormal network status, store the business message queue in the local disk, and after the network status returns to normal, send the business message queue from the local disk to the data compression module.

[0014] According to another aspect of an embodiment of the present invention, a data storage device is also provided, including: a data acquisition module, used to acquire business data currently required to be stored, and business characteristics of the business data; a first construction module, used to construct a data storage directory and a storage block serial number based on the business characteristics, wherein the data storage directory is used to represent the directory used when storing business data, and the storage block serial number is used to represent the identification information of the storage data block where the business data is to be stored; a data storage module, used to store the business data in the storage data block based on the data storage directory and the storage block serial number; and a data compression module, used to compress the storage data block in response to the business data being successfully stored in the storage data block to obtain a compressed data block.

[0015] According to another aspect of an embodiment of the present invention, there is further provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the method in each embodiment of the present invention is executed when the program is running.

[0016] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0017] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0018] According to another aspect of an embodiment of the present invention, a computer program product is provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0019] According to another aspect of the embodiments of the present invention, a computer program is further provided. When the computer program is executed by a processor, the methods in the embodiments of the present invention are implemented.

[0020] In an embodiment of the present invention, the business data currently required to be stored and the business characteristics of the business data are obtained; based on the business characteristics, a data storage directory and a storage block number are constructed; based on the data storage directory and the storage block number, the business data is stored in a storage data block; in response to the business data being successfully stored in the storage data block, the storage data block is compressed to obtain a compressed data block. By reasonably constructing a storage directory structure, the business data is stored, and the stored data is further efficiently compressed. This can not only reduce the storage space occupied to reduce storage costs, but also quickly locate and retrieve data to shorten the query response time and improve user experience, thereby achieving the purpose of improving the read and write performance of the data storage system, thereby achieving the technical effect of being able to quickly query and return the stored data while saving data storage space, thereby solving the technical problem of poor storage effect of large data volumes in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0022] Figure 1 is a flow chart of a data storage method according to an embodiment of the present invention;

[0023] Figure 2 is a schematic diagram of an optional directory structure according to an embodiment of the present invention;

[0024] Figure 3 is an optional storage data block compression flow chart according to an embodiment of the present invention;

[0025] Figure 4 is a flow chart of an optional data query according to an embodiment of the present invention;

[0026] Figure 5 is a schematic diagram of a data storage system according to an embodiment of the present invention;

[0027] Figure 6 is a schematic diagram of an optional service data collection and push according to an embodiment of the present invention;

[0028] Figure 7 is a schematic diagram of a data storage device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0031] According to an embodiment of the present invention, an embodiment of a data storage method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] Figure 1 is a flow chart of a data storage method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0033] Step S102, obtaining the business data currently required to be stored and the business characteristics of the business data.

[0034] The business data may refer to information that can reflect the status, results, and trends of business activities. The business features may refer to features of business models or business attributes contained in the business data.

[0035] In an optional embodiment, considering that the data storage system is used to store business data, the data storage system may first obtain the business data that currently needs to be stored. Further considering that business features such as data type, business category, and generation time can help the data storage system classify data to improve storage efficiency, the data storage system may also obtain business features of the business data.

[0036] For example, the data storage system can collect user query requests and query results, and parse out business data. The business data includes but is not limited to user ID (Identification), query time, flight information, ticket price, etc. The data storage system can use a pre-trained feature extraction model to extract time features, user features, business type features, etc. from the above business data, thereby providing a data basis for subsequent storage operations.

[0037] Step S104, constructing a data storage directory and a storage block serial number based on the business characteristics, wherein the data storage directory is used to represent the directory used when storing business data, and the storage block serial number is used to represent the identification information of the storage data block where the business data is to be stored.

[0038] The data storage directory may refer to a logical or physical directory structure in a data storage system for organizing and classifying storage of business data. The storage block serial number may be a serial number for uniquely identifying a business data storage block.

[0039] In an optional embodiment, considering that by organizing business data into a specific data storage directory according to corresponding business characteristics, the search scope of the data storage system when performing data query can be reduced, thereby accelerating the data retrieval process. Therefore, the data storage system can construct a data storage directory and a storage block sequence number based on the business characteristics of the above business data. The construction of the above data storage directory and storage block sequence number can enable the data storage system to quickly locate the relevant data block according to the data query parameters, so as to avoid scanning the entire amount of stored data, thereby improving the query response speed.

[0040] For example, it can be assumed that the data storage system needs to process freight rate search requests, and each request carries business features such as user ID, query timestamp, and query type. In order to improve the efficiency of data retrieval, the data storage system can construct the first-level directory of the data storage directory based on the time feature using the year-month-day format, and construct the second-level directory of the data storage directory based on the user ID feature, and construct the third-level directory of the data storage directory based on the business type feature. Then, the data storage system can construct the above-mentioned storage block sequence number based on the generation frequency and data volume of business data. Based on the above-mentioned data storage directory and storage block sequence number, the freight rate search request data can be efficiently classified and stored.

[0041] Step S106: based on the data storage directory and the storage block sequence number, store the business data in the storage data block.

[0042] The above-mentioned storage data block may be a physical or logical structure for storing data.

[0043] In an optional embodiment, considering establishing a directory structure according to business characteristics (such as time, user ID, and business type), data can be stored in an organized and classified manner. In addition, during the data storage process, the data storage system can identify and manage different data blocks by using storage block numbers. Therefore, the data storage system can store business data in storage data blocks based on the data storage directory and storage block numbers. The above storage method can not only improve the efficiency of data storage, but also enable the data storage system to directly locate the directory where the relevant data is located when data is retrieved subsequently according to specific query conditions, thereby reducing the data search range and query time.

[0044] For example, in the freight rate search scenario, it can be assumed that the data storage system has built a data storage directory and storage block serial number. Based on this, business data can be written to a specific storage data block, and the file name for searching the storage data block can contain the storage block serial number to facilitate positioning and management by the scene storage system.

[0045] Step S108 , in response to the business data being successfully stored in the storage data block, compressing the storage data block to obtain a compressed data block.

[0046] The compressed data block mentioned above may refer to a storage data block processed by a compression algorithm.

[0047] In an optional embodiment, considering that the amount of data included in the freight rate search is usually large, hundreds of millions of data records may be generated every day, if the data is not compressed, the original data will occupy a huge storage space. Therefore, after the business data is successfully stored in the storage data block, the data storage system can compress the storage data block to obtain a compressed data block, thereby reducing the volume of the original data and saving storage space.

[0048] For example, it can be assumed that the data storage system has successfully stored the freight search data into the storage data block. In order to save storage space and improve the efficiency of data query and archiving, the data storage system can use the LZ4 compression algorithm to compress the storage data block. After the compression is completed, the data storage system can write the compressed data into a new file, which is the compressed data block. The volume of the search compressed data block is smaller than the storage data block, but the information of all the original data in the storage data block is retained. Subsequently, the data storage system can replace the above storage database with the above compressed data block to complete the compression operation.

[0049] In an embodiment of the present invention, the business data currently required to be stored and the business characteristics of the business data are obtained; based on the business characteristics, a data storage directory and a storage block number are constructed; based on the data storage directory and the storage block number, the business data is stored in a storage data block; in response to the business data being successfully stored in the storage data block, the storage data block is compressed to obtain a compressed data block. By reasonably constructing a storage directory structure, the business data is stored, and the stored data is further efficiently compressed. This can not only reduce the storage space occupied to reduce storage costs, but also quickly locate and retrieve data to shorten the query response time and improve user experience, thereby achieving the purpose of improving the read and write performance of the data storage system, thereby achieving the technical effect of being able to quickly query and return the stored data while saving data storage space, thereby solving the technical problem of poor storage effect of large data volumes in related technologies.

[0050] Furthermore, based on the business features, a data storage directory is constructed, including: obtaining feature priorities corresponding to the business features; and constructing the data storage directory based on the business features and the feature priorities.

[0051] The above feature priority may refer to the order of importance of various business features for data organization and query efficiency.

[0052] In an optional embodiment, considering that the freight rate search data may involve multiple query dimensions, such as user ID, business type, timestamp, etc., the data storage system can first divide the above business features according to priority, and build a data storage directory based on the business features and feature priorities. The above construction process allows data to be organized and stored according to more commonly used query dimensions. When performing data queries, the data storage system can quickly locate the target data based on the structure of the data storage directory, avoiding full scanning of the data and significantly shortening the query time.

[0053] For example, it can be assumed that the freight rate search system mainly processes three types of business features: timestamp, user ID, and business type. First, since timestamp is a more commonly used query dimension, the data storage system can configure a higher priority for timestamp. Secondly, user ID can be used to distinguish different freight rate providers, which is more important for data security, access control, and business analysis. Therefore, the data storage system can configure a medium priority for user ID. Finally, business type is used to further refine the data, but it also has certain query requirements, so the data storage system can assign a lower priority to the business type. That is, the feature priority size relationship of timestamp, user ID, and business type can be shown as follows:

[0054] Timestamp > User ID > Business Type.

[0055] Based on this, the data storage system can construct the following directory structure: Based on the timestamp feature, create a first-level directory in the format of year, month, and day. Based on the merchant ID feature, create a subdirectory under the first-level directory as a second-level directory. Based on the business type feature, create a subdirectory under the second-level directory as a third-level directory. By constructing the data storage directory through the above steps, the data storage system can efficiently store and retrieve data.

[0056] For ease of understanding, Figure 2 is a schematic diagram of an optional directory structure according to an embodiment of the present invention, such as Figure 2 As shown, the directory hierarchy starts with the root directory. The first-level directory under the root directory is generated according to the specific date when the freight rate query request occurs. The figure shows date 1, date 2 and date a. Under each first-level date directory, there are multiple second-level directories, which are divided according to different merchant IDs. The figure shows airline ID1, airline ID2 and airline IDb, thereby providing better data access control and ensuring that each merchant can only access data related to its ID, thereby enhancing data security. Each second-level merchant ID directory is further subdivided into three-level directories. These directories are based on the freight rate search business type. The figure shows two business type directories, "i shop" (international freight rate query) and "dshop" (domestic freight rate query). This division method enables the data storage system to perform more refined data management and query based on the differences in freight rate search types. Under the three-level directory, specific data blocks are stored. The figure shows "0.dat", "1.dat" and "c.dat". Each data block stores specific business data records. The figure shows message 1, message 2 and message d. Figure 2 The directory structure shown can not only effectively utilize storage space, but also quickly respond to business query needs. It also supports regular archiving and backup of data to ensure data persistence and availability.

[0057] Furthermore, the business characteristics include: a time offset of the business data and a data volume of the business data, and constructing a storage block serial number based on the business characteristics includes: obtaining a data block size of a storage data block and a data storage volume per second of the storage data block; obtaining a first value based on the product of the data block size and a preset value, and obtaining a second value based on the product of the data storage volume per second and the data volume of the business data; obtaining a third value based on a quotient of the first value and the second value; and obtaining a storage block serial number based on a quotient of the time offset and the third value.

[0058] The above-mentioned time offset may refer to the time difference between the business time when querying or writing and the zero point of the day (0 hours, 0 minutes, 0 seconds). The above-mentioned time offset may be used to determine the specific position of the data in the time dimension, so as to locate the corresponding data block. The above-mentioned preset value may be a constant used to calculate the size of the above-mentioned first value. The above-mentioned preset value may be set according to actual needs to ensure that the first value can properly reflect the influence of the data block size on the time slice span. The above-mentioned first value may be the product of the data block size and the preset value, which is used to determine the span of the time slice later. The above-mentioned second value may be the product of the amount of data stored per second and the amount of business data. The above-mentioned second value represents the data storage capacity per unit time when processing specific business data. The above-mentioned third value may be the quotient of the above-mentioned first value and the second value. The above-mentioned third value calculates the time span corresponding to each data block, that is, under the current storage conditions, how long the data can be stored in a single data block.

[0059] In an optional embodiment, in order to accurately calculate the storage block sequence number, the storage block sequence number may be calculated based on the following formula:

[0060]

[0061] In the formula, f(num) represents the storage block number, t 1 Indicates the business time when querying or writing, t c represents the zero o'clock of the day (0 hours, 0 minutes, 0 seconds), then the above time offset is t in the above formula 1 -t c 1024 is the preset value, blocksize indicates the data block size of the data block, and the first value is 1024×blocksize in the above formula. size indicates the data volume of business data, tps indicates the data storage volume per second, and the second value is size×tps in the above formula, and the third value is

[0062] Further, compressing the storage data block to obtain a compressed data block includes: obtaining the last modification time of the storage data block; determining the data block type of the storage data block based on the last modification time, wherein the data block type is used to reflect whether the storage data block is marked; in response to the data block type being that the storage data block is marked, compressing the storage data block to obtain a compressed data block.

[0063] The data block type may be an attribute used to identify the current state of the data block. The data block type may help the data storage system determine whether a compression operation can be performed on the currently stored data block.

[0064] In an optional embodiment, it is considered that before compressing the storage data block, it is necessary to determine whether the data block has been newly written or modified since the last operation, because before the storage data block is marked as compressible, there may be new data written. Therefore, by checking the last modification time of the storage data block, the data storage system can avoid compressing the storage data block when the storage data block has not been completed, thereby ensuring the consistency and integrity of the data before compression. Further considering that the above data block type is marked as compressible, it usually means that the storage data block is relatively stable and no longer frequently receives new data writes. Therefore, the data storage system determines whether to compress based on the last modification time, which can not only reduce the interference of the compression operation on the real-time data writing, but also obtain higher compression efficiency after the data is stable. Finally, after the storage data block is marked as compressible, the time storage system can compress the storage data block to obtain a compressed data block, thereby effectively reducing the storage space occupied.

[0065] For example, it can be assumed that there is a storage data block with a size of n. Before compressing the storage data block, the data storage system will first obtain the last modification time of the data block. Based on the last modification time, the data storage system can determine whether the storage data block meets the compression condition. If the last modification time is far away from the current time point, it indicates that no data item has been written to the storage data block. Therefore, the storage data block meets the compression condition. The data storage system can mark the data block type of the storage data block as compressible and start the compression task. The task will read the content stored in the storage data block and compress the storage data block by executing the compression algorithm, thereby obtaining a compressed data block of size m, where m is <n。

[0066] Further, based on the last modification time, determining the data block type of the storage data block includes: determining the time span corresponding to the storage data block based on the data block size of the storage data block and the amount of data stored per second, wherein the time span is used to characterize the time required for the storage data block to be filled; obtaining the time difference between the last modification time and the current time; and determining the data block type of the storage data block based on the shown time difference and time span.

[0067] The above time span may be a time period used to measure the time from when a storage data block starts to receive data to when it is filled and ready for compression.

[0068] In an optional embodiment, in order to determine the data block type of the storage data block according to the usage and update frequency of the storage data block, the data storage system can calculate the time span required for the storage data block to be filled, that is, the time required for the storage data block to be completely filled, through the data block size of the storage data block and the amount of data stored per second. Then, the data storage system can obtain the time difference between the last modification time and the current time to determine the time when the data block was last updated. Finally, the data storage system can compare the time difference with the time span to determine the type of the data block. For example, if the above time difference is less than the time span, it means that the data block is frequently updated and therefore cannot be compressed. On the contrary, if the time difference is greater than the time span, it means that the data block is less updated, so the storage data block can be marked as compressible. Through the above operations, the storage and access of the storage data blocks can be better managed and optimized, and the efficiency and performance of data storage can be improved.

[0069] Specifically, it can be assumed that there is a storage data block of size n, and the data storage capacity of the storage data block per second is T. The data storage system can determine that the time span corresponding to the storage data block is n / T by calculating the quotient of the data block size and the data storage capacity per second of the storage data block. Based on the above time span, the data storage system can calculate the difference between the last modification time t0 and the current time t1 as t1-t0. If this difference is greater than or equal to k times the time span, that is, kn / T, it means that the storage data block has entered a stable state since the last modification and no longer accepts new data writing. Therefore, the data block type of the storage data block can be determined as "marked state".

[0070] For ease of understanding, Figure 3 is an optional storage data block compression flow chart according to an embodiment of the present invention, such as Figure 3As shown, the process starts with receiving business data, which can be collected in real time from the freight rate search business data collection module, has been sorted, and contains key fields such as user ID, freight rate search business type, and freight rate query request occurrence time. Subsequently, the data storage system can calculate the block number of the data block. Specifically, according to the aforementioned calculation formula, the data storage system can calculate the corresponding data block number according to the timestamp and average data size of each business data. This step ensures that the data can be reasonably allocated to different data blocks, thereby realizing classified storage and efficient management of data. After calculating the block number of the data block, the data is written into the data block, that is, according to the calculated data block number, the data storage system can write the business data into the corresponding uncompressed data block. The size of the above data block can be set between 50M and 80M to balance storage efficiency and compression performance. At the same time, the compression task scheduling service creates a compression task, scans and marks the data block. In this process, the data compression module is responsible for creating a compression task, and periodically scans the uncompressed data blocks under the storage directory to mark the data blocks that meet the compression conditions. After marking the data block, the process enters the determination of modification time. The data storage system will check the last modification time of the marked data block to ensure that the modification time of the data block is relatively close to the current time. Once it is confirmed that the data block meets the compression conditions, the data storage system will perform compression operations to compress the content of the data block into a smaller file, so that the size of the compressed data block is reduced, usually only 2M to 3M, thereby saving storage space.

[0071] It should be noted that the specific values ​​such as the size of the above-mentioned data block and the size of the compressed data block are only shown for example. The staff can set them according to actual needs and are not limited here.

[0072] Furthermore, the business data currently required to be stored and the business features of the business data are obtained, including: obtaining the initial service data output by the business engine; performing feature extraction on the initial service data to obtain business features; and aggregating the initial service data based on the business features to obtain business data.

[0073] The business engine may be a program module responsible for core business logic processing. The business engine may receive query requests from users or systems, perform complex searches and calculations based on these requests, retrieve and calculate freight rate data from various data sources, and then generate detailed search results. The initial service data may be the original data record containing a large amount of search details and results output by the business engine after completing the search and calculation of freight rate information.

[0074] In an optional embodiment, considering that the initial service data may contain a large amount of redundant information, this information not only occupies valuable storage resources during the storage process, but also may increase unnecessary computing burdens during data retrieval and analysis. Therefore, after the data storage system obtains the initial service data output by the business engine, it can screen out service data with high business relevance through feature extraction as the above-mentioned business features, such as user ID, search time, search type, etc. Subsequently, the data storage system can summarize the initial service data based on the business features to obtain business data. Through the above steps, the demand for data storage can be significantly reduced, and the query efficiency and performance of the data storage system can be improved.

[0075] For example, it can be assumed that the service engine of the fare search system receives a user request, which includes the user's ID, search type (domestic or international), the departure and destination of the search, the departure date, flight preferences and other information. The above-mentioned service engine can process this information and then output the fare data related to the search request, that is, the above-mentioned initial service data. Subsequently, the data storage system can extract more important business features from the above-mentioned initial service data for subsequent data processing and storage. The above-mentioned business features may include the user's ID, search type, date and flight information. Finally, the data storage system can summarize the initial service data based on the above-mentioned business features to generate business data that is easier to store and query.

[0076] For ease of understanding, Figure 4 is a flow chart of an optional data query according to an embodiment of the present invention. Figure 4 As shown, first, the user queries business data and enters query parameters through the data query service interface. These parameters may include user ID, query type (domestic or international), query date and other information. Subsequently, the data storage system can locate the storage directory according to the business field, that is, locate the directory in the storage medium according to the query parameters entered by the user, such as user ID and query type. After locating the correct storage directory, the data storage system can locate the data block number according to the business time to ensure that the query service can accurately find the data block containing the user's query data. Then, the data storage system can read the compressed data into the cache, avoiding frequent reading directly from the physical storage device, thereby reducing I / O operations and improving query efficiency. After the data block is read into the cache, the data storage system can decompress the compressed data and filter the query, thereby retaining the data records that meet the query conditions. After the screening is completed, the data storage system can return the query results. For example, the query results can be sent back to the user's application through the API interface, or a query result page can be directly provided to the user. The specific return method depends on the specific implementation of the data storage system and the type of user request, which is not limited here.

[0077] According to an embodiment of the present invention, an embodiment of a data storage system is provided. It should be noted that the system can be used to execute the above data storage method. Figure 5 is a schematic diagram of a data storage system according to an embodiment of the present invention. Figure 5 As shown, the system includes:

[0078] The data collection module 502 is used to collect the initial service data output by the service engine, extract features from the initial service data to obtain service features of the initial service data, and summarize the initial service data based on the service features to obtain service data.

[0079] The data collection module may be a module responsible for real-time monitoring and collecting freight search data generated by the freight search engine service.

[0080] In an optional embodiment, considering that the initial service data output by the business engine may contain a large amount of redundant information and non-critical fields, direct storage will increase unnecessary storage burden and data processing complexity. By extracting features from the initial service data through the data acquisition module, the most critical information for business analysis and storage management can be screened out, such as business features such as user ID, search time, and search type, which helps to quickly process and streamline data storage. In addition, aggregating the initial service data based on business features is conducive to classifying and organizing business data according to key business features, thereby greatly improving the speed and efficiency of data query. For example, data is aggregated and stored according to dimensions such as date, merchant ID, and business type, so that the query system can quickly respond to query requests and avoid full scanning of the original data, thereby significantly shortening the query response time and improving the user experience.

[0081] The data transmission module 504 is connected to the data acquisition module 502 and the data compression module 506 , and is used to transmit the business data from the data acquisition module 502 to the data compression module 506 .

[0082] The data transmission module may be a module for transmitting the data collected by the data acquisition module to the data compression module.

[0083] In an optional embodiment, the data transmission module can ensure that the above-mentioned business data is safely and completely transmitted from the above-mentioned data acquisition module to the data compression module. In the case of network instability or high load, the data transmission module can handle data retransmission and error detection, ensure the continuity and consistency of data, and avoid the risk of data loss or incorrect processing. In addition, the data transmission module optimizes the transmission process of data from acquisition to compressed storage, and through efficient transmission protocols and strategies, high-speed data transmission can be achieved, even when processing large amounts of data, the timeliness and efficiency of data processing can be maintained. For example, the use of kafka (Kafka, a distributed stream processing platform) message queue can ensure the order and integrity of data while supporting high-throughput data transmission.

[0084] The data compression module 506 is used to construct a data storage directory and a storage sequence number based on the business characteristics of the business data, and based on the storage directory and the storage sequence number, store the business data in the corresponding storage data block, and compress the storage data block when the storage is successful to obtain a compressed data block, wherein the data storage directory is used to represent the directory used when storing different business data, and the data storage target includes multiple storage data blocks, and different storage data blocks correspond to different storage block numbers.

[0085] The above-mentioned data compression module may be a module used to compress and store data to reduce the required storage space and improve system performance.

[0086] In an optional embodiment, data compression is an important means to improve storage efficiency and reduce storage costs. By storing data in corresponding storage data blocks and then compressing them, the actual storage space occupied can be significantly reduced. In the data storage system, the data compression module can compress each data block to a very small proportion of its original size, thereby saving storage resources and reducing the hardware cost and management cost of long-term data storage.

[0087] Furthermore, the data transmission module is also used to: read business data from the data acquisition module and generate a business message queue based on the business data; obtain the network status of the data transmission network; in response to the normal network status, transmit the business message queue to the data compression module; in response to the abnormal network status, store the business message queue in the local disk, and after the network status returns to normal, send the business message queue from the local disk to the data compression module.

[0088] The above-mentioned business message queue can be a data structure for storing and managing business data obtained from the data acquisition module. For example, the business message queue shown can be a kafka message queue, but is not limited to this. The above-mentioned network status can refer to the network connection status during the data transmission process from the data transmission module to the data compression module.

[0089] In an optional embodiment, the data transmission module reads the business data from the data acquisition module and generates a business message queue based on the data. The use of the message queue ensures the orderliness of data processing. When the network status is normal, based on the above-mentioned business message queue, the data transmission module can efficiently transmit the business data to the data compression module. When the network status is abnormal, the data transmission module will temporarily store the business message queue to the local disk, and after the network status returns to normal, the business message queue will be sent from the local disk to the data compression module, thereby avoiding data loss caused by network problems. This mechanism ensures that even when the network is unstable or fails, the data storage system can still maintain the continuity of data transmission and the integrity of the data.

[0090] For ease of understanding, Figure 6 is a schematic diagram of an optional service data collection and push according to an embodiment of the present invention, such as Figure 6 As shown, the freight search engine can receive and process user query requests to generate freight search data. The freight search data collection module can capture freight search data from the freight search engine in real time to prepare for subsequent data processing. After completing the above data capture, the freight search data collection module can organize and send business data. For example, the freight search data collection module can organize the data according to predefined query dimensions (such as user ID, freight search business type, search request time, etc.), and then send the data to the kafka cluster. As a message queue service, the kafka cluster plays the role of a data transfer station. It receives and temporarily stores data from the freight search data collection module, further distributes the data to the freight search data compression module, and judges the result. If the data transmission fails due to network abnormalities and other reasons, the failed data will be temporarily written to the local disk. After the network is restored, the compensation sending task is started to reorganize and send the failed data in the local disk to ensure the integrity of data transmission and the high availability of the system, so as to compensate and send the compensation data in the local disk in a timely and quantitative manner, thereby ensuring the integrity of data transmission and the high availability of the entire system.

[0091] According to an embodiment of the present invention, an embodiment of a data storage device is provided. It should be noted that the device can be used to execute the above data storage method. The specific implementation method and application scenario are the same as those of the above embodiment, and will not be repeated here. Figure 7 is a schematic diagram of a data storage device according to an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0092] The data acquisition module 702 is used to acquire the business data that currently needs to be stored and the business characteristics of the business data.

[0093] The first construction module 704 is used to construct a data storage directory and a storage block serial number based on business characteristics, wherein the data storage directory is used to represent the directory used when storing business data, and the storage block serial number is used to represent the identification information of the storage data block where the business data is to be stored.

[0094] The data storage module 706 is used to store the business data in the storage data block based on the data storage directory and the storage block sequence number.

[0095] The data compression module 708 , in response to the business data being successfully stored in the storage data block, compresses the storage data block to obtain a compressed data block.

[0096] Furthermore, the first construction module is also used to: obtain the feature priority corresponding to the business feature; and construct a data storage directory based on the business feature and the feature priority.

[0097] Furthermore, the business characteristics include: the time offset of the business data and the data volume of the business data. The first construction module is also used to: obtain the data block size of the storage data block and the data storage volume per second of the storage data block; obtain a first value based on the product of the data block size and a preset value, and obtain a second value based on the product of the data storage volume per second and the data volume of the business data; obtain a third value based on the quotient of the first value and the second value; obtain a storage block sequence number based on the quotient of the time offset and the third value.

[0098] Furthermore, the data compression module is also used to: obtain the last modification time of the storage data block; determine the data block type of the storage data block based on the last modification time, wherein the data block type is used to reflect whether the storage data block is marked; in response to the data block type being that the storage data block is marked, compress the storage data block to obtain a compressed data block.

[0099] Furthermore, the data compression module is also used to: determine the time span corresponding to the storage data block based on the data block size of the storage data block and the data storage volume per second, wherein the time span is used to characterize the time required for the storage data block to be filled; obtain the time difference between the last modification time and the current time; and determine the data block type of the storage data block based on the shown time difference and time span.

[0100] Furthermore, the data acquisition module is also used to: acquire initial service data output by the business engine; perform feature extraction on the initial service data to obtain business features; and summarize the initial service data based on the business features to obtain business data.

[0101] An embodiment of the present application further provides an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein the program executes the methods in various embodiments of the present invention when running.

[0102] An embodiment of the present application further provides a computer-readable storage medium, which includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the methods in various embodiments of the present invention.

[0103] An embodiment of the present application further provides a computer program product, including a computer program, which implements the methods in various embodiments of the present invention when executed by a processor.

[0104] An embodiment of the present application further provides a computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium is used to store a computer program, and when the computer program is executed by a processor, the method in each embodiment of the present invention is implemented.

[0105] The embodiments of the present application further provide a computer program, which implements the methods in the above-mentioned embodiments of the present invention when executed by a processor.

[0106] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0107] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0108] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0109] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0110] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0111] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A data storage method, characterized in that: include: Acquire the business data that currently needs to be stored and the business characteristics of the business data; Based on the service characteristics, construct a data storage directory and a storage block sequence number, wherein the data storage directory is used to represent a directory used when storing the service data, and the storage block sequence number is used to represent identification information of a storage data block where the service data is to be stored; Based on the data storage directory and the storage block sequence number, storing the business data in the storage data block; In response to the business data being successfully stored in the storage data block, the storage data block is compressed to obtain a compressed data block.

2. The method according to claim 1, characterized in that: Based on the business characteristics, a data storage directory is constructed, including: Obtaining a feature priority corresponding to the service feature; The data storage directory is constructed based on the service characteristics and the characteristic priorities.

3. The method according to claim 1, characterized in that The service characteristics include: the time offset of the service data and the data volume of the service data. Constructing the storage block sequence number based on the service characteristics includes: Obtaining a data block size of the storage data block and a data storage capacity per second of the storage data block; A first value is obtained based on a product of the data block size and a preset value, and a second value is obtained based on a product of the data storage volume per second and the data volume of the service data; Obtaining a third value based on a quotient of the first value and the second value; The storage block sequence number is obtained based on the quotient of the time offset and the third value.

4. The method according to claim 1, characterized in that: Compressing the stored data block to obtain a compressed data block includes: Obtaining the last modification time of the storage data block; Determine, based on the last modification time, a data block type of the storage data block, wherein the data block type is used to reflect whether the storage data block is marked; In response to the data block type being marked as the storage data block, compressing the storage data block to obtain the compressed data block.

5. The method according to claim 4, characterized in that Determining the data block type of the storage data block based on the last modification time includes: Determine a time span corresponding to the storage data block based on a data block size of the storage data block and a data storage amount per second, wherein the time span is used to represent a time required for the storage data block to be filled; Get the time difference between the last modification time and the current time; Based on the time difference value and the time span, a data block type of the stored data block is determined.

6. The method according to claim 1, characterized in that Obtain the business data that currently needs to be stored, as well as the business characteristics of the business data, including: Obtain the initial service data output by the business engine; Extracting features from the initial service data to obtain the service features; The initial service data is aggregated based on the service characteristics to obtain the service data.

7. A data storage system, characterized in that: include: A data collection module, used to collect initial service data output by the service engine, perform feature extraction on the initial service data to obtain service features of the initial service data, and summarize the initial service data based on the service features to obtain service data; A data transmission module, connected to the data acquisition module and the data compression module, and used to transmit the business data from the data acquisition module to the data compression module; The data compression module is used to construct a data storage directory and a storage sequence number based on the business characteristics of the business data, and based on the storage directory and the storage sequence number, store the business data in a corresponding storage data block, and compress the storage data block in the case of successful storage to obtain a compressed data block, wherein the data storage directory is used to represent the directory used when storing different business data, the data storage target includes a plurality of the storage data blocks, and different storage data blocks correspond to different storage block numbers.

8. The system according to claim 7, characterized in that The data transmission module is also used for: Reading the business data from the data acquisition module, and generating a business message queue based on the business data; Get the network status of the data transmission network; In response to the network status being normal, transmitting the service message queue to the data compression module; In response to the abnormal network status, the service message queue is stored in a local disk, and after the network status returns to normal, the service message queue is sent from the local disk to the data compression module.

9. An electronic device, characterized in that: include: A memory storing an executable program; A processor, configured to run the program, wherein the program executes the method according to any one of claims 1 to 6 when running.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is executed, the device where the storage medium is located is controlled to execute the method according to any one of claims 1 to 6.

Images