Data storage method, search method, device and electronic equipment

Abstract

The present disclosure provides a data storage method, a search method, a device and an electronic device, relates to the field of data processing, and in particular to the field of data storage. The specific implementation scheme is: a data storage method applied to a server, the server comprising a data storage unit, the data storage unit comprising at least two data storage shards, the method comprising: obtaining a first identifier of a first target shard, wherein the first target shard is one of the at least two data storage shards, and the first identifier is used to represent the data type of the data stored in the first target shard; in the case that the first identifier represents that the first target shard is used to store cold and hot type data, and a second identifier of the first target shard is obtained, migrating the data stored in the first target shard to an object storage system, wherein the second identifier is used to represent that the data in the first target shard meets a preset refrigeration condition. The present disclosure can improve the data storage effect.

Description

Technical Field

[0001] This disclosure relates to the field of data processing, and more particularly to the field of data storage. Specifically, it relates to a data storage method, a search method, an apparatus, and an electronic device. Background Technology

[0002] Existing servers typically include data storage units, such as solid-state drives (SSDs). These data storage units store the data required during the server's service provision process. When the server receives a data query, it can retrieve the data corresponding to the query request from the data storage unit and output the retrieved data. Summary of the Invention

[0003] This disclosure provides a data storage method, a search method, an apparatus, and an electronic device.

[0004] According to a first aspect of this disclosure, a data storage method is provided, applied to a server, the server including a data storage unit, the data storage unit including at least two data storage fragments, the method comprising:

[0005] Obtain the first identifier of the first target fragment, wherein the first target fragment is one of the at least two data storage fragments, and the first identifier is used to characterize the data type of the data stored in the first target fragment;

[0006] When the first identifier indicates that the first target segment is used to store hot and cold type data, and the second identifier of the first target segment is obtained, the data stored in the first target segment is migrated to the object storage system, and the second identifier is used to indicate that the data in the first target segment meets the preset cooling conditions.

[0007] According to a second aspect of this disclosure, a search method is provided, applied to a server, the method comprising:

[0008] Receive query terms;

[0009] The query is performed in the data storage unit of the server based on the query terms;

[0010] If no query result corresponding to the query term is found, the query term is sent to the object storage system so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit.

[0011] Upon receiving a query result corresponding to the query term sent by the object storage system, the query result is output.

[0012] According to a third aspect of this disclosure, a data storage device is provided, the data storage device including a data storage unit, the data storage unit including at least two data storage fragments, the device comprising:

[0013] The acquisition module is used to acquire a first identifier of a first target fragment, wherein the first target fragment is one of the at least two data storage fragments, and the first identifier is used to characterize the data type of the data stored in the first target fragment;

[0014] The migration module is used to migrate the data stored in the first target shard to the object storage system when the first identifier indicates that the first target shard is used to store hot and cold type data and the second identifier of the first target shard is obtained. The second identifier is used to indicate that the data in the first target shard meets the preset cooling conditions.

[0015] According to a fourth aspect of this disclosure, a search apparatus is provided, the method comprising:

[0016] The second receiving module is used to receive query terms;

[0017] The query module is used to perform a query in the data storage unit of the search device based on the query terms;

[0018] The sending module is used to send the query term to the object storage system when no query result corresponding to the query term is found, so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit.

[0019] The output module is used to output the query result when it receives the query result corresponding to the query term sent by the object storage system.

[0020] According to a fifth aspect of this disclosure, an electronic device is provided, comprising:

[0021] At least one processor; and

[0022] A memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in the first aspect or the second aspect.

[0024] According to a sixth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are configured to cause the computer to perform the method described in the first or second aspect.

[0025] According to a seventh aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described in the first or second aspect.

[0026] In this embodiment of the disclosure, by migrating the cold and hot type data that meet the preset cooling conditions from the data storage unit of the server to the object storage system, it is beneficial to alleviate the data storage pressure in the data storage unit, thereby improving the data storage effect. Attached Figure Description

[0027] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0028] Figure 1 This is a flowchart of a data storage method provided in an embodiment of this disclosure;

[0029] Figure 2 This is a flowchart of a search method provided in an embodiment of this disclosure;

[0030] Figure 3 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this disclosure;

[0031] Figure 4 This is a schematic diagram of the structure of a search device provided in an embodiment of this disclosure;

[0032] Figure 5 This is a block diagram of an electronic device for implementing a data storage method or a search method, provided in an embodiment of this disclosure. Detailed Implementation

[0033] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0034] Please see Figure 1This disclosure provides a data storage method applied to a server. The server includes a data storage unit, which includes at least two data storage shards. The data storage method includes the following steps:

[0035] Step S101: Obtain the first identifier of the first target fragment, wherein the first target fragment is one of the data storage fragments of the at least two data storage fragments, and the first identifier is used to characterize the data type of the data stored in the first target fragment;

[0036] Step S102: When the first identifier indicates that the first target segment is used to store hot and cold type data, and the second identifier of the first target segment is obtained, the data stored in the first target segment is migrated to the object storage system. The second identifier is used to indicate that the data in the first target segment meets the preset cooling conditions.

[0037] The aforementioned server can be a server in various service platforms, such as a server in a distributed full-text search (Elasticsearch, ES) system. The aforementioned data storage unit can be a storage unit formed by the server's local disk. Different data storage shards in the at least two data storage shards can be used to store different types of data. When creating the data storage shard, a first identifier can be set for the data storage shard, the first identifier being used to characterize the data type of the data stored in the data storage shard.

[0038] The data types can include hot data and cold / hot data. Hot data refers to data with high access frequency and high efficiency requirements, while cold / hot data refers to data with low access frequency and low efficiency requirements. Because hot data has high access frequency and high efficiency requirements, it can always be stored in one data storage shard of the data storage unit. Since cold / hot data has low access frequency and low efficiency requirements, it can be initially stored in one data storage shard of the data storage unit, and then migrated to the object storage system once preset cooling conditions are met.

[0039] Specifically, when the first target shard is a data storage shard used to store hot and cold data, the data in the first target shard can be migrated to the object storage system at preset intervals. For example, when the time interval between the last data migration of the first target shard and the first target shard reaches the preset time interval, a second identifier can be set for the first target shard. At this time, since the server can obtain the second identifier of the first target shard, the data stored in the first target shard can be migrated to the object storage system. That is, the preset cooling condition can be: the time interval between the last data migration of the first target shard and the first target shard reaches the preset time interval.

[0040] During the migration of data stored in the first target shard, all data in the first target shard can be traversed, and an object ID can be generated for each data file in the first target shard. This facilitates subsequent data retrieval based on the object ID, and then the data files are migrated to the object storage system. After the migration of all data in the first target shard is completed, the second identifier of the first target shard can be modified to a third identifier, whereby the third identifier indicates that the data in the first target shard has been migrated. When the first target shard has the third identifier, the server can write new hot / cold type data to the first target shard. After the preset time period, the third identifier of the first target shard is modified back to the second identifier to facilitate the next data migration. The value of the preset time period can be modified according to actual needs.

[0041] The first, second, and third identifiers can be stored in the index files of the data storage fragments. This way, the server only needs to traverse the index files of each data storage fragment to determine the identifiers contained in each data storage fragment.

[0042] It is understood that the aforementioned object storage system may include multiple data shards, and any data shard in the storage system may correspond to a shard in the data storage unit used for storing hot and cold type data. For example, when the first target shard is a data storage shard used for storing hot and cold type data, the object storage system includes a first shard corresponding to the first target shard. When the data in the first target shard meets preset cooling conditions, the data in the first target shard can be migrated to the first shard of the object storage system.

[0043] In related technologies, storing all data on a server's local disk results in relatively high data storage costs because these disks are typically high-performance disks. In an Elasticsearch (ES) system, expanding the server's local disk usually requires restarting the ES data nodes, making the expansion process noticeable to users. Furthermore, when an ES system includes multiple servers, each server can only access data from its own disk node. This can lead to the same data needing to be stored on multiple servers' local disks, resulting in wasted disk space. Additionally, the storage space on a server's local disk is typically much smaller than that of an object storage system, leading to limitations on the amount of data that can be stored.

[0044] Based on this, in this embodiment, hot and cold data in the Elasticsearch (ES) system is stored using a sample object storage system. Since the data storage cost of the object storage system is relatively low, this reduces data storage costs. Furthermore, during the expansion of the object storage system, there is no need to restart the ES nodes, thus achieving a seamless expansion process for users. Because multiple servers in the ES system can share a single object storage system, identical data only needs to be stored once in the object storage system for each server to access, thereby avoiding data redundancy. In addition, the object storage system supports ES-level data storage, offering higher storage capacity; therefore, it can store more data compared to existing storage solutions.

[0045] In this embodiment, by migrating the data of cold and hot types that meet the preset cooling conditions from the data storage unit of the server to the object storage system, the data storage pressure in the data storage unit is relieved, thereby improving the data storage effect.

[0046] Optionally, the method further includes:

[0047] Receive target data;

[0048] If the second identifier of the second target fragment is not obtained, the target data is written to the second target fragment;

[0049] If the second identifier of the second target fragment is obtained, the target data shall be refused to be written to the second target fragment;

[0050] The data type stored in the second target fragment matches the data type of the target data.

[0051] The aforementioned target data can be various types of data written to the server, specifically data actively acquired by the server or data imported by relevant personnel. That is, when the server receives the target data to be written, it can first determine the second target shard in the data storage unit, and then obtain the second identifier of the second target shard. If the second identifier is obtained, meaning the second target shard meets the preset cooling conditions, it indicates that the second target shard may be currently undergoing data migration. However, if new data is continuously written to the second target shard during the data migration process, the data migration process of the second target shard may never be completed, leading to the inability to migrate the hot and cold type data of other shards. Therefore, in this embodiment, if the second identifier is obtained, the target data can be refused to be written to the second target shard. Simultaneously, the target data can be written to the second target shard after the data migration of the second target shard is completed.

[0052] Accordingly, if the second identifier of the second target segment is not obtained, it means that the second target segment does not currently meet the preset cooling conditions. At this time, the data in the second target segment is not in the data migration state. Therefore, the target data can be parsed and then written into the second target segment.

[0053] The data type of the data stored in the second target fragment is matched with the data type of the target data, which means that the data type of the data stored in the second target fragment is the same as the data type of the target data.

[0054] In this implementation, when the target data to be written is received, it is determined whether the second target shard is currently in a data migration state based on the second identifier. If it is in a data migration state, the target data is refused to be written to the second target shard. In this way, it can be ensured that the data of the second target shard completes the data migration process normally.

[0055] Optionally, before writing the target data into the second target fragment if the second identifier of the second target fragment is not obtained, the method further includes:

[0056] If the second target fragment is not included in the at least two data storage fragments, the second target fragment is created in the data storage unit.

[0057] Specifically, after receiving the target data, it can be first determined whether the at least two data storage shards include a second target shard that matches the data type of the target data. If it exists, the second identifier of the second target shard is directly obtained to determine whether the second identifier can be successfully obtained. If it does not exist, the second target shard and its index file can be created in the data storage unit. It is understood that when creating the index file of the second target shard, a first identifier of the second target shard can be set in the index file. The first identifier can be set manually or automatically by the server based on pre-configured rules. Then, the second identifier of the second target shard is obtained to determine whether the second identifier can be successfully obtained.

[0058] In this embodiment, by creating the second target fragment in the data storage unit when the second target fragment is not included in the at least two data storage fragments, it is beneficial to ensure that the target data can be stored normally in the data storage unit.

[0059] Optionally, after creating the second target fragment in the data storage unit, the method further includes:

[0060] Obtain the first identifier of the second target fragment;

[0061] When the first identifier of the second target shard indicates that the second target shard is used to store hot and cold type data, a data mapping file is created between the second target shard and the object storage system. The server is used to migrate the data stored in the second target shard to the object storage system based on the data mapping file.

[0062] The data mapping file can be a metaState.json file. The data mapping file may include the attribute contents listed in the table below:

[0063] Serial Number property describe 1 cluster_uuid Cluster ID 2 cluster_name Cluster Name 3 index_name Index Name 4 index_uuid index id 5 shard_id Shard ID 6 shard_version Fragmented version 7 path_uid Path prefix for storing data in object storage systems

[0064] Since the data mapping file includes the attribute content in the table above, data in the second target shard can be migrated to the corresponding location in the object storage system based on the data mapping file.

[0065] In this embodiment, by creating a data mapping file between the second target shard and the object storage system, it is convenient to migrate the data in the second target shard to the corresponding location in the object storage system.

[0066] Optionally, after migrating the data stored in the first target shard to the object storage system, the method further includes:

[0067] The data stored in the first target shard is deleted, and the second identifier of the first target shard is updated to a third identifier, wherein the third identifier is used to indicate that the data in the first target shard has been migrated.

[0068] In this embodiment, after migrating the data stored in the first target shard to the object storage system, the data stored in the first target shard is deleted to free up storage space in the first target shard. When the server needs to access the data in the first target shard, it can retrieve it from the object storage system. Simultaneously, the second identifier of the first target shard is updated to a third identifier to enable the first target shard to receive newly written data.

[0069] Optionally, after updating the second identifier of the first target fragment to the third identifier, the method further includes:

[0070] After a preset time interval, the third identifier of the first target fragment is updated to the second identifier.

[0071] In this embodiment, the third identifier of the first target shard is updated to the second identifier after a preset time interval, so that the server can obtain the second identifier of the first target shard and migrate the data of the first target shard to the object storage system again, thereby achieving the purpose of periodically migrating the data in the first target shard to the object storage system.

[0072] Optionally, the data storage shard has a preset code, and the preset codes of the at least two data storage shards are arranged in a preset order. After obtaining the first identifier of the first target shard, the method further includes:

[0073] When the first identifier of the first target fragment indicates that the first target fragment is used to store non-hot / cold type data, the first identifier of the third target fragment is obtained. The third target fragment is: the next data storage fragment of the first target fragment among the at least two data storage fragments arranged in the preset order.

[0074] If the first identifier of the third target partition indicates that the third target partition is used to store hot and cold type data, and the second identifier of the third target partition is obtained, the data stored in the third target partition is migrated to the object storage system.

[0075] In this embodiment, the server can traverse each data storage segment of the data storage unit in a preset order at preset intervals. When a data storage segment is used to store non-hot / cold data, that is, when the data in the segment does not need to be migrated to the object storage system, the data storage segment is skipped, and the first identifier of the next data storage segment is obtained to further determine whether the data storage segment is used to store hot / cold data, until all data storage segments of the data storage unit are traversed, thereby completing one data migration process.

[0076] Optionally, after migrating the data stored in the first target shard to the object storage system, the method further includes:

[0077] Obtain the first identifier of the third target fragment;

[0078] If the first identifier of the third target partition indicates that the third target partition is used to store hot and cold type data, and the second identifier of the third target partition is obtained, the data stored in the third target partition is migrated to the object storage system.

[0079] In this embodiment, when the server traverses each data storage segment of the data storage unit in the preset order, after completing one data storage segment, it obtains the first identifier of the next data storage segment to further determine whether the data storage segment is used to store hot or cold data, until all data storage segments of the data storage unit are traversed, thereby completing a data migration process.

[0080] Please see Figure 2 This disclosure provides a search method comprising the following steps:

[0081] Step S201: Receive query terms;

[0082] Step S202: Perform a query on the data storage unit of the server based on the query terms;

[0083] Step S203: If no query result corresponding to the query term is found, the query term is sent to the object storage system so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit.

[0084] Step S204: Upon receiving the query result corresponding to the query term sent by the object storage system, output the query result.

[0085] It is understandable that if a query is performed on the server's data storage unit based on the query term and a query result corresponding to the query term is found, the query result can be directly output.

[0086] Specifically, when the query service in the server receives a query request, it obtains the query terms from the query request and then converts the query terms into a set of query commands for underlying index files. It then iterates through the query commands; if a corresponding operation file or folder is found in the data storage unit, the query results of each index file are aggregated, assembled, and the query result is returned. Conversely, if the operation file or folder cannot be found in the data storage unit, the file is downloaded from the remote object storage system block by block to complete the query operation, and the query results of each index file are aggregated, assembled, and the query result is returned.

[0087] In this implementation, a query is first performed on the local data storage unit of the server based on the query term. If no query result corresponding to the query term is found, a query is then performed based on the object storage system to ensure that the data query process proceeds normally.

[0088] Please see Figure 3 , Figure 3 This is a schematic diagram of the structure of a data storage device 300 provided in an embodiment of the present disclosure. The data storage device 300 includes a data storage unit, the data storage unit includes at least two data storage fragments, and the device includes:

[0089] The acquisition module 301 is used to acquire the first identifier of the first target fragment, wherein the first target fragment is one of the data storage fragments of the at least two data storage fragments, and the first identifier is used to characterize the data type of the data stored in the first target fragment;

[0090] The migration module 302 is used to migrate the data stored in the first target segment to the object storage system when the first identifier indicates that the first target segment is used to store hot and cold type data and the second identifier of the first target segment is obtained. The second identifier is used to indicate that the data in the first target segment meets the preset cooling conditions.

[0091] Optionally, the device further includes:

[0092] The first receiving module is used to receive target data;

[0093] The writing module is used to write the target data into the second target fragment if the second identifier of the second target fragment is not obtained;

[0094] The writing module is further configured to refuse to write the target data to the second target fragment when the second identifier of the second target fragment is obtained;

[0095] The data type stored in the second target fragment matches the data type of the target data.

[0096] Optionally, the device further includes:

[0097] A creation module is configured to create the second target fragment in the data storage unit if the second target fragment is not included in the at least two data storage fragments.

[0098] Optionally, the acquisition module 301 is further configured to acquire the first identifier of the second target fragment;

[0099] The creation module is further configured to create a data mapping file between the second target partition and the object storage system when the first identifier of the second target partition indicates that the second target partition is used to store hot and cold type data. The data storage device 300 is configured to migrate the data stored in the second target partition to the object storage system based on the data mapping file.

[0100] Optionally, the device further includes:

[0101] The processing module is used to delete the data stored in the first target shard and update the second identifier of the first target shard to a third identifier, wherein the third identifier is used to indicate that the data in the first target shard has been migrated.

[0102] Optionally, the device further includes:

[0103] The update module is used to update the third identifier of the first target fragment to the second identifier after a preset time interval.

[0104] Optionally, the data storage shard has a preset code, and the preset codes of the at least two data storage shards are arranged in a preset order. The acquisition module 301 is further configured to acquire the first identifier of the third target shard when the first identifier of the first target shard indicates that the first target shard is used to store non-hot / cold type data. The third target shard is the next data storage shard of the first target shard among the at least two data storage shards arranged in the preset order.

[0105] The migration module 302 is further configured to migrate the data stored in the third target shard to the object storage system when the first identifier of the third target shard indicates that the third target shard is used to store hot and cold type data, and the second identifier of the third target shard is obtained.

[0106] Optionally, the acquisition module 301 is further configured to acquire the first identifier of the third target fragment;

[0107] The migration module 302 is further configured to migrate the data stored in the third target shard to the object storage system when the first identifier of the third target shard indicates that the third target shard is used to store hot and cold type data, and the second identifier of the third target shard is obtained.

[0108] It should be noted that the data storage device 300 provided in this embodiment can be the server in the above embodiments, and the data storage device 300 can implement all the technical solutions of the above data storage method embodiments, so it can at least achieve all the above technical effects, which will not be repeated here.

[0109] Please see Figure 4 , Figure 4 This is a schematic diagram of a search device 400 provided in an embodiment of the present disclosure. The method includes:

[0110] The second receiving module 401 is used to receive query terms;

[0111] Query module 402 is used to perform a query in the data storage unit of the search device based on the query terms;

[0112] The sending module 403 is used to send the query term to the object storage system when no query result corresponding to the query term is found, so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit.

[0113] The output module 404 is used to output the query result when it receives the query result corresponding to the query term sent by the object storage system.

[0114] It should be noted that the search device 400 provided in this embodiment can be the server in the above embodiments, and the search device can implement all the technical solutions of the above search method embodiments, so it can at least achieve all the above technical effects, which will not be repeated here.

[0115] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0117] Figure 5 A schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 5 As shown, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 502 or a computer program loaded from a storage unit 508 into a random access memory (RAM) 503. The RAM 503 may also store various programs and data required for the operation of the device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0119] Multiple components in electronic device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of monitors, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as data storage methods or search methods. For example, in some embodiments, the data storage methods or search methods may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and / or installed on device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by the computing unit 501, one or more steps of the data storage methods or search methods described above are performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform a data storage method or a search method by any other suitable means (e.g., by means of firmware).

[0121] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0122] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0123] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0124] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0125] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0126] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0127] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0128] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data storage method applied to a server, the server including a data storage unit, the data storage unit including at least two data storage shards, the method comprising: Obtain the first identifier of the first target fragment, wherein the first target fragment is one of the at least two data storage fragments, and the first identifier of the first target fragment is used to characterize the data type of the data stored in the first target fragment; When the first identifier of the first target segment indicates that the first target segment is used to store hot and cold type data, and the second identifier of the first target segment is obtained, the data stored in the first target segment is migrated to the object storage system. The second identifier of the first target segment is used to indicate that the data in the first target segment meets the preset cooling conditions. Receive target data; If the second identifier of the second target fragment is not obtained, the target data is written to the second target fragment; If the second identifier of the second target fragment is obtained, the target data will not be written to the second target fragment. The data type stored in the second target fragment matches the data type of the target data.

2. The method according to claim 1, wherein, Before writing the target data into the second target fragment if the second identifier of the second target fragment has not been obtained, the method further includes: If the second target fragment is not included in the at least two data storage fragments, the second target fragment is created in the data storage unit.

3. The method according to claim 2, wherein, After creating the second target fragment in the data storage unit, the method further includes: Obtain the first identifier of the second target fragment; When the first identifier of the second target shard indicates that the second target shard is used to store hot and cold type data, a data mapping file is created between the second target shard and the object storage system. The server is used to migrate the data stored in the second target shard to the object storage system based on the data mapping file.

4. The method according to claim 1, wherein, After migrating the data stored in the first target shard to the object storage system, the method further includes: The data stored in the first target segment is deleted, and the second identifier of the first target segment is updated to the third identifier, wherein the third identifier of the first target segment is used to indicate that the data in the first target segment has been migrated.

5. The method according to claim 4, wherein, After updating the second identifier of the first target fragment to the third identifier, the method further includes: After a preset time interval, the third identifier of the first target fragment is updated to the second identifier.

6. The method according to claim 1, wherein, The data storage shards have preset codes, and the preset codes of the at least two data storage shards are arranged in a preset order. After obtaining the first identifier of the first target shard, the method further includes: When the first identifier of the first target fragment indicates that the first target fragment is used to store non-hot / cold type data, the first identifier of the third target fragment is obtained. The third target fragment is: the next data storage fragment of the first target fragment among the at least two data storage fragments arranged in the preset order. If the first identifier of the third target partition indicates that the third target partition is used to store hot and cold type data, and the second identifier of the third target partition is obtained, the data stored in the third target partition is migrated to the object storage system.

7. The method according to claim 6, wherein, After migrating the data stored in the first target shard to the object storage system, the method further includes: Obtain the first identifier of the third target fragment; If the first identifier of the third target partition indicates that the third target partition is used to store hot and cold type data, and the second identifier of the third target partition is obtained, the data stored in the third target partition is migrated to the object storage system.

8. A search method applied to a server, wherein the server is the server according to any one of claims 1 to 7, the method comprising: Receive query terms; The query is performed in the data storage unit of the server based on the query terms; If no query result corresponding to the query term is found, the query term is sent to the object storage system so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit. Upon receiving a query result corresponding to the query term sent by the object storage system, the query result is output.

9. A data storage device, the data storage device comprising a data storage unit, the data storage unit comprising at least two data storage fragments, the device comprising: The acquisition module is used to acquire the first identifier of the first target fragment, wherein the first target fragment is one of the at least two data storage fragments, and the first identifier of the first target fragment is used to characterize the data type of the data stored in the first target fragment; The migration module is used to migrate the data stored in the first target partition to the object storage system when the first identifier of the first target partition indicates that the first target partition is used to store hot and cold type data and the second identifier of the first target partition is obtained. The second identifier of the first target partition is used to indicate that the data in the first target partition meets the preset cooling conditions. The first receiving module is used to receive target data; The writing module is used to write the target data into the second target fragment if the second identifier of the second target fragment is not obtained; The writing module is further configured to refuse to write the target data to the second target fragment when the second identifier of the second target fragment is obtained; The data type stored in the second target fragment matches the data type of the target data.

10. The apparatus according to claim 9, wherein, The device further includes: A creation module is configured to create the second target fragment in the data storage unit if the second target fragment is not included in the at least two data storage fragments.

11. The apparatus according to claim 10, wherein, The acquisition module is further configured to acquire the first identifier of the second target fragment; The creation module is further configured to create a data mapping file between the second target partition and the object storage system when the first identifier of the second target partition indicates that the second target partition is used to store hot and cold type data. The data storage device is configured to migrate the data stored in the second target partition to the object storage system based on the data mapping file.

12. The apparatus according to claim 9, wherein, The device further includes: The processing module is used to delete the data stored in the first target segment and update the second identifier of the first target segment to a third identifier, wherein the third identifier of the first target segment is used to indicate that the data in the first target segment has been migrated.

13. The apparatus according to claim 12, wherein, The device further includes: The update module is used to update the third identifier of the first target fragment to the second identifier after a preset time interval.

14. The apparatus according to claim 9, wherein, The data storage shard has a preset code, and the preset codes of the at least two data storage shards are arranged in a preset order. The acquisition module is further configured to acquire the first identifier of the third target shard when the first identifier of the first target shard indicates that the first target shard is used to store non-hot / cold type data. The third target shard is the next data storage shard of the first target shard among the at least two data storage shards arranged in the preset order. The migration module is further configured to migrate the data stored in the third target shard to the object storage system when the first identifier of the third target shard indicates that the third target shard is used to store hot and cold type data, and the second identifier of the third target shard is obtained.

15. The apparatus according to claim 14, wherein, The acquisition module is further configured to acquire the first identifier of the third target fragment; The migration module is further configured to migrate the data stored in the third target shard to the object storage system when the first identifier of the third target shard indicates that the third target shard is used to store hot and cold type data, and the second identifier of the third target shard is obtained.

16. A search device, wherein the search device is a server as described in any one of claims 1 to 7, the method comprising: The second receiving module is used to receive query terms; The query module is used to perform a query in the data storage unit of the search device based on the query terms; The sending module is used to send the query term to the object storage system when no query result corresponding to the query term is found, so that the object storage system can perform a query based on the query term. The object storage system includes data migrated from the target shard and meeting the preset cooling conditions. The target shard is the data storage shard in the data storage unit. The output module is used to output the query result when it receives the query result corresponding to the query term sent by the object storage system.

17. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the data storage method of any one of claims 1-7, or to perform the search method of claim 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the data storage method of any one of claims 1-7, or to perform the search method of claim 8.

19. A computer program product comprising a computer program that, when executed by a processor, implements the data storage method of any one of claims 1-7, or implements the search method of claim 8.

Images