Data format online conversion method and device, equipment and medium
By performing online conversion of data from old storage engines, the problem of incompatibility of data formats when upgrading the underlying storage engine is solved, and data format conversion without downtime and user-aware migration is realized, improving the user experience.
Patent Information
- Application Number
- CN202311825267.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-12-27
- Publication Date
- 2025-06-27
Smart Images
Figure CN120216573A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technologies, and in particular, to an online data format conversion method, apparatus, device, and medium. Background Art
[0002] When upgrading the underlying storage engine, since the new and old underlying storage formats are incompatible, it is necessary to convert the original storage format into a new storage format.
[0003] In the related art, it is necessary to perform an overall relocation of all data to complete the data conversion. Therefore, it is necessary to introduce a downtime, or it is necessary for the user to perceive the existence of the migration behavior, and the conversion behavior is not automatically and dynamically completed, which affects the user experience. Summary of the Invention
[0004] To solve the above technical problems, the present disclosure provides an online data format conversion method, apparatus, device, and medium.
[0005] In a first aspect, the present disclosure provides an online data format conversion method, including:
[0006] Responding to a data conversion request, and obtaining the new storage format included in the data conversion request;
[0007] Performing data conversion processing on the old key-value pair data stored in the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format;
[0008] Saving the new key-value pair data into the new storage engine.
[0009] In a second aspect, the present disclosure provides an online data format conversion apparatus, including:
[0010] A data acquisition module, configured to respond to a data conversion request and obtain the new storage format included in the data conversion request;
[0011] A first processing module, configured to perform data conversion processing on the old key-value pair data stored in the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format;
[0012] A first saving module, configured to save the new key-value pair data into the new storage engine.
[0013] In a third aspect, the present disclosure provides an online data format conversion device, including:
[0014] A processor;
[0015] A memory, configured to store executable instructions;
[0016] Among them, the processor is used to read executable instructions from the memory and execute the executable instructions to implement the online data format conversion method in the first aspect.
[0017] In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program, which, when executed by a processor, causes the processor to implement the online data format conversion method in the first aspect.
[0018] The technical solutions provided in the embodiments of the present disclosure have the following advantages compared with the prior art:
[0019] The online data format conversion method, device, equipment and medium in the embodiments of the present disclosure can, in response to a data conversion request, obtain the new storage format included in the data conversion request, and then perform data conversion processing on the old key-value pair data stored in the old storage engine to obtain new key-value pair data, where the new key-value pair data is the new storage format, and finally save the new key-value pair data to the new storage engine. Thus, according to the data conversion request, data conversion processing can be performed on the old key-value pair data stored in the old storage engine, and the obtained new key-value pair data can be saved to the new storage engine, thereby reducing the downtime, reducing the resource occupation, and improving the user experience. Description of the Drawings
[0020] In combination with the drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic and the original elements and elements are not necessarily drawn to scale.
[0021] Figure 1 It is a schematic flowchart of an online data format conversion method provided by an embodiment of the present disclosure;
[0022] Figure 2 It is a schematic structural diagram of an online data format conversion device provided by an embodiment of the present disclosure;
[0023] Figure 3 It is a schematic structural diagram of an online data format conversion device provided by an embodiment of the present disclosure. Detailed Embodiments
[0024] The embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.
[0025] It should be understood that the various steps described in the method embodiments of the present disclosure can be executed in different orders and / or in parallel. In addition, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this regard.
[0026] As used herein, the term "including" and its variations are open-ended, that is, "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.
[0027] It should be noted that the concepts such as "first", "second", etc. mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions performed by these devices, modules or units or their interdependent relationships.
[0028] It should be noted that the modifications of "one" and "plural" mentioned in the present disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".
[0029] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only for illustrative purposes and do not limit the scope of these messages or information.
[0030] To solve the above problems, the embodiments of the present disclosure provide a method, device, equipment and medium for online conversion of data formats. The following combines Figure 1 to detail the method for online conversion of data formats provided by the embodiments of the present disclosure.
[0031] Figure 1 The flowchart of a method for online conversion of data formats provided by the embodiments of the present disclosure is shown.
[0032] In the embodiments of the present disclosure, the method for online conversion of data formats can be executed by an electronic device. Among them, the electronic device may include but is not limited to devices such as computer devices, cloud servers or cloud server clusters.
[0033] As Figure 1 shown, the method for online conversion of data formats may include the following steps.
[0034] S110. In response to a data conversion request, obtain the new storage format included in the data conversion request.
[0035] In an embodiment of the present disclosure, the electronic device may, in response to a data conversion request, obtain the new storage format included in the data conversion request.
[0036] Optionally, the data conversion request may be a request for converting the storage format of data. Among them, the data conversion request may include a new storage format.
[0037] Optionally, the new storage format may be a new data storage format.
[0038] Specifically, when a user needs to perform data conversion, a data conversion request may be sent to the electronic device. The electronic device may, in response to the data conversion request, obtain the new storage format included in the data conversion request.
[0039] S120. Perform data conversion processing on the old key-value pair data saved by the old storage engine to obtain new key-value pair data, where the new key-value pair data is the new storage format.
[0040] In an embodiment of the present disclosure, the electronic device may perform data conversion processing on the old key-value pair data saved by the old storage engine to obtain new key-value pair data.
[0041] Optionally, the old storage engine may be a storage engine that has not yet performed data conversion.
[0042] Optionally, the old key-value pair data may be data to be subjected to data conversion. Among them, the old key-value pair data is in the old storage format.
[0043] Optionally, the old storage format may be the original data storage format.
[0044] Optionally, the data conversion processing may be a process of converting the storage format of data.
[0045] Optionally, the new key-value pair data may be data obtained after data conversion. Among them, the new key-value pair data is in the new storage format.
[0046] Specifically, the electronic device may perform data conversion processing on the old key-value pair data saved by the old storage engine, thereby obtaining new key-value pair data, that is, performing data conversion processing on the old key-value pair data in the old storage format to obtain new key-value pair data in the new storage format.
[0047] S130. Save the new key-value pair data into the new storage engine.
[0048] In an embodiment of the present disclosure, the electronic device may save the new key-value pair data into the new storage engine.
[0049] Optionally, the new storage engine may be a storage engine for storing the new key-value pair data after data conversion.
[0050] Specifically, after obtaining the new key-value pair data, the electronic device can save the new key-value pair data to the new storage engine.
[0051] Thus, in the embodiments of the present disclosure, in response to a data conversion request, the new storage format included in the data conversion request can be obtained, and then data conversion processing is performed on the old key-value pair data saved in the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format. Finally, the new key-value pair data is saved to the new storage engine. Thus, according to the data conversion request, data conversion processing can be performed on the old key-value pair data saved in the old storage engine, and the obtained new key-value pair data is saved to the new storage engine, thereby reducing the downtime, reducing resource occupation, and improving the user experience.
[0052] Optionally, S120 may specifically include: performing a full scan on the old storage engine to obtain the old key-value pair data; converting the old key-value pair data from the old storage format to the new storage format to obtain the new key-value pair data.
[0053] In the embodiments of the present disclosure, the electronic device can perform a full scan on the old storage engine to obtain the old key-value pair data.
[0054] Optionally, the full scan may be to scan all the data saved in the storage engine.
[0055] Specifically, after responding to the data conversion request, the electronic device can perform a full scan on the old storage engine to obtain the old key-value pair data.
[0056] Further, after obtaining the old key-value pair data, the electronic device can convert the old key-value pair data from the old storage format to the new storage format to obtain the new key-value pair data.
[0057] Specifically, after obtaining the old key-value pair data, the electronic device can perform a storage format conversion on the old key-value pair data, that is, convert the old key-value pair data in the old storage format to the new storage format, thereby obtaining the new key-value pair data.
[0058] Optionally, S130 may specifically include: generating a corresponding file set according to the new key-value pair data and saving the file set to the underlying layer of the new storage engine.
[0059] In the embodiments of the present disclosure, the electronic device can generate a corresponding file set according to the new key-value pair data and save the file set to the underlying layer of the new storage engine.
[0060] Optionally, the file set may be a set composed of multiple files.
[0061] Optionally, the underlying layer of the new storage engine may be the underlying layer of the LSMTree (Log Structured Merge Tree).
[0062] Specifically, after obtaining the new key-value pair data, the electronic device generates a corresponding file set according to the new key-value pair data, and saves the file set to the underlying layer of the new storage engine, such as saving it to the underlying layer of the LSMTree.
[0063] Optionally, after S130, the data format online conversion method may further include: generating secondary data corresponding to the new key-value pair data according to the new storage format; creating a column family in the new storage engine, and saving the secondary data in the column family.
[0064] In the embodiments of the present disclosure, the electronic device may generate secondary data corresponding to the new key-value pair data according to the new storage format.
[0065] Optionally, the secondary data may be statistical data corresponding to the new key-value pair data. For example, the secondary data may be an index, an aggregation view, etc., which is not limited herein.
[0066] Specifically, the electronic device may generate secondary data corresponding to the new key-value pair data according to the new storage format, such as generating a corresponding index, an aggregation view, etc.
[0067] Furthermore, the electronic device may create a column family in the new storage engine, and save the secondary data in the column family.
[0068] Optionally, a column family may isolate the storage of different columns in a table, and different columns have different storage directories.
[0069] Specifically, after obtaining the secondary data, the electronic device may create a new column family in the new storage engine, and save the secondary data to the column family.
[0070] Optionally, when performing S120, the data format online conversion method may further include: in response to a data processing request, performing a read operation and a delete operation on the old storage engine and the new storage engine, and performing a write operation on the new storage engine.
[0071] In the embodiments of the present disclosure, the electronic device may, in response to a data processing request, perform a read operation and a delete operation on the old storage engine and the new storage engine, and perform a write operation on the new storage engine.
[0072] Optionally, the data processing request may be a request for processing key-value pair data. For example, the data processing request may include a read request, a write request, a delete request, etc., which is not limited herein.
[0073] Optionally, the read operation may be an operation for reading key-value pair data.
[0074] Optionally, the delete operation may be an operation for deleting key-value pair data.
[0075] Optionally, the write operation may be an operation for writing key-value pair data.
[0076] Specifically, during the process of data format conversion, when the user wants to make a data processing request, the electronic device may respond to the data processing request and perform a read operation and a delete operation on the old storage engine and the new storage engine, that is, perform a double-read and double-delete operation on the old storage engine and the new storage engine, and perform a write operation on the new storage engine.
[0077] Optionally, after S120, the data format online conversion method may further include: responding to a data processing request, and performing a read operation, a delete operation, and a write operation on the new storage engine.
[0078] In an embodiment of the present disclosure, the electronic device may respond to a data processing request and perform a read operation, a delete operation, and a write operation on the new storage engine.
[0079] Specifically, after the data format conversion is completed, when the user wants to make a data processing request at this time, the electronic device may respond to the data processing request and perform a read operation, a delete operation, and a write operation on the new storage engine.
[0080] Figure 2 The structure diagram of a data format online conversion device provided by an embodiment of the present disclosure is shown.
[0081] As Figure 2 shown, the data format online conversion device 200 may include a data acquisition module 210, a first processing module 220, and a first storage module 230.
[0082] The data acquisition module 210 may be configured to obtain the new storage format included in the data conversion request in response to the data conversion request.
[0083] The first processing module 220 may be configured to perform data conversion processing on the old key-value pair data stored in the old storage engine to obtain new key-value pair data, and the new key-value pair data is the new storage format.
[0084] The first storage module 230 may be configured to store the new key-value pair data in the new storage engine.
[0085] Thus, in the embodiments of the present disclosure, in response to a data conversion request, the new storage format included in the data conversion request can be obtained, and then data conversion processing is performed on the old key-value pair data saved by the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format. Finally, the new key-value pair data is saved to the new storage engine. Thus, according to the data conversion request, data conversion processing can be performed on the old key-value pair data saved by the old storage engine, and the obtained new key-value pair data can be saved to the new storage engine, thereby reducing the downtime, reducing resource occupation, and improving the user experience.
[0086] In some embodiments of the present disclosure, the old key-value pair data can be in the old storage format.
[0087] In some embodiments of the present disclosure, the first processing module 220 may specifically include a data scanning unit and a data conversion unit.
[0088] The data scanning unit can be used to perform a full scan on the old storage engine to obtain the old key-value pair data.
[0089] The data conversion unit can be used to convert the old key-value pair data from the old storage format to the new storage format to obtain the new key-value pair data.
[0090] In some embodiments of the present disclosure, the first saving module 230 may specifically include a data storage unit.
[0091] The data storage unit can be used to generate a corresponding file set according to the new key-value pair data and save the file set to the underlying layer of the new storage engine.
[0092] In some embodiments of the present disclosure, the data format online conversion device 200 may further include a data generation module and a second saving module.
[0093] The data generation module can be used to generate secondary data corresponding to the new key-value pair data according to the new storage format after saving the new key-value pair data to the new storage engine.
[0094] The second saving module can be used to create a column family in the new storage engine and save the secondary data in the column family.
[0095] In some embodiments of the present disclosure, the data format online conversion device 200 may further include a second processing module.
[0096] The second processing module can be used to perform read operations and delete operations on the old storage engine and the new storage engine, and perform write operations on the new storage engine in response to a data processing request when performing data conversion processing on the old key-value pair data saved by the old storage engine.
[0097] In some embodiments of the present disclosure, the data format online conversion device 200 may further include a third processing module.
[0098] The third processing module can be used to perform read operations, delete operations, and write operations on the new storage engine in response to a data processing request after performing data conversion processing on the old key-value pair data saved by the old storage engine.
[0099] It should be noted that Figure 2 the shown data format online conversion device 200 can execute Figure 1 each step in the method embodiments shown, and achieve Figure 1 each process and effect in the method embodiments shown, which will not be elaborated here.
[0100] Figure 3 FIG. shows a schematic structural diagram of a data format online conversion device provided by an embodiment of the present disclosure.
[0101] In some embodiments of the present disclosure, Figure 3 the shown data format online conversion device may be an electronic device. Specifically, the electronic device may include, but is not limited to, devices such as computer devices, cloud servers, or cloud server clusters.
[0102] Such as Figure 3 shown, the data format online conversion device may include a processor 301 and a memory 302 storing computer program instructions.
[0103] Specifically, the above-mentioned processor 301 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application.
[0104] The memory 302 may include a mass storage for information or instructions. By way of example and not limitation, the memory 302 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 302 may include removable or non-removable (or fixed) media. Where appropriate, the memory 302 may be internal or external to the integrated gateway device. In a particular embodiment, the memory 302 is a non-volatile solid-state memory. In a particular embodiment, the memory 302 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically alterable ROM (EAROM), or a flash memory, or a combination of two or more of these.
[0105] The processor 301 reads and executes the computer program instructions stored in the memory 302 to perform the steps of the data format online conversion method provided by the embodiments of the present disclosure.
[0106] In one example, the data format online conversion device may further include a transceiver 303 and a bus 304. Among them, as Figure 3 shown, the processor 301, the memory 302, and the transceiver 303 are connected through the bus 304 and complete communication with each other.
[0107] The bus 304 includes hardware, software, or both. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Extended Industry Standard Architecture (EISA) bus, a Front Side BUS (FSB), a Hyper Transport (HT) interconnect, an Industrial Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local Bus (VLB) bus, or other suitable bus or a combination of two or more of these. Where appropriate, the bus 304 may include one or more buses. Although embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.
[0108] Embodiments of the present disclosure also provide a computer-readable storage medium that may store a computer program, which when executed by a processor, causes the processor to implement the data format online conversion method provided by the embodiments of the present disclosure.
[0109] The above storage medium may, for example, include a memory 302 storing computer program instructions, and the above instructions may be executed by a processor 301 of the data format online conversion device to complete the data format online conversion method provided by the embodiments of the present disclosure. Optionally, the storage medium may be a non-transitory computer-readable storage medium. For example, the non-transitory computer-readable storage medium may be a ROM, a Random Access Memory (RAM), a Compact Disc ROM (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.
[0110] It should be noted that in this text, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising" is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device.
[0111] The above are only specific embodiments of the present disclosure, enabling those skilled in the art to understand or implement the present disclosure. Various modifications to these embodiments will be obvious to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure will not be limited to these embodiments described herein, but rather will conform to the broadest scope consistent with the principles and novel features disclosed herein.
Claims
1. An online data format conversion method, characterized in that, Including: In response to a data conversion request, obtain the new storage format included in the data conversion request; Perform data conversion processing on the old key-value pair data saved by the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format; Save the new key-value pair data into the new storage engine.
2. The method according to claim 1, characterized in that, The old key-value pair data is in the old storage format.
3. The method according to claim 2, wherein The performing data conversion processing on the old key-value pair data saved by the old storage engine to obtain new key-value pair data includes: Perform a full scan on the old storage engine to obtain the old key-value pair data; Convert the old key-value pair data from the old storage format to the new storage format to obtain the new key-value pair data.
4. The method according to claim 1, wherein The saving the new key-value pair data into the new storage engine includes: Generate a corresponding file set according to the new key-value pair data and save the file set to the underlying layer of the new storage engine.
5. The method according to claim 1, characterized in that, After saving the new key-value pair data into the new storage engine, the method further includes: Generate secondary data corresponding to the new key-value pair data according to the new storage format; Create a column family in the new storage engine and save the secondary data in the column family.
6. The method according to claim 1, wherein When performing data conversion processing on the old key-value pair data saved by the old storage engine, the method further includes: In response to a data processing request, perform read operations and delete operations on the old storage engine and the new storage engine, and perform write operations on the new storage engine.
7. The method according to claim 1, characterized in that, After performing data conversion processing on the old key-value pair data saved by the old storage engine, the method further includes: In response to a data processing request, perform read operations, delete operations, and write operations on the new storage engine.
8. An online data format conversion device, characterized in that, Including: A data acquisition module, configured to obtain the new storage format included in the data conversion request in response to a data conversion request; A first processing module, configured to perform data conversion processing on the old key-value pair data saved by the old storage engine to obtain new key-value pair data, where the new key-value pair data is in the new storage format; A first saving module, configured to save the new key-value pair data into the new storage engine.
9. An online data format conversion device, characterized in that, Including: A processor; A memory, configured to store executable instructions; Wherein, the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the data format online conversion method according to any one of claims 1-7 above.
10. A non-volatile computer-readable storage medium, characterized in that, The storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to implement the data format online conversion method according to any one of claims 1-7 above.