Data storage method and device, equipment and storage medium

Abstract

The invention relates to a data storage method and device, equipment and a storage medium, and relates to the technical field of data storage. According to the method and the device, the storage mode of the target file data can be determined according to the tag information corresponding to the target file data, and the target file data can be stored in a data deduplication mode only when the tag information corresponding to the target file data is the target tag information; and under the condition that the tag information corresponding to the target file data is not the target tag information, other modes except the data deduplication mode are adopted to store the target file data, that is, data deduplication processing is only carried out on the target file data of the specific tag, the data processing amount of data deduplication is reduced, and the efficiency of data deduplication processing is improved. Compared with the prior art, computing resources and computing time needed in the data storage process are effectively reduced, the number of indexes built in the data deduplication process is reduced, the memory space needing to be occupied is reduced, the data processing performance of a computer system is improved, and the data storage efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of data storage technology, and in particular to a data storage method, apparatus, device and storage medium. Background Technology

[0002] In recent years, with the rapid development of machine learning, more and more large models with various functions have emerged. Training a model requires a large amount of file data, which constitutes the model training data. Before training the model, the file data required for model training needs to be stored in a specific storage space, and then the file data is read from the storage space for model training.

[0003] Currently, in the process of storing the file data required for model training into the storage space, data deduplication technology is used to process the file data required for model training. This process consumes a lot of computing resources and computing time, and the index built by data deduplication requires a lot of memory space, which leads to a decrease in the data processing performance of the computer system and reduces the efficiency of data processing and storage. Summary of the Invention

[0004] To address the problems existing in the aforementioned related technologies, this disclosure provides a data storage method, apparatus, device, and storage medium.

[0005] A first aspect of this disclosure provides a data storage method, the method comprising:

[0006] In response to receiving target file data, determine the tag information corresponding to the target file data;

[0007] Identify whether the label information is the target label information;

[0008] When the tag information is the target tag information, the target file data is stored in the storage space based on the data deduplication method;

[0009] When the tag information is not the target tag information, the target file data is stored in the storage space based on a storage method different from the data deduplication method.

[0010] Optionally, the tag information for determining the target file data mentioned above includes:

[0011] Write the metadata of the target file data into the inode of the target file data;

[0012] Search for the tag information of the target file data in the inode of the target file data.

[0013] Optionally, when the tag information is target tag information, storing the target file data in the storage space based on the data deduplication method includes:

[0014] When the tag information is the target tag information, the target file data is divided into multiple data blocks, and a target fingerprint table corresponding to the target file data is created. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0015] The fingerprints in the target fingerprint table are compared with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint that is different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space.

[0016] Based on the data block storage method corresponding to the first target fingerprint, the first data block corresponding to the first target fingerprint in the target file data is stored;

[0017] Based on the data block storage method corresponding to the second target fingerprint, the second data block corresponding to the second target fingerprint in the target file data is stored.

[0018] Optionally, the above-mentioned data block storage method based on the first target fingerprint stores the first data block corresponding to the first target fingerprint in the target file data, including:

[0019] For each first target fingerprint corresponding to the first data block in the target file data, create index information for the first data block. The index information points to the target data block corresponding to the first target fingerprint that has been stored in the storage space.

[0020] Write the index information of each first data block into the index node of the target file data;

[0021] Delete the first data block of each data block in the target file.

[0022] Optionally, the above-mentioned data block storage method based on the second target fingerprint stores the second data block corresponding to the second target fingerprint in the target file data, including:

[0023] For each second target fingerprint corresponding to the target file data, the second data block is written into the storage space;

[0024] Each second target fingerprint is added to the first fingerprint table to obtain the updated first fingerprint table.

[0025] Optionally, when the tag information is not the target tag information, the above-mentioned storage method for storing the target file data in the storage space based on a storage method different from the data deduplication method includes:

[0026] If the tag information is not the target tag information, the target file data is divided into multiple data blocks;

[0027] Write each data block from the target file into the storage space;

[0028] Create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0029] The target fingerprint table is added to the first fingerprint table corresponding to the file data already stored in the storage space to obtain the updated first fingerprint table.

[0030] Optionally, the target file data is divided into multiple data blocks, including:

[0031] The granularity of the target data block is determined based on the size of the target file data, and the granularity of the target data block is greater than or equal to the preset minimum granularity.

[0032] The target file data is divided into multiple data blocks according to the granularity of the target data block.

[0033] Optionally, after determining the tag information corresponding to the target file data, the method further includes:

[0034] If the tag information corresponding to the target file data does not exist, create the tag information corresponding to the target file data;

[0035] Write the tag information into the inode of the target file data.

[0036] Optionally, the tag information corresponding to the target file data created above includes:

[0037] Using a pre-defined tag creation tool, tag information corresponding to the target file data is created based on the content of the target file data.

[0038] Optionally, the tag information corresponding to the target file data created above includes:

[0039] In response to a tag creation operation on target file data, retrieve the tag information created by the tag creation operation;

[0040] The tag information created by the tag creation operation is identified as the tag information corresponding to the target file data.

[0041] Optionally, after determining the tag information corresponding to the target file data, the method further includes:

[0042] If the tag information corresponding to the target file data does not exist, the target file data is stored in the storage space based on the data deduplication method.

[0043] Optionally, the target label information mentioned above includes at least one of the model training code label and the model training dataset label.

[0044] A second aspect of this disclosure provides a data storage device, the device comprising:

[0045] The determination module is used to determine the tag information corresponding to the target file data in response to receiving target file data;

[0046] The identification module is used to identify whether the label information is the target label information;

[0047] The first storage module is used to store the target file data into the storage space based on the data deduplication method when the tag information is the target tag information;

[0048] The second storage module is used to store the target file data in the storage space using a storage method different from the data deduplication method when the tag information is not the target tag information.

[0049] Optionally, the above-mentioned determining module includes:

[0050] The write submodule is used to write the metadata of the target file data into the inode of the target file data;

[0051] The search submodule is used to search for tag information of target file data in the inode of the target file data.

[0052] Optionally, the first storage module mentioned above includes:

[0053] The partitioning and creation submodule is used to divide the target file data into multiple data blocks when the tag information is the target tag information, and to create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0054] The comparison submodule is used to compare the fingerprints in the target fingerprint table with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint that is different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data stored in the storage space.

[0055] The first storage submodule is used to store the first data block corresponding to the first target fingerprint in the target file data based on the data block storage method corresponding to the first target fingerprint;

[0056] The second storage submodule is used to store the second data block corresponding to the second target fingerprint in the target file data based on the data block storage method corresponding to the second target fingerprint.

[0057] Optionally, the first storage submodule mentioned above includes:

[0058] The index creation unit is used to create index information for the first data block corresponding to each first target fingerprint in the target file data. The index information points to the target data block corresponding to the first target fingerprint that has been stored in the storage space.

[0059] The first writing unit is used to write the index information of each first data block into the index node of the target file data;

[0060] The deletion unit is used to delete each first data block in the target file data.

[0061] Optionally, the second storage submodule mentioned above includes:

[0062] The second writing unit is used to write the second data block corresponding to each second target fingerprint in the target file data into the storage space.

[0063] The update unit is used to add each second target fingerprint to the first fingerprint table to obtain the updated first fingerprint table.

[0064] Optionally, the second storage module mentioned above includes:

[0065] The data block partitioning submodule is used to divide the target file data into multiple data blocks when the tag information is not the target tag information;

[0066] The data block writing submodule is used to write each data block from the target file into the storage space;

[0067] The fingerprint table creation submodule is used to create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0068] The update submodule is used to add the target fingerprint table to the first fingerprint table corresponding to the file data already stored in the storage space, so as to obtain the updated first fingerprint table.

[0069] Optionally, the above-mentioned sub-modules for creating data blocks or creating sub-modules include:

[0070] The granularity determination unit is used to determine the granularity of the target data block corresponding to the target file data based on the size of the target file data. The granularity of the target data block is greater than or equal to the preset minimum granularity.

[0071] A partitioning unit is used to divide the target file data into multiple data blocks according to the granularity of the target data block.

[0072] Optionally, the above data storage device includes:

[0073] The tag creation module is used to create tag information for target file data when the tag information for the target file data does not exist.

[0074] The tag writing module is used to write tag information into the inode of the target file data.

[0075] Optionally, the above-mentioned tag creation module includes:

[0076] The tag creation submodule is used to create tag information corresponding to the target file data based on the content of the target file data using a preset tag creation tool.

[0077] Optionally, the above-mentioned tag creation module includes:

[0078] The `get` submodule is used to respond to tag creation operations on target file data and retrieve tag information created by the tag creation operation.

[0079] The tag determination submodule is used to determine the tag information created by the tag creation operation as the tag information corresponding to the target file data.

[0080] Optionally, the above data storage device includes:

[0081] The third storage module is used to store the target file data in the storage space based on the data deduplication method when the tag information corresponding to the target file data does not exist.

[0082] Optionally, the target label information mentioned above includes at least one of the model training code label and the model training dataset label.

[0083] A third aspect of this disclosure provides a computer device including a memory and a processor, wherein the memory stores a computer program that, when executed by the processor, can implement the data storage method of the first aspect described above.

[0084] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the data storage method of the first aspect described above.

[0085] The technical solution provided in this disclosure has the following advantages compared with the prior art:

[0086] This disclosure, in response to receiving target file data, determines the tag information corresponding to the target file data; identifies whether the tag information is target tag information; if the tag information is target tag information, stores the target file data in storage space based on data deduplication; if the tag information is not target tag information, stores the target file data in storage space based on a storage method different from data deduplication. Thus, the storage method of the target file data can be determined based on the tag information corresponding to the target file data. Data deduplication is only used to store the target file data when the tag information is target tag information, and other methods are used to store the target file data when the tag information is not target tag information. In other words, this disclosure only performs data deduplication processing on target file data with specific tags, reducing the amount of data processing involved in data deduplication, effectively reducing the computing resources and time required for data storage, reducing the number of indexes built during data deduplication, reducing the memory space required, improving the data processing performance of the computer system, and improving data storage efficiency.

[0087] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0088] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0089] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0090] Figure 1 This is a flowchart of a data storage method provided in an embodiment of this disclosure;

[0091] Figure 2 This is a flowchart of another data storage method provided in this embodiment of the disclosure;

[0092] Figure 3 This is a schematic diagram of a target fingerprint table corresponding to target file data provided in an embodiment of this disclosure;

[0093] Figure 4 This is a schematic diagram of a data deduplication technology provided in an embodiment of this disclosure;

[0094] Figure 5This is a flowchart of yet another data storage method provided in this disclosure embodiment;

[0095] Figure 6 This is a flowchart of another data storage method provided in this disclosure embodiment;

[0096] Figure 7 This is a flowchart of another data storage method provided in this disclosure embodiment;

[0097] Figure 8 This is a schematic diagram of the structure of a data storage device provided in an embodiment of this disclosure;

[0098] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure. Detailed Implementation

[0099] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0100] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0101] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0102] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0103] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0104] In recent years, with the rapid development of machine learning, more and more large models with various functions have emerged. Training a model requires a large amount of file data, mainly including model training code data, model training dataset, log file data, checkpoint data, etc. These file data constitute the model training data. Before model training, the file data required for model training needs to be stored in a specific storage space, and then the file data is read from the storage space for model training.

[0105] Currently, in the process of storing the file data required for model training into the storage space, data deduplication technology is used to process the file data required for model training. This process consumes a lot of computing resources and computing time, and the index built by data deduplication requires a lot of memory space, which leads to a decrease in the data processing performance of the computer system and reduces the efficiency of data processing and storage.

[0106] This disclosure determines the storage method of target file data based on the tag information corresponding to the target file data. Data deduplication is only used to store the target file data when the tag information is the target tag information. Otherwise, other methods besides data deduplication are used to store the target file data. In other words, this disclosure only performs data deduplication on target file data with specific tags, reducing the amount of data processing for data deduplication, effectively reducing the computing resources and time required for data storage, reducing the number of indexes built in the data deduplication process, reducing the memory space required, improving the data processing performance of the computer system, and improving the data storage efficiency.

[0107] To better understand the inventive concept of the embodiments of this disclosure, the technical solutions of the embodiments of this disclosure will be described below in conjunction with exemplary embodiments.

[0108] The data storage method provided in this disclosure can be executed by a computer device. This device can be understood as any device with processing and computing capabilities, including but not limited to electronic devices such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals, wearable devices, digital TVs, desktop computers, smart home devices, etc.

[0109] Figure 1This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing functions and processing capabilities. Figure 1 As shown, the data storage method provided in this embodiment includes the following steps:

[0110] Step 110: In response to receiving the target file data, determine the tag information of the target file data.

[0111] In this embodiment of the disclosure, the target file data can be understood as file data used for model training.

[0112] The target file data may include model training code data, model training dataset, model training log file data, checkpoint data, etc.

[0113] Tag information can be understood as a type of annotation data, which can be represented by a string of characters. The tag information of target file data can be understood as data that characterizes the content type of the target file data.

[0114] For example, the label information of the target file data can include model training code labels, model training dataset labels, log labels, checkpoint labels, etc.

[0115] Among them, the model training code label can be understood as data that represents the content type of the model training code data.

[0116] The labels for the model training dataset can be understood as data that characterizes the content type of the model training dataset.

[0117] Log tags can be understood as data that represents the content type of the log file data used in model training.

[0118] Checkpoint labels can be understood as data that represents the content type of the checkpoint data used in model training.

[0119] In this embodiment of the disclosure, the computer device can receive target file data and then, in response to receiving the target file data, determine the tag information corresponding to the target file data.

[0120] In some embodiments, determining the tag information corresponding to the target file data may include S11-S12:

[0121] S11. Write the metadata of the target file data into the inode of the target file data.

[0122] Metadata of target file data can be understood as data that describes the target file data, that is, descriptive information about the target file data.

[0123] The metadata of the target file data can include the tag information, data structure, source, attributes, quality, processing rules and management information of the target file data.

[0124] An inode can be understood as the core data structure in a computer device's file system. It can be used to store metadata of file data and index information of the file data's storage address.

[0125] In a computer device's file system, each file data corresponds to an inode.

[0126] In this embodiment of the disclosure, after receiving the target file data, the computer device can create an index node for the target file data, then extract the metadata of the target file data, and write the metadata of the target file data into the index node of the target file data.

[0127] S12. Search for the tag information of the target file data in the inode of the target file data.

[0128] In this embodiment of the disclosure, since the metadata of the target file data includes the tag information corresponding to the target file data, after the metadata of the target file data is written into the index node of the target file data, the computer device can search for the tag information of the target file data in the index node of the target file data.

[0129] Therefore, the tag information of the target file data can be found from the inode of the target file data, and the tag information of the file data can be accurately and quickly determined.

[0130] Step 120: Identify whether the label information is the target label information.

[0131] In this embodiment, the target tag information can be understood as pre-set tag information corresponding to file data that is allowed to be stored using data deduplication. The target tag information can be set as needed, and is not limited here.

[0132] The computer device pre-stores target tag information. After determining the tag information corresponding to the target file data, the computer device can identify whether the tag information corresponding to the target file data is the target tag information.

[0133] For example, target tag information can be tag information containing a specific value or certain specific values. Computer devices can identify whether a tag is target tag information by determining whether a specific value or certain specific values ​​exist in the tag information corresponding to the target file data. If a specific value or certain specific values ​​are present in the tag information, it can be determined that the tag information is target tag information; if a specific value or certain specific values ​​are not present in the tag information, it can be determined that the tag information is not target tag information.

[0134] Specific values ​​can be set as needed; for example, a specific value can be the model training code, but this is not limited here.

[0135] For example, target tag information can be tag information with specific rules or patterns. Computer devices can identify whether the tag information corresponding to the target file data is target tag information by determining whether the tag information has specific rules or patterns. If the tag information has specific rules or patterns, it can be determined that the tag information is target tag information; if the tag information does not have specific rules or patterns, it can be determined that the tag information is not target tag information.

[0136] Specific rules or patterns can be set as needed. For example, a specific rule or pattern can be set to whether the label starts with "dataset_", where dataset represents a dataset, and this is not limited here.

[0137] For example, target tag information can be tag information with specific attributes. Computer devices can identify whether the tag information corresponding to the target file data is target tag information by determining whether the tag information has specific attributes. If the tag information has specific attributes, it can be determined that the tag information is target tag information; if the tag information does not have specific attributes, it can be determined that the tag information is not target tag information.

[0138] Specific attributes can be set as needed. For example, specific attributes may include tag priority, tag type, etc., which are not limited here.

[0139] Step 130: If the tag information is the target tag information, store the target file data in the storage space based on the data deduplication method.

[0140] In this embodiment of the disclosure, when the tag information corresponding to the target file data is target tag information, the computer device can store the target file data in the storage space based on the data deduplication method.

[0141] Data deduplication, also known as data duplication removal, can be understood as a technique to optimize storage space by eliminating redundant data copies. Its core idea is to store identical data only once, with subsequent duplicate data only retaining references. Specifically, data deduplication is a data reduction technique based on disk backup systems, replacing duplicate data blocks with index information.

[0142] Step 140: If the tag information is not the target tag information, store the target file data in the storage space based on a storage method different from the data deduplication method.

[0143] In this embodiment of the disclosure, when the tag information corresponding to the target file data is not the target tag information, the computer device can store the target file data in the storage space based on a storage method different from the data deduplication method.

[0144] Storage space can be understood as the storage space corresponding to the file storage system in a computer device. For example, storage space can be the hard drive storage space in a computer device.

[0145] This embodiment of the disclosure, in response to receiving target file data, determines the tag information corresponding to the target file data; identifies whether the tag information is target tag information; if the tag information is target tag information, stores the target file data in the storage space based on a data deduplication method; if the tag information is not target tag information, stores the target file data in the storage space based on a storage method different from the data deduplication method. Thus, the storage method of the target file data can be determined based on the tag information corresponding to the target file data. Data deduplication is only used to store the target file data when the tag information is target tag information, while other methods besides data deduplication are used to store the target file data when the tag information is not target tag information.

[0146] In other words, this disclosure only performs data deduplication on target file data with specific tags, reducing the amount of data processing required for data deduplication, effectively reducing the computing resources and time required for data storage, reducing the number of indexes built during the data deduplication process, reducing the memory space required, improving the data processing performance of the computer system, and improving the data storage efficiency.

[0147] In some embodiments of this disclosure, the target label information may include at least one of model training code label and model training dataset label.

[0148] Therefore, this embodiment of the disclosure can perform data deduplication only on the target file data (model training code data and / or model training dataset) corresponding to the model training code label and / or model training dataset label, while storing data with other labels (such as log file data, checkpoint data, etc.) using storage methods other than data deduplication. Since the model training code data and model training dataset are the main data in the model training data, using data deduplication to perform data deduplication on the model training code data and model training dataset can avoid fingerprint calculation comparison and data deduplication on some unimportant files or files with little relevance to model training (such as log file data, checkpoint data, etc.), saving data processing resources, reducing the amount of data processing for data deduplication, effectively reducing the computing resources and computing time required for data storage, reducing the number of indexes built in the data deduplication process, reducing the memory space required, improving the data processing performance of the computer system, and improving the data storage efficiency.

[0149] Furthermore, the training code for most models is generally based on popular open-source frameworks (such as Megatron) with slight modifications. Therefore, the training code for most models is quite similar, and it can be assumed that more than 90% of the code is identical. Many datasets are publicly available and similar; datasets from the same organization are often identical (e.g., different developers and researchers may train simultaneously). Therefore, the datasets for most models are also quite similar. Thus, this embodiment of the disclosure only performs data deduplication on the target file data (model training code data and / or model training dataset) corresponding to the model training code labels and / or model training dataset labels, which can effectively reduce the storage space occupied.

[0150] Figure 2 This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing functions and processing capabilities. Figure 2 As shown, the data storage method provided in this embodiment includes the following steps:

[0151] Step 210: In response to receiving the target file data, determine the tag information corresponding to the target file data.

[0152] Step 220: Identify whether the label information is the target label information.

[0153] The content of steps 210-220 in this embodiment can be referred to the above. Figure 1 The corresponding content in [the document] will not be repeated here.

[0154] Step 230: If the tag information is the target tag information, divide the target file data into multiple data blocks and create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block of the target file data.

[0155] In this embodiment of the disclosure, when the tag information corresponding to the target file data is target tag information, the computer device can divide the target file data into multiple data blocks and create a target fingerprint table corresponding to the target file data. The target fingerprint table can include the target fingerprints corresponding to each data block in the target file data.

[0156] The target fingerprint corresponding to a data block can be understood as information that can uniquely identify the data block. For example, the target fingerprint corresponding to a data block can be represented by a string composed of several characters.

[0157] The target fingerprint corresponding to a data block can be determined based on the content of the data block.

[0158] For example, Figure 3 This is a schematic diagram of a target fingerprint table corresponding to target file data provided in an embodiment of this disclosure, such as... Figure 3 As shown, the data blocks in the target file data include data block 21, data block 22, data block 23, and data block 24. The target fingerprint table corresponding to the target file data includes fingerprint 1 corresponding to data block 21, fingerprint 2 corresponding to data block 22, fingerprint 3 corresponding to data block 23, and fingerprint 4 corresponding to data block 24.

[0159] Step 240: Compare the fingerprints in the target fingerprint table with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint that is different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data stored in the storage space.

[0160] In this embodiment of the present disclosure, after obtaining the target fingerprint table corresponding to the target file data, the computer device can compare the fingerprints in the target fingerprint table with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table.

[0161] The first target fingerprint is the fingerprint that is identical to the fingerprint in the first fingerprint table. Identical fingerprints correspond to identical data blocks. Identical data blocks can be understood as data blocks containing identical data.

[0162] The second target fingerprint is a fingerprint that differs from the fingerprint in the first fingerprint table. Different fingerprints correspond to different data blocks. Different data blocks can be understood as data blocks containing different data.

[0163] The first fingerprint table can be understood as a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space of a computer device.

[0164] Step 250: Based on the data block storage method corresponding to the first target fingerprint, store the first data block corresponding to the first target fingerprint in the target file data.

[0165] In this embodiment of the present disclosure, after determining the first target fingerprint in the target fingerprint table, the computer device can store the first data block corresponding to the first target fingerprint in the target file data based on the data block storage method corresponding to the first target fingerprint.

[0166] In some embodiments, storing the first data block corresponding to the first target fingerprint in the target file data based on the above-described data block storage method may include steps 2501-2503:

[0167] Step 2501: For each first target fingerprint corresponding to the first data block in the target file data, create index information for the first data block. The index information points to the target data block corresponding to the first target fingerprint that has been stored in the storage space.

[0168] In this embodiment of the disclosure, the computer device can create index information for each first data block corresponding to a first target fingerprint in the target file data. The index information can point to the target data block corresponding to the first target fingerprint that has been stored in the storage space.

[0169] In other words, the index information of the first data block corresponding to the first target fingerprint can be understood as a reference identifier pointing to the target data block corresponding to the first target fingerprint that has been stored in the storage space. That is to say, the index information of the data block can indicate the storage address of the data block in the storage space.

[0170] Step 2502: Write the index information of each first data block into the index node of the target file data.

[0171] In this embodiment of the disclosure, in the file system of the computer device, each file data corresponds to an inode.

[0172] After obtaining the index information of each first data block, the computer device can write the index information of each first data block into the index node of the target file data. At this time, the index node of the target file data can include the metadata of the target file data and the index information of the storage address of each first data block of the target file data.

[0173] Step 2503: Delete each of the first data blocks in the target file data.

[0174] In this embodiment of the disclosure, since the storage space already contains target data blocks identical to each first data block, the computer device can delete each first data block from the target file data after determining the index information of each first data block.

[0175] For example, Figure 4 This is a schematic diagram of a data deduplication technology provided in an embodiment of this disclosure, such as... Figure 4 As shown, the data blocks in the target file data include data block 21, data block 22, data block 23, and data block 24, and the file data already stored in the storage space includes data block 11, data block 12, data block 13, data block 14, and data block 15; the first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space, and the first fingerprint table includes the fingerprints corresponding to data block 11, data block 12, data block 13, data block 14, and data block 15;

[0176] If the target fingerprint corresponding to data block 22 in the target file data is the same as the fingerprint corresponding to data block 11 of the stored file data, then it can be determined that data block 22 is the same as data block 11, and data block 22 is the first data block. The computer device can create index information 1 for data block 22, which points to the data block 11 already stored in the storage space. The index information 1 is written into the index node of the target file data, and data block 22 is deleted.

[0177] If the target fingerprint corresponding to data block 23 in the target file data is the same as the fingerprint corresponding to data block 14 of the stored file data, then it can be determined that data block 23 is the same as data block 14, and data block 23 is the first data block. The computer device can create index information 2 for data block 23, which points to the data block 14 already stored in the storage space. The index information 2 is written into the index node of the target file data, and data block 23 is deleted.

[0178] Therefore, during the data deduplication process, an index can be built for the first data block in the target file data that is the same as the data block already stored in the storage space. The index points to the data block already stored in the storage space and the first data block is deleted. This can realize the data deduplication processing of the target file data with specific tags, which can reduce the storage space required for the target file data and save storage resources.

[0179] Step 260: Based on the data block storage method corresponding to the second target fingerprint, store the second data block corresponding to the second target fingerprint in the target file data.

[0180] In this embodiment of the present disclosure, after determining the second target fingerprint in the target fingerprint table, the computer device can store the second data block corresponding to the second target fingerprint in the target file data based on the data block storage method corresponding to the second target fingerprint.

[0181] In some embodiments, the above-described storage method based on the data block corresponding to the second target fingerprint in the target file data for storing the second data block may include steps 2601-2602:

[0182] Step 2601: For each second target fingerprint in the target file data, write the second data block into the storage space.

[0183] After determining the second target fingerprint in the target fingerprint table, the computer device can write the second data block corresponding to each second target fingerprint in the target file data into the storage space of the computer device.

[0184] Step 2602: Add each second target fingerprint to the first fingerprint table to obtain the updated first fingerprint table.

[0185] After writing each second data block into the storage space, the computer device can add the second target fingerprint corresponding to each second data block to the first fingerprint table, thus obtaining an updated first fingerprint table. The first fingerprint table can be understood as a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the computer device's storage space.

[0186] For example, such as Figure 4 As shown, the target fingerprint corresponding to data block 21 in the target file data is different from the fingerprints corresponding to each data block of the file data already stored in the storage space. Therefore, it can be determined that data block 21 is different from the stored file data. Data block 21 is the second data block, and the computer device can write data block 21 into the storage space.

[0187] If the target fingerprint corresponding to data block 24 in the target file data is different from the fingerprints corresponding to each data block of the file data already stored in the storage space, then it can be determined that data block 24 is different from the stored file data. Data block 24 is the second data block, and the computer device can write data block 24 into the storage space.

[0188] Then the computer device can add the target fingerprint corresponding to data block 21 and the target fingerprint corresponding to data block 24 to the first fingerprint table to obtain the updated first fingerprint table. The updated first fingerprint table includes the target fingerprint corresponding to data block 21, the target fingerprint corresponding to data block 24, the fingerprint corresponding to data block 11, the fingerprint corresponding to data block 12, the fingerprint corresponding to data block 13, the fingerprint corresponding to data block 14, and the fingerprint corresponding to data block 15.

[0189] Therefore, during the data deduplication process, a second data block that is different from the data block already stored in the storage space can be directly written into the storage space, and the fingerprint table of the file data already stored in the storage space can be updated, thereby improving the data processing performance of the computer system and improving the data storage efficiency.

[0190] In this embodiment, when the tag information corresponding to the target file data is target tag information, the target file data is divided into multiple data blocks, and a target fingerprint table corresponding to the target file data is created. The fingerprints in the target fingerprint table are compared with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space. Based on the data block storage method corresponding to the first target fingerprint, the first data block corresponding to the first target fingerprint in the target file data is stored, and based on the data block storage method corresponding to the second target fingerprint, the second data block corresponding to the second target fingerprint in the target file data is stored. This realizes data deduplication processing only for target file data with specific tags, reduces fingerprint calculation and comparison in the data deduplication process, effectively reduces the computing resources and computing time required for data storage, reduces the number of indexes built in the data deduplication process, reduces the memory space required, improves the data processing performance of the computer system, and improves the data storage efficiency.

[0191] Figure 5 This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing functions and processing capabilities. Figure 5 As shown, the data storage method provided in this embodiment includes the following steps:

[0192] Step 510: In response to receiving the target file data, determine the tag information corresponding to the target file data.

[0193] Step 520: Identify whether the label information is the target label information.

[0194] The content of steps 510-520 in this embodiment can be referred to the above. Figure 1 The corresponding content in [the document] will not be repeated here.

[0195] Step 530: If the tag information is not the target tag information, divide the target file data into multiple data blocks.

[0196] In this embodiment of the disclosure, when the tag information corresponding to the target file data is not the target tag information, the computer device can divide the target file data into multiple data blocks and then write each data block in the target file data into the storage space.

[0197] Step 540: Write each data block of the target file data into the storage space.

[0198] In this embodiment of the disclosure, after dividing the target file data into multiple data blocks, the computer device can write each data block in the target file data into the storage space.

[0199] Step 550: Create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0200] In this embodiment of the disclosure, after dividing the target file data into multiple data blocks, the computer device can calculate the target fingerprint corresponding to each data block in the target file data, and create a target fingerprint table corresponding to the target file data based on the target fingerprint corresponding to each data block in the target file data. The target fingerprint table includes the target fingerprint corresponding to each data block in the target file data.

[0201] Step 560: Add the target fingerprint table to the first fingerprint table corresponding to the file data already stored in the storage space to obtain the updated first fingerprint table.

[0202] In this embodiment of the disclosure, after creating a target fingerprint table corresponding to the target file data, the computer device can add the target fingerprint table to a first fingerprint table corresponding to the file data already stored in the storage space, thereby obtaining an updated first fingerprint table. The first fingerprint table can be understood as a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space.

[0203] Therefore, the storage method of the target file data can be determined based on the tag information corresponding to the target file data. When the tag information corresponding to the target file data is not the target tag information, other methods besides data deduplication can be used to store the target file data. That is, for target file data without specific tags, other methods besides data deduplication can be used to store the target file data. There is no need to perform fingerprint comparison of data blocks, which reduces the amount of data processing for data deduplication, effectively reduces the computing resources and computing time required for data storage, reduces the number of indexes built in the data deduplication process, reduces the memory space required, improves the data processing performance of the computer system, and improves the data storage efficiency.

[0204] In some embodiments of this disclosure, the above-described division of the target file data into multiple data blocks may include S21-S22:

[0205] S21. Determine the target data block partitioning granularity based on the size of the target file data. The target data block partitioning granularity is greater than or equal to the preset minimum partitioning granularity.

[0206] The size of file data can be understood as the amount of storage space that the file data needs to occupy, usually expressed in bytes (B) and its derived units.

[0207] The granularity of data block partitioning can be understood as the number of characters in each data block. The larger the granularity of data block partitioning, the more characters are in each data block.

[0208] In one example, the granularity of the target data block partition can be positively correlated with the size of the target file data. The larger the target file data, the larger the granularity of the target data block partition; the smaller the target file data, the smaller the granularity of the target data block partition.

[0209] The preset minimum partition granularity can be understood as the minimum partition granularity of the data block set in advance. The preset minimum partition granularity can be set as needed, and there is no limitation here.

[0210] S22. Divide the target file data into multiple data blocks according to the target data block granularity. The target data block granularity is greater than or equal to the preset minimum granularity.

[0211] After determining the granularity of the target data block partitioning, the computer device can divide the target file data into multiple data blocks according to the granularity of the target data block partitioning.

[0212] Therefore, the target file data can be divided into multiple data blocks based on the target data block granularity, which is greater than or equal to the preset minimum granularity. This achieves larger granularity data block partitioning, which can avoid fingerprint calculation and comparison for data deduplication between two files with only a small part in common, thus improving data processing and storage efficiency.

[0213] In some embodiments of this disclosure, the creation of the target fingerprint table corresponding to the target file data described above may include steps S31-S32:

[0214] S31. Calculate the target fingerprint corresponding to each data block in the target file data.

[0215] In this embodiment of the disclosure, after dividing the target file data into multiple data blocks, the computer device can calculate the target fingerprint corresponding to each data block in the target file data. For example, it can perform hash calculation on each data block in the target file data based on a preset hash algorithm to obtain the target fingerprint corresponding to each data block in the target file data. The preset hash algorithm can be set as needed and is not limited here. For example, it can be the SHA-1 hash algorithm or the SHA-256 hash algorithm.

[0216] S32. Based on the target fingerprints corresponding to each data block, create a target fingerprint table corresponding to the target file data.

[0217] After obtaining the target fingerprints corresponding to each data block in the target file data, the computer device can create a target fingerprint table corresponding to the target file data based on the target fingerprints corresponding to each data block in the target file data.

[0218] Therefore, by calculating the target fingerprints corresponding to each data block of the target file data, a target fingerprint table corresponding to the target file data can be accurately constructed.

[0219] Figure 6 This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing functions and processing capabilities. Figure 6 As shown, the data storage method provided in this embodiment includes the following steps:

[0220] Step 610: In response to receiving the target file data, determine the tag information corresponding to the target file data.

[0221] The content of step 610 in this embodiment can be referred to the above. Figure 1 The corresponding content is not repeated here.

[0222] Step 620: If the tag information corresponding to the target file data does not exist, create the tag information corresponding to the target file data.

[0223] In this embodiment of the disclosure, if the tag information corresponding to the target file data does not exist, the computer device can create the tag information corresponding to the target file data.

[0224] In some embodiments, if the tag information corresponding to the target file data does not exist, the computer device can create the tag information corresponding to the target file data based on the content of the target file data using a preset tag creation tool.

[0225] Tag creation tools can create tag information corresponding to target file data based on the content of the target file data.

[0226] For example, a label creation tool could be LabelTool, an open-source tool primarily used for image and video annotation.

[0227] Therefore, tag creation tools can be used to create tag information corresponding to target file data, improving the accuracy and efficiency of tag creation.

[0228] In other embodiments, if the tag information corresponding to the target file data does not exist, the computer device may, in response to the tag creation operation on the target file data, obtain the tag information created by the tag creation operation; and determine the tag information created by the tag creation operation as the tag information corresponding to the target file data.

[0229] In cases where the tag information corresponding to the target file data does not exist, the user can determine the tag information corresponding to the target file data based on the content type of the target file data, and then perform a tag creation operation on the computer device. For example, the tag creation operation can be a selection operation or a click operation of the tag option corresponding to the target file data, so that the computer device can determine the tag information corresponding to the target file data. Then, the computer device can respond to the tag creation operation of the target file data, obtain the tag information created by the tag creation operation, and determine the tag information created by the tag creation operation as the tag information corresponding to the target file data.

[0230] Therefore, tag information corresponding to target file data can be created through tag creation operations, improving the convenience of tag creation and user experience.

[0231] Step 630: Write the tag information into the inode of the target file data.

[0232] In this embodiment of the disclosure, after creating the tag information corresponding to the target file data, the computer device writes the tag information into the index node of the target file data.

[0233] Step 640: Identify whether the label information is the target label information.

[0234] Step 650: If the tag information is the target tag information, store the target file data in the storage space based on the data deduplication method.

[0235] Step 660: If the tag information is not the target tag information, store the target file data in the storage space based on a storage method different from the data deduplication method.

[0236] The content of steps 640-660 in this embodiment can be referred to the above. Figure 1 The corresponding content in [the document] will not be repeated here.

[0237] In this embodiment, even if the tag information corresponding to the target file data does not exist, tag information corresponding to the target file data can be created. The storage method of the target file data is determined based on the tag information corresponding to the target file data. Data deduplication is only used to store the target file data when the tag information is the target tag information. Otherwise, other methods besides data deduplication are used to store the target file data. In other words, this disclosure only performs data deduplication processing on target file data with specific tags, reducing the amount of data processing for data deduplication, effectively reducing the computing resources and computing time required for data storage, reducing the number of indexes built in the data deduplication process, reducing the memory space required, improving the data processing performance of the computer system, and improving the data storage efficiency.

[0238] Figure 7 This is a flowchart illustrating a data storage method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing functions and processing capabilities. Figure 7 As shown, the data storage method provided in this embodiment includes the following steps:

[0239] Step 710: In response to receiving the target file data, determine the tag information corresponding to the target file data.

[0240] The content of step 710 in this embodiment can be referred to the above. Figure 1 The corresponding content in [the document] will not be repeated here.

[0241] Step 720: If the tag information corresponding to the target file data does not exist, store the target file data in the storage space based on the data deduplication method.

[0242] In some embodiments, when the tag information corresponding to the target file data does not exist, storing the target file data in the storage space based on data deduplication may include:

[0243] The computer device can divide the target file data into multiple data blocks and create a target fingerprint table corresponding to the target file data, even when the tag information corresponding to the target file data does not exist. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data. The fingerprints in the target fingerprint table are compared with the fingerprints in a first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data already stored in the storage space. For each first data block corresponding to a first target fingerprint in the target file data, index information is created for the first data block. The index information points to the target data block corresponding to the first target fingerprint already stored in the storage space. The index information of each first data block is written into the inode of the target file data. Each first data block in the target file data is then deleted. For each second data block corresponding to a second target fingerprint in the target file data, the second data block is written into the storage space. Each second target fingerprint is added to the first fingerprint table, resulting in an updated first fingerprint table.

[0244] Therefore, even when the tag information corresponding to the target file data does not exist, the target file data can be stored in the storage space based on the data deduplication method. Data deduplication is only performed on target file data that has tag information and the tag information is a specific tag. This reduces the amount of data processing for data deduplication, effectively reduces the computing resources and computing time required for data storage, reduces the number of indexes built in the data deduplication process, reduces the memory space required, improves the data processing performance of the computer system, and improves the data storage efficiency.

[0245] Figure 8 This is a schematic diagram of a data storage device provided in an embodiment of the present disclosure. This device can be understood as the aforementioned computer equipment or a functional module within the aforementioned computer equipment. As shown in the figure, the data storage device 800 may include:

[0246] The determination module 810 is used to determine the tag information corresponding to the target file data in response to receiving the target file data;

[0247] The identification module 820 is used to identify whether the label information is the target label information;

[0248] The first storage module 830 is used to store the target file data into the storage space based on the data deduplication method when the tag information is the target tag information;

[0249] The second storage module 840 is used to store the target file data in the storage space based on a storage method different from the data deduplication method when the tag information is not the target tag information.

[0250] Optionally, the above-mentioned determining module includes:

[0251] The write submodule is used to write the metadata of the target file data into the inode of the target file data;

[0252] The search submodule is used to search for tag information of target file data in the inode of the target file data.

[0253] Optionally, the first storage module mentioned above includes:

[0254] The partitioning and creation submodule is used to divide the target file data into multiple data blocks when the tag information is the target tag information, and to create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0255] The comparison submodule is used to compare the fingerprints in the target fingerprint table with the fingerprints in the first fingerprint table to determine the first target fingerprint and the second target fingerprint in the target fingerprint table. The first target fingerprint is the same as the fingerprint in the first fingerprint table, and the second target fingerprint is the fingerprint that is different from the fingerprint in the first fingerprint table. The first fingerprint table is a fingerprint table constructed from the fingerprints corresponding to each data block of the file data stored in the storage space.

[0256] The first storage submodule is used to store the first data block corresponding to the first target fingerprint in the target file data based on the data block storage method corresponding to the first target fingerprint;

[0257] The second storage submodule is used to store the second data block corresponding to the second target fingerprint in the target file data based on the data block storage method corresponding to the second target fingerprint.

[0258] Optionally, the first storage submodule mentioned above includes:

[0259] The index creation unit is used to create index information for the first data block corresponding to each first target fingerprint in the target file data. The index information points to the target data block corresponding to the first target fingerprint that has been stored in the storage space.

[0260] The first writing unit is used to write the index information of each first data block into the index node of the target file data;

[0261] The deletion unit is used to delete each first data block in the target file data.

[0262] Optionally, the second storage submodule mentioned above includes:

[0263] The second writing unit is used to write the second data block corresponding to each second target fingerprint in the target file data into the storage space.

[0264] The update unit is used to add each second target fingerprint to the first fingerprint table to obtain the updated first fingerprint table.

[0265] Optionally, the second storage module mentioned above includes:

[0266] The data block partitioning submodule is used to divide the target file data into multiple data blocks when the tag information is not the target tag information;

[0267] The data block writing submodule is used to write each data block from the target file into the storage space;

[0268] The fingerprint table creation submodule is used to create a target fingerprint table corresponding to the target file data. The target fingerprint table includes the target fingerprints corresponding to each data block in the target file data.

[0269] The update submodule is used to add the target fingerprint table to the first fingerprint table corresponding to the file data already stored in the storage space, so as to obtain the updated first fingerprint table.

[0270] Optionally, the above-mentioned sub-modules for creating data blocks or creating sub-modules include:

[0271] The granularity determination unit is used to determine the granularity of the target data block corresponding to the target file data based on the size of the target file data. The granularity of the target data block is greater than or equal to the preset minimum granularity.

[0272] A partitioning unit is used to divide the target file data into multiple data blocks according to the granularity of the target data block.

[0273] Optionally, the above data storage device includes:

[0274] The tag creation module is used to create tag information for target file data when the tag information for the target file data does not exist.

[0275] The tag writing module is used to write tag information into the inode of the target file data.

[0276] Optionally, the above-mentioned tag creation module includes:

[0277] The tag creation submodule is used to create tag information corresponding to the target file data based on the content of the target file data using a preset tag creation tool.

[0278] Optionally, the above-mentioned tag creation module includes:

[0279] The `get` submodule is used to respond to tag creation operations on target file data and retrieve tag information created by the tag creation operation.

[0280] The tag determination submodule is used to determine the tag information created by the tag creation operation as the tag information corresponding to the target file data.

[0281] Optionally, the above data storage device includes:

[0282] The third storage module is used to store the target file data in the storage space based on the data deduplication method when the tag information corresponding to the target file data does not exist.

[0283] Optionally, the target label information mentioned above includes at least one of the model training code label and the model training dataset label.

[0284] The data storage device provided in this disclosure can implement the methods of any of the above embodiments, and its execution method and beneficial effects are similar, so they will not be described again here.

[0285] This disclosure also provides a computer device, which includes a processor and a memory, wherein the memory stores a computer program. When the computer program is executed by the processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0286] The computer device in this disclosure can be understood as any device with processing and computing capabilities, including but not limited to electronic devices such as smartphones, laptops, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals, wearable devices, digital TVs, desktop computers, smart home devices, etc.

[0287] Figure 9 This is a schematic diagram of the structure of a computer device provided in an embodiment of this disclosure, such as... Figure 9 As shown, the computer device 900 may include a processor 910 and a memory 920. The memory 920 stores a computer program 921. When the computer program 921 is executed by the processor 910, it can implement the method provided in any of the above embodiments. The execution method and beneficial effects are similar and will not be described again here.

[0288] Of course, for the sake of simplicity, Figure 9Only some of the components of the computer device 900 relevant to the present invention are shown in this illustration; components such as buses, input / output interfaces, input devices, and output devices are omitted. In addition, the computer device 900 may include any other suitable components depending on the specific application.

[0289] This disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it can implement the methods of any of the above embodiments. The execution method and beneficial effects are similar, and will not be described again here.

[0290] The aforementioned computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may, for example, include, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatuses, or devices, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: electrical connections having one or more wires, portable disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0291] The computer program described above can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this disclosure. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer device, partially on the user's device, as a standalone software package, partially on the user's computer device and partially on a remote computer device, or entirely on a remote computer device or server.

[0292] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0293] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0294] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data storage method, characterized by, The method comprises: in response to receiving target file data, determining label information corresponding to the target file data; identifying whether the label information is target label information; in the case where the label information is target label information, storing the target file data in the storage space based on a data deduplication manner; in the case where the label information is not target label information, storing the target file data in the storage space based on a storage manner different from the data deduplication manner.

2. The data storage method of claim 1, wherein, The determination of the label information of the target file data comprises: writing metadata of the target file data into an inode of the target file data; finding the label information of the target file data in the inode of the target file data.

3. The data storage method of claim 1, wherein, In the case where the label information is target label information, the storing of the target file data in the storage space based on the data deduplication manner comprises: in the case where the label information is target label information, dividing the target file data into a plurality of data blocks and creating a target fingerprint table corresponding to the target file data, the target fingerprint table comprising target fingerprints corresponding to the data blocks in the target file data; comparing the fingerprints in the target fingerprint table with the fingerprints in a first fingerprint table to determine first target fingerprints and second target fingerprints in the target fingerprint table, the first target fingerprints being the same as the fingerprints in the first fingerprint table, and the second target fingerprints being different from the fingerprints in the first fingerprint table, the first fingerprint table being a fingerprint table constructed from the fingerprints corresponding to the data blocks of the file data already stored in the storage space; storing first data blocks corresponding to the first target fingerprints in the target file data based on a data block storage manner corresponding to the first target fingerprints; storing second data blocks corresponding to the second target fingerprints in the target file data based on a data block storage manner corresponding to the second target fingerprints.

4. The data storage method of claim 3, wherein, The storing of the first data blocks corresponding to the first target fingerprints in the target file data based on the data block storage manner corresponding to the first target fingerprints comprises: for each first data block corresponding to the first target fingerprints in the target file data, creating index information of the first data block, the index information pointing to target data blocks corresponding to the first target fingerprints that have been stored in the storage space; writing the index information of each first data block into an inode of the target file data; deleting each first data block in the target file data.

5. The data storage method of claim 3, wherein, The storing of the second data blocks corresponding to the second target fingerprints in the target file data based on the data block storage manner corresponding to the second target fingerprints comprises: for each second data block corresponding to the second target fingerprints in the target file data, writing the second data block into the storage space; adding each second target fingerprint to the first fingerprint table to obtain an updated first fingerprint table.

6. The data storage method of claim 1, wherein, The storing the target file data into the storage space based on the storage mode different from the data deduplication mode in the case that the tag information is not the target tag information comprises: In the case that the tag information is not the target tag information, the target file data is divided into a plurality of data blocks; Each data block in the target file data is written into the storage space; A target fingerprint table corresponding to the target file data is created, and the target fingerprint table comprises target fingerprints corresponding to each data block in the target file data; The target fingerprint table is added to a first fingerprint table corresponding to the file data stored in the storage space, to obtain an updated first fingerprint table.

7. The data storage method according to claim 3 or 6, characterized by, The dividing the target file data into a plurality of data blocks comprises: A target data block division granularity corresponding to the target file data is determined based on the size of the target file data, and the target data block division granularity is greater than or equal to a preset minimum division granularity; The target file data is divided into a plurality of data blocks according to the target data block division granularity.

8. The data storage method of claim 1, wherein, After the determining the tag information corresponding to the target file data, the method further comprises: In the case that the tag information corresponding to the target file data does not exist, the tag information corresponding to the target file data is created; The tag information is written into an index node of the target file data.

9. The data storage method of claim 8, wherein, The creating the tag information corresponding to the target file data comprises: The tag information corresponding to the target file data is created based on the content of the target file data by using a preset tag creation tool.

10. The data storage method of claim 8, wherein, The creating the tag information corresponding to the target file data comprises: In response to a tag creation operation on the target file data, tag information created by the tag creation operation is obtained; The tag information created by the tag creation operation is determined as the tag information corresponding to the target file data.

11. The data storage method of claim 1, wherein, After the determining the tag information corresponding to the target file data, the method further comprises: In the case that the tag information corresponding to the target file data does not exist, the target file data is stored into the storage space based on the data deduplication mode.

12. The data storage method of claim 1, wherein, The target tag information comprises at least one of a model training code tag and a model training data set tag.

13. A data storage device, characterized by The apparatus comprises: A determining module configured to determine tag information corresponding to target file data in response to receiving the target file data; An identifying module configured to identify whether the tag information is target tag information; A first storing module configured to store the target file data into a storage space based on a data deduplication mode in the case that the tag information is target tag information; A second storing module configured to store the target file data into the storage space based on a storage mode different from the data deduplication mode in the case that the tag information is not target tag information.

14. A computer device, comprising: Comprise: A memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the data storage method of any one of claims 1-12 is implemented.

15. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, which, when executed by a processor, implements the data storage method as described in any one of claims 1-12.

Images