Data processing method and device, equipment and storage medium
By calculating the index information of the data to be written in the storage system and matching it with a pre-built index table, a virtual copy operation is performed, which solves the problems of low efficiency and wasted storage space in traditional file system copying methods, and achieves fast data copying and improved system performance.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- JINAN INSPUR DATA TECH CO LTD
- Filing Date
- 2025-07-30
- Publication Date
- 2026-08-04
AI Technical Summary
Traditional file system copying methods are inefficient, waste storage space, and lead to a decline in system performance.
By calculating the index information of the data to be written in the storage system and matching it with a pre-built index table, a virtual copy operation is performed if a match is found, thus avoiding the actual copying of the same data. Physical copying is only performed when the data is modified.
Reduce unnecessary read and write operations, improve data copying efficiency, save storage space, and enhance system performance.
Smart Images
Figure CN120994122B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of data processing technology, and in particular to data processing methods, apparatus, devices and storage media. Background Technology
[0002] With the rapid development of information technology, data storage and management have become increasingly important. Traditional file system copying methods suffer from inefficiency, wasted storage space, and performance degradation. While some methods can reduce storage space waste, they require maintaining an index table of differencing data, increasing system complexity. Summary of the Invention
[0003] This application provides a data processing method, apparatus, device, and storage medium to at least solve the problems of low efficiency, wasted storage space, and system performance degradation caused by data copying methods in related technologies.
[0004] This application provides a data processing method, which is executed by a storage system, and the method includes:
[0005] When a data write request is received from a client, the data to be written is extracted from the data write request.
[0006] The data to be written is indexed according to preset index rules to obtain the first index information;
[0007] The first index information is matched with the index information in the pre-built index table, and the index information in the index table is generated based on the previously read data;
[0008] When the first index information matches the second index information in the index table, a virtual copy operation is performed on the data to be written in the data to be written information.
[0009] This application also provides a data processing apparatus, corresponding to a storage system, comprising:
[0010] The receiving module is used to receive data write requests sent by the client;
[0011] The extraction module is used to extract the data to be written information from the data write request when a data write request is received from the client.
[0012] The processing module is used to perform index calculations on the data to be written according to preset index rules and obtain the first index information;
[0013] The matching module is used to match the first index information with the index information in the pre-built index table. The index information in the index table is generated based on the previously read data.
[0014] The copy module is used to perform a virtual copy operation on the data to be written in the data information to be written when the first index information matches the second index information in the index table.
[0015] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data processing methods.
[0016] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data processing methods.
[0017] This application utilizes a method to calculate a first index of the data to be written and then compare it with index information in a pre-built index table. This allows determination of whether the same data exists in the backend storage device. If the first index matches a second index in the index table, it indicates that the same data exists in the backend storage device. In this case, a virtual copy operation can be performed directly on the data to be written, reducing unnecessary read and write operations, achieving fast data copying, and effectively improving system performance. Furthermore, the use of virtual copy technology reduces the actual storage space required, avoiding the waste of storage resources caused by repeatedly storing the same data. Attached Figure Description
[0018] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0019] Figure 1 This is a schematic flowchart of a data processing method provided in an embodiment of this application;
[0020] Figure 2 This is a schematic diagram of another data processing method provided in an embodiment of this application;
[0021] Figure 3 This is a schematic diagram illustrating the generation of virtual data after a virtual copy operation is performed on the data to be written, as provided in an embodiment of this application.
[0022] Figure 4 This is a schematic diagram of another data processing method provided in an embodiment of this application;
[0023] Figure 5 This is a schematic diagram of the index information structure provided in the embodiments of this application;
[0024] Figure 6 This is a schematic diagram of a data processing device structure provided in an embodiment of this application;
[0025] Figure 7 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation
[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.
[0027] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.
[0028] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0029] With the rapid development of information technology, data storage and management have become increasingly important. Traditional file system copying methods, such as full copying, suffer from storage redundancy and performance bottlenecks, especially when frequent copy creation is required (e.g., backups, testing), leading to inefficiency. This method not only wastes storage space but also degrades system performance.
[0030] Traditional file system copying methods mainly include full copy and incremental copy. A full copy copies all data from the source file to the target file. This method is simple to implement, but it wastes a lot of storage space, especially when the source file is large. An incremental copy only copies the differences between the source and target files. This method reduces storage space waste, but it requires maintaining an index table of these differences, increasing system complexity.
[0031] In certain application scenarios, such as copy-on-write (CoW) maintenance operations, when CoW file systems (such as ZFS and Btrfs) or snapshot technologies modify data, the original data must first be copied to a new location before updating the metadata. Even if the data has not been modified by the user, background tasks (such as deduplication and compression) may need to read data block information and rewrite it. In this case, even if the read data has not changed, it still needs to be rewritten to the background, which is equivalent to performing a copy operation in the background storage device. Another example is storage layer cache consistency maintenance: when the storage system has multi-level caching (such as SSD cache + HDD backend), the OS may need to ensure consistency between the cache and persistent storage through writeback. The specific process is: after the OS reads data, if the caching policy requires flushing dirty data (such as due to timeout or LRU policy), it may rewrite the unmodified data to synchronize the state. The aforementioned documents merely illustrate a scenario where data needs to be read from a background storage device and then rewritten to it. This scenario is not applicable to the solution of this application; it is only intended to illustrate that the solution of this application's embodiments involves reading data from a background storage device and then writing that data back to the background storage device at some point. However, rewriting unmodified data to the background storage device inevitably occupies storage space, especially with numerous backups. This not only wastes a significant amount of storage space and resources but also leads to a decline in system performance.
[0032] To address the aforementioned problems, embodiments of this application provide a data processing method, as detailed below. Figure 1 As shown, this method is executed by the storage system. In an optional example, the storage system establishes communication connections with the client and the backend storage device, respectively. The method includes the following steps:
[0033] Step S101: When a data write request is received from the client, extract the data information to be written from the data write request.
[0034] Specifically, when a data write request is received from the client, the data to be written is extracted from the data write request. To avoid the aforementioned situations such as wasting storage space resources, in this embodiment of the application, the data to be written is not directly written to the backend storage device, but the following operation steps are performed.
[0035] Step S102: Perform index calculation on the data to be written according to the preset index rules to obtain the first index information.
[0036] Specifically, the first index information is unique. The data to be written may include the data to be written. When calculating the index of the data to be written according to the preset index rules to obtain the first index information, in an optional example, the fingerprint value can be directly calculated based on the data to be written as the index information.
[0037] Step S103: Match the first index information with the index information in the pre-built index table.
[0038] The index information in the index table is generated based on previously read data.
[0039] Specifically, as described above, the index information is generated based on the data that has already been read. The reason for generating index information only for the read data is that the read data is usually small in quantity, and often consists of frequently accessed data. Calculating index information based on read data consumes relatively few computational resources. Furthermore, the calculated index information occupies minimal storage space, for example, in bytes, making subsequent comparisons easier. Even if the same data is subsequently written, based on the application scenario of this application, it involves copying the already read data—that is, reading the data and then writing it back at a later time. Therefore, it is possible to calculate the index information for the already read data first. Since the index information is unique, if it is necessary to repeatedly write the already read data to the backend storage device later, the index information can be used to determine whether the data is already stored in the backend storage device.
[0040] Step S104: When the first index information matches the second index information in the index table, a virtual copy operation is performed on the data to be written in the data to be written information.
[0041] Specifically, as described above, if the first index information corresponding to the data to be written matches the second index information in the index table, it means that the data to be written has already been stored in the backend storage device, and this write operation is considered a duplicate write. Directly performing a write operation would occupy redundant storage space in the backend storage device. Therefore, to save storage space in the backend storage device, a virtual copy operation can be performed on the data to be written. Virtual copy is a logical data replication; its core idea is not to actually copy the data content, but to use metadata operations (such as reference counting and pointer mapping) to make multiple logical copies point to the same physical data. A physical copy (i.e., copy-on-write, CoW) is only triggered when the data is modified.
[0042] This application provides a data processing method that calculates a first index of the data to be written and compares it with index information in a pre-built index table. This determines whether the same data is stored in the backend storage device. If the first index matches a second index in the index table, it indicates that the same data is stored in the backend storage device. In this case, a virtual copy operation can be performed directly on the data to be written, reducing unnecessary read / write operations, achieving fast data copying, and effectively improving system performance. Furthermore, the use of virtual copy technology reduces the actual storage space occupied, avoiding the waste of storage resources caused by repeatedly storing the same data.
[0043] In an optional example, based on the foregoing embodiments, when the data information to be written includes data block information corresponding to multiple data blocks, the method may further include the following method steps, see details below. Figure 2 As shown, it includes:
[0044] Step S201: For each data block, perform index calculation according to the preset index rules to obtain the index information corresponding to each data block.
[0045] Specifically, the process of calculating index information includes, for example, directly calculating fingerprint values from the data in the data block information to obtain the fingerprint value corresponding to each data block information.
[0046] Step S202: Match the index information corresponding to each data block with the index information in the index table.
[0047] Specifically, the index information corresponding to each data block is matched with the index information in the index table to determine whether each data block has been stored in the background storage device.
[0048] Step S203: Perform a virtual copy operation on the data block that matches the index information in the index table.
[0049] Step S204: When the index information of at least one data block does not match any of the index information in the index table, then at least one data block is written to the back-end storage device corresponding to the storage system.
[0050] Specifically, when the data to be written includes multiple data blocks, one or more of these blocks may already be stored in the background storage device. Some of these blocks may have been modified. In this case, data blocks whose index information matches the index information in the index table can be directly copied using a virtual copy operation. However, data blocks whose index information does not match the index information in the index table will not be copied using a virtual copy operation; instead, these data blocks will be directly written to the background storage device, and corresponding metadata information will be generated.
[0051] In practical applications, each piece of data to be written will generate corresponding new metadata information. Even data to be written that undergoes virtual copying will generate corresponding metadata information to distinguish it from the original metadata information.
[0052] In an optional embodiment, in addition to the foregoing description, when the first index information matches the second index information in the index table, after performing a virtual copy operation on the data to be written in the data to be written information, the method further includes: establishing an association between the new metadata information and the first index information.
[0053] See details Figure 3 As shown, Figure 3 This illustrates that when multiple data stripes exist, and each stripe contains data to be written, and each has a corresponding index information matched in the index table, then virtual data is generated for each data stripe separately, which is equivalent to performing a virtual copy operation. Therefore, in Figure 3 This includes file a and file b. File a contains multiple data strips to be written. Figure 3 The diagram shows three data stripes. The b file is a file composed of multiple dummy data.
[0054] In another optional example, the data to be written may also include the file name to which the data belongs, the corresponding position offset of the data to be written, and the data length. Then, index information is calculated based on the data to be written and this information. The specific calculation process is the same as the process of calculating index information for the data to be read described below, so it will not be repeated here.
[0055] Based on any of the foregoing embodiments, the method may further include the following method steps, as detailed in the following examples. Figure 4 As shown, the method also includes the following steps:
[0056] Step S401: When a data read request is received from the client, the data read request is parsed to obtain the data information to be read.
[0057] Specifically, the information to be read includes the file name to which the data belongs, the position offset of the data, and the data length.
[0058] Step S402: Determine the data reading location information based on the file name, location offset, and data length.
[0059] Step S403: Based on the data read location information, send the data read request to the backend storage device corresponding to the storage system.
[0060] Step S404: After receiving the data to be read from the backend storage device, the data to be read is fed back to the client.
[0061] Specifically, the file name can be used to locate the file containing the data to be read in the background storage device. Then, based on the position offset and data length, the final read position information is determined. Finally, the data to be read is retrieved based on the read position information.
[0062] In an optional example, the read location information may include the cached memory address and physical address information corresponding to the data to be read. The actual physical address information indicates the actual location where the data to be read is stored. The cached memory address indicates that after data is read from the actual physical address of the backend storage device, it needs to be cached in the cached memory address before being sent to the client via the storage system.
[0063] Step S405: Perform index calculation on the data information to be read according to the preset index rules to obtain the third index information corresponding to the data to be read.
[0064] By parsing client requests, the system can quickly locate specific data positions, enabling efficient data reading. By recording file names, location offsets, and data lengths, the system can precisely manage data, avoiding erroneous or incomplete data readings. Indexing data using indexing rules optimizes storage resource usage and improves the speed and efficiency of data retrieval. Client-requested data can be located and read in a short time, providing a rapid service response. Indexing rules ensure data consistency; even if data changes within the storage system, the index accurately reflects the data's new location and status. Fast data access and stable service quality enhance the user experience.
[0065] In one optional example, index calculation for the data to be read can include directly calculating the fingerprint of the data to be read to obtain index information. Alternatively, index information can be generated based on the position offset, the data to be read, and the data read position information. Specifically, it is sufficient to ensure that the method of calculating the index information for the data to be read is consistent with the method of calculating the index information for the data to be written.
[0066] The method for calculating fingerprint values based on the data to be read can be found in existing fingerprint value calculation methods, such as using hash algorithms, which will not be elaborated here.
[0067] The following will illustrate the process of obtaining the third index information corresponding to the data to be read by calculating the index according to the preset index rules (based on the position offset, the data to be read, and the data reading position information, etc.). The specific steps include the following:
[0068] Step a1: Build a first-level index based on the file name and location offset according to the preset indexing rules.
[0069] Specifically, the file name and position offset are used as primary index information. This information is then used to group the index information corresponding to the data to be read, thus allowing multiple indexes belonging to the same primary index to be grouped together.
[0070] Step a2: Perform fingerprint calculation based on the data to be read to obtain the fingerprint value corresponding to the data to be read.
[0071] Step a3: Generate fingerprint key-value pairs as secondary indexes based on fingerprint values and data read location information.
[0072] Specifically, the secondary index can include a key that is a fingerprint value and a value that is a cached memory address and a physical address (i.e., a stored physical address).
[0073] The primary and secondary indexes together constitute the third index information.
[0074] Step S406: Store the third index information in the index table.
[0075] By constructing first-level and second-level indexes to form third-level index information, it is easier to quickly determine whether the data to be written has been stored in the backend storage device. The number of indexes traversed is less, resulting in higher retrieval efficiency and faster speed.
[0076] More specifically, by establishing a primary index (based on filename and location offset), the system can quickly locate data within a specific file, thereby improving the accuracy of data retrieval. A secondary index (based on fingerprint values and data read location information) further optimizes the data access process because it allows the system to quickly find data using a unique identifier (fingerprint) of the data content without needing to read the entire file. Fingerprint calculation ensures that each data block has a unique fingerprint value, which helps prevent duplicate data storage, thus saving storage space. Through fingerprint key-value pairs, the system can quickly compare data blocks, facilitating subsequent determination of whether this data has been written repeatedly. This index structure can significantly reduce data access time, improve the system's ability to process large amounts of data, and thus enhance overall performance. Third-level index information (a combination of primary and secondary indexes) simplifies the data management process by providing a multi-level data access and location mechanism. It supports complex query operations, such as fuzzy matching and range queries, and helps reduce storage costs by minimizing unnecessary data copying and storage. Fast data location and access improve data availability, especially in data-intensive applications requiring rapid response. It provides an efficient, secure, and flexible solution for data storage and management, helping to optimize system performance and resource utilization.
[0077] In an optional example, after generating fingerprint key-value pairs as secondary indexes based on fingerprint values and data read location information, the method further includes the following steps:
[0078] A fingerprint index tree is generated based on the secondary indexes corresponding to all the data to be read that have the same primary index.
[0079] For details, see Figure 5 As shown, in Figure 5 The image displays multiple first-level indexes, such as 1, 2, 3 to N in the diagram. Each first-level index contains multiple second-level indexes, forming a fingerprint index tree. Each node in the fingerprint index tree includes a fingerprint key-value pair, where the key is the fingerprint value, and the value is the cache memory address and the physical storage address.
[0080] The fingerprint index tree data structure can be illustrated as follows:
[0081] Root node [ / data / app.log, offset 0]
[0082] ├─Fingerprint X
[0083] │├─Memory address 0x1000→Physical address 0x5000
[0084] └─Fingerprint Y
[0085] └─Memory address 0x2000 → Physical address 0x6000.
[0086] Specifically, a fingerprint index tree is a highly efficient data structure that can quickly sort and search fingerprint key-value pairs, thereby accelerating the data retrieval process. By organizing secondary indexes of all data with the same primary index into an index tree, storage space can be utilized more effectively, reducing redundancy. Index trees allow for fast data traversal and searching, significantly reducing access time, especially when processing large amounts of data. Fingerprint index trees support more complex query operations, such as range queries and prefix queries, increasing the system's query flexibility. Although building an index tree may take some time, once established, it reduces the computational burden in subsequent queries, thus lowering long-term maintenance costs. Through index trees, the system can more effectively manage large amounts of indexed data, improving data management capabilities. As the data volume grows, the fingerprint index tree can effectively expand to accommodate more data without significantly reducing query efficiency. Fingerprint index trees can be used as part of a multi-level index structure, combined with other types of indexes (such as primary indexes), providing more comprehensive data location and access capabilities. By organizing data with the same primary index into an index tree, the uniformity of data distribution can be improved, reducing data skew problems and thus enhancing system performance. The ordered structure of an index tree helps ensure the orderliness of data, which is especially important for applications that require maintaining data sequence.
[0087] In an optional example, the method can also include integrating multiple dimensions into the primary index, such as the file type of the data to be read, the read time, and the access frequency, to provide a more comprehensive index. In the secondary index, multiple fingerprint values (such as the hash value of the file content, text digest, etc.) are combined to form a multi-dimensional index to support more complex query requirements.
[0088] The specific implementation includes the following steps:
[0089] Step 1, for example, determine the primary index dimensions that need to be integrated;
[0090] Examples of factors include file type (text, image, video, etc.), creation time, access frequency, and file size.
[0091] Step 2, Data Preprocessing;
[0092] When reading data, the data to be read is preprocessed to extract the relevant dimensional information mentioned above.
[0093] For example, file types can be determined by file extensions or MIME types, and creation times can be determined by file metadata or system timestamps.
[0094] Step 3: Establish the index structure;
[0095] Design an index structure that can store multidimensional information, such as using a B-tree, hash table, or Trie tree. For each dimension, create an index field to store information about that dimension.
[0096] Step 4, index update;
[0097] When a file is read, the corresponding index fields are updated based on the information in the data to be read.
[0098] Use transactions or locking mechanisms to ensure the atomicity and consistency of index updates.
[0099] Step 5, query optimization;
[0100] When querying, data is retrieved from the index based on the combination of dimensions required for the query.
[0101] Use indexes to optimize query statements and reduce full table scans.
[0102] Multiple fingerprint values are combined to form a multi-dimensional index.
[0103] Step 1, fingerprint value calculation;
[0104] As described above, fingerprint calculations are performed on the file content, such as SHA-256 hash, MD5 hash, or more complex fingerprint algorithms (such as perceptual hashing for images).
[0105] When the data to be read is text content, a text digest algorithm can also be used to generate a digest as a fingerprint.
[0106] Step 2: Construct a multidimensional index;
[0107] Create an index entry containing multiple fingerprint values for each piece of data to be read.
[0108] Design a multidimensional index structure, such as a multidimensional hash table or a spatial database, that can store and query multiple fingerprint values.
[0109] Step 4, query execution;
[0110] During the query, multiple fingerprint values are combined based on the query conditions.
[0111] Use a multidimensional index structure to perform queries to find matching files.
[0112] Step 5, Index Maintenance;
[0113] Regularly check and maintain the indexes to ensure their accuracy and efficiency.
[0114] By integrating multiple dimensions, queries become more precise and flexible. Users can combine queries based on multiple conditions such as file type, creation time, and access frequency, improving search efficiency. Multidimensional index structures can quickly locate data, reducing query time, especially when processing large amounts of data. By reducing full table scans, index-optimized queries can reduce CPU and I / O resource consumption, improving overall system performance. Multidimensional indexes combining multiple fingerprint values support more complex query needs, such as fuzzy queries and range queries, enhancing the system's query capabilities. Using transactions or locking mechanisms ensures the atomicity and consistency of index updates, maintaining data consistency even under concurrent access and data modification. This indexing method can adapt to different types of data and query patterns, improving the system's versatility and adaptability. Multidimensional index structures are easily extensible, allowing for the easy addition of new index dimensions or fingerprint value types to accommodate data growth and new query requirements. Fast data retrieval and powerful query capabilities directly enhance the user experience. Regularly maintaining indexes ensures data accuracy and index efficiency, thereby improving the overall efficiency of data management.
[0115] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.
[0116] Embodiments of this application also provide a data processing apparatus, see details below. Figure 6 As shown, the device includes: a receiving module 601, an extraction module 602, a processing module 603, a matching module 604, and a copying module 605.
[0117] The receiving module 601 is used to receive data write requests sent by the client;
[0118] Extraction module 602 is used to extract the data to be written information from the data write request when a data write request is received from the client;
[0119] Processing module 603 is used to perform index calculation on the data to be written according to preset index rules and obtain the first index information;
[0120] The matching module 604 is used to match the first index information with the index information in the pre-built index table. The index information in the index table is generated based on the previously read data.
[0121] The copy module 605 is used to perform a virtual copy operation on the data to be written in the data information to be written when the first index information matches the second index information in the index table.
[0122] In an optional example, the processing module 603 is also used to perform index calculations for each data block information according to preset index rules, and obtain the index information corresponding to each data block.
[0123] The matching module 604 is also used to match the index information corresponding to each data block with the index information in the index table;
[0124] The copy module 605 is also used to perform a virtual copy operation on data blocks that match the index information in the index table;
[0125] The processing module 603 is also used to write at least one data block to the back-end storage device corresponding to the storage system when the index information of at least one data block does not match all the index information in the index table.
[0126] In an optional example, processing module 603 is also used to generate new metadata information corresponding to the data to be written;
[0127] Establish the association between the new metadata information and the first index information.
[0128] In an optional example, the device further includes: a transmitting module 606;
[0129] The receiving module 601 is also used to receive data read requests sent by the client;
[0130] The processing module 603 is also used to parse the data read request, obtain the data information to be read, including the file name to which the data to be read belongs, the position offset of the data to be read, and the data length; and determine the data read position information based on the file name, position offset, and data length.
[0131] The sending module 606 is used to send a data reading request to the back-end storage device corresponding to the storage system based on the data reading location information;
[0132] The receiving module 601 is also used to receive data to be read sent by the back-end storage device;
[0133] The sending module 606 is also used to send the data to be read back to the client;
[0134] The processing module 603 is also used to perform index calculation on the data information to be read according to the preset index rules, obtain the third index information corresponding to the data to be read, and store the third index information in the index table.
[0135] In an optional example, the processing rules are specifically used to build a first-level index based on the file name and position offset according to preset indexing rules;
[0136] Additionally, fingerprint calculation is performed based on the data to be read to obtain the fingerprint value corresponding to the data to be read;
[0137] Based on the fingerprint value and data read location information, a fingerprint key-value pair is generated as a secondary index;
[0138] The first-level index and the second-level index together constitute the third index information.
[0139] In an optional example, the data read location information includes the cache memory address and physical address information corresponding to the data to be read.
[0140] In an optional example, the processing module 603 is also configured to generate a fingerprint index tree based on the secondary indexes corresponding to all the data to be read that have the same primary index.
[0141] The description of the features of the data processing apparatus provided in this application can be found in the relevant description of the data processing method, which will not be repeated here.
[0142] This application provides a data processing apparatus that calculates first index information of the data to be written, and then compares the first index information with index information in a pre-built index table. This determines whether the same data is stored in the background storage device. If the first index information matches the second index information in the index table, it indicates that the same data is stored in the background storage device. In this case, a virtual copy operation can be directly performed on the data to be written, thereby reducing unnecessary read and write operations, achieving fast data copying, and effectively improving system performance. Simultaneously, the use of virtual copy technology reduces the actual storage space occupied, avoiding the waste of storage resources caused by repeatedly storing the same data.
[0143] Embodiments of this application also provide an electronic device, such as... Figure 7 As shown, it includes a memory 10 and a processor 20. The memory 10 stores a computer program, and the processor 20 is configured to run the computer program to perform the steps in any of the above-described data processing method embodiments.
[0144] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data processing method embodiments or the steps in any of the above-described data reading method embodiments when it is run.
[0145] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.
[0146] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above data processing method embodiments.
[0147] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data reading method embodiments.
[0148] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0149] The data processing method, apparatus, device, and storage medium provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and its core ideas. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.
Claims
1. A data processing method, characterized in that, The method is executed by a storage system, and the method includes: When a data read request is received from the client, the data read request is parsed to obtain the data information to be read. The data information to be read includes the file name to which the data to be read belongs, the position offset of the data to be read, and the data length. The data reading location information is determined based on the file name, the location offset, and the data length; Based on the data read location information, the data read request is sent to the backend storage device corresponding to the storage system; After receiving the data to be read from the backend storage device, the data to be read is fed back to the client; In addition, the data to be read is indexed according to a preset indexing rule to obtain the third index information corresponding to the data to be read; Store the third index information in the index table; When a data write request is received from a client, the data information to be written is extracted from the data write request; The data to be written is indexed according to a preset indexing rule to obtain the first index information; The first index information is matched with the index information in the pre-built index table, wherein the index information in the index table is generated based on the previously read data; When the first index information matches the second index information in the index table, a virtual copy operation is performed on the data to be written in the data to be written information.
2. The method according to claim 1, characterized in that, When the data information to be written includes data block information corresponding to multiple data blocks, the method further includes: For each data block, an index calculation is performed according to the preset indexing rules to obtain the index information corresponding to each data block. And match the index information corresponding to each data block with the index information in the index table; Perform a virtual copy operation on the data blocks that match the index information in the index table; Furthermore, when the index information of at least one of the data blocks does not match any of the index information in the index table, at least one of the data blocks is written to the back-end storage device corresponding to the storage system.
3. The method according to claim 1, characterized in that, When the first index information matches the second index information in the index table, after performing a virtual copy operation on the data to be written in the data to be written information, the method further includes: Generate new metadata information corresponding to the data to be written; Establish the association between the new metadata information and the first index information.
4. The method according to claim 1, characterized in that, The step of performing index calculation on the data to be read according to a preset indexing rule to obtain the third index information corresponding to the data to be read specifically includes: A first-level index is established based on the file name and the position offset according to the preset indexing rules; And, based on the data to be read, perform fingerprint calculation to obtain the fingerprint value corresponding to the data to be read; Based on the fingerprint value and the data reading location information, a fingerprint key-value pair is generated as a secondary index; The first-level index and the second-level index together constitute the third index information.
5. The method according to claim 4, characterized in that, The data read location information includes the cache memory address and physical address information corresponding to the data to be read.
6. The method according to claim 4, characterized in that, After generating fingerprint key-value pairs as secondary indexes based on the fingerprint value and the data read location information, the method further includes: A fingerprint index tree is generated based on the secondary indexes corresponding to all the data to be read that have the same primary index.
7. A data processing apparatus, characterized in that, The device corresponds to a storage system and includes: The receiving module is used to receive data read requests sent by the client; The processing module is used to parse the data read request, obtain the data information to be read, which includes the file name to which the data to be read belongs, the position offset of the data to be read, and the data length; and determine the data read position information based on the file name, the position offset, and the data length. The sending module is used to send the data reading request to the backend storage device corresponding to the storage system based on the data reading location information; A receiving module is used to receive the data to be read sent by the backend storage device; The sending module is used to send the data to be read back to the client; The processing module is used to perform index calculation on the data to be read according to a preset index rule, and obtain the third index information corresponding to the data to be read; Store the third index information in the index table; The receiving module is also used to receive data write requests sent by the client; The extraction module is used to extract the data to be written information from the data write request when a data write request is received from the client. The processing module is also used to perform index calculation on the data information to be written according to a preset index rule to obtain the first index information; The matching module is used to match the first index information with the index information in the pre-built index table, wherein the index information in the index table is generated based on the previously read data; The copy module is used to perform a virtual copy operation on the data to be written in the data to be written information when the first index information matches the second index information in the index table.
8. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the data processing method as described in any one of claims 1 to 6.
9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data processing method as described in any one of claims 1 to 6.