Data storage method and device, management node, medium and data storage system

Abstract

This application provides a data storage method, apparatus, management node, medium, and data storage system. The data storage system is applied to a management node within a data storage system. The system further includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The multiple first storage nodes are used to respond to data read requests sent by terminal devices. The method includes: receiving a write request sent by the terminal device, the write request carrying first data; determining a first logical storage unit corresponding to the first data based on its type; wherein the data stored in the logical storage unit is mapped to each storage node; and appending the first data to the first logical storage unit, so that the first logical storage unit appends the first data to a first storage location in each storage node. This method effectively reduces energy loss caused by switching energy-saving levels.

Description

Technical Field

[0001] This application relates to the field of data storage technology, and in particular to a data storage method, apparatus, management node, medium and data storage system. Background Technology

[0002] With the rapid development of information technology, the demand for data storage has exploded. To meet this demand, distributed storage systems have emerged, employing various technologies to ensure high reliability and performance of data. In these systems, the hard drive, as the core component for data storage, has a significant impact on the overall system's operating efficiency and cost due to its energy consumption and performance.

[0003] Existing distributed storage systems typically employ a strategy of separating hot and cold data. Based on the size of the data's I / O operations, it is divided into hot and cold data, and stored separately in disk pools with different energy efficiency levels. Hot data is usually stored in low-energy-efficiency disk pools to meet the system's I / O performance requirements; while cold data is stored in high-energy-efficiency disk pools to save energy.

[0004] However, existing distributed storage systems cannot accurately distinguish between hot and cold data, and their energy consumption is high. Summary of the Invention

[0005] This application provides a data storage method, apparatus, management node, medium, and data storage system to solve the technical problems of frequent switching of energy-saving levels and high energy consumption in the prior art.

[0006] Firstly, this application provides a data storage method, including:

[0007] A data storage method is applied to a management node in a data storage system. The data storage system further includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The multiple first storage nodes are used to respond to data read requests sent by terminal devices. The method includes:

[0008] Receive a write request sent by the terminal device, the write request carrying the first data;

[0009] Based on the type of the first data, determine the first logical storage unit corresponding to the first data; wherein, the data stored in the logical storage unit will be mapped to each storage node;

[0010] The first data is appended to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node.

[0011] Alternatively, the method described above may also include:

[0012] The mapping relationship is updated based on the first data, the first logical storage unit, and the first storage location. The mapping relationship is used to represent the correspondence between data and stored information. The stored information includes the identifier of the logical storage unit and the storage location.

[0013] Alternatively, the method described above may also include:

[0014] Obtain the first data read request sent by the terminal device, the first data read request carrying the identifier of the second data;

[0015] Based on the identifier and mapping relationship of the second data, determine the second storage information corresponding to the second data;

[0016] The target first storage node is determined from all first storage nodes;

[0017] Send a second data read request to the target first storage node. The second data read request includes the identifier of the second data and the second storage information.

[0018] Obtain the second data sent by the target first storage node, which is obtained by the target first storage node based on the second data read request;

[0019] The second data is sent to the terminal device.

[0020] Optionally, after obtaining the first data read request sent by the terminal device, the method as described above further includes:

[0021] Determine the number of first-level storage nodes that are in operation in the data storage system;

[0022] If the number of active storage nodes in the storage node cluster is less than the preset number of nodes, a read failure message is returned to the terminal device.

[0023] Optionally, the method described above, before determining the target first storage node from all first storage nodes, further includes:

[0024] Obtain the new first data read request resent by the terminal device based on the read failure information;

[0025] Convert a preset number of second storage nodes into first storage nodes.

[0026] Optionally, in the method described above, the storage information also includes the data volume and the type of logical storage unit, where the logical storage unit type includes read-only and read-write. The method further includes:

[0027] Based on the mapping relationship, determine the data volume stored in the logical storage unit of the read / write type;

[0028] If the data volume of any logical storage unit of read / write type is greater than the preset data volume, the type of the logical storage unit will be updated from read / write to read-only, and the preset data volume will be less than or equal to the maximum storable data volume of the logical storage unit.

[0029] Optionally, in the method described above, the stored information also includes data status, which includes invalid and valid data. The method further includes:

[0030] Obtain a data modification request sent by the terminal device. The data modification request is used to instruct the third data to be replaced with the fourth data.

[0031] Based on the mapping relationship and the third data, the third storage information corresponding to the third data is determined. The third storage information includes the identifier of the second logical storage unit.

[0032] The fourth data is appended to the second logical storage unit corresponding to the identifier of the second logical storage unit, so that the second logical storage unit appends the fourth data to the second storage location of each storage node;

[0033] Update the mapping relationship between the fourth data, the second logical storage unit, and the second storage location;

[0034] Set the data status corresponding to the third data in the mapping relationship to invalid, and set the data status corresponding to the fourth data to valid.

[0035] Alternatively, the method described above may further include:

[0036] Based on the mapping relationship, determine the ratio of invalid data to valid data in the read-only logical storage unit;

[0037] If the proportion of any read-only logical storage unit is greater than the preset proportion, the valid data in the logical storage unit will be migrated to the updated logical storage unit.

[0038] The mapping relationship is updated based on the updated logical storage unit.

[0039] Secondly, this application provides a data storage device, comprising:

[0040] The receiving module is used to receive write requests sent by the terminal device, and the write requests carry first data.

[0041] The determination module is used to determine the first logical storage unit corresponding to the first data based on the type of the first data; wherein the data stored in the logical storage unit is mapped to each storage node;

[0042] The write module is used to append first data to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node.

[0043] Thirdly, this application provides a management node, including: a processor, and a memory communicatively connected to the processor;

[0044] The memory stores instructions that the computer executes;

[0045] The processor executes computer execution instructions stored in memory to implement the methods of the embodiments of this application.

[0046] Fourthly, this application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the methods of the embodiments of this application.

[0047] Fifthly, this application proposes a data storage system, including a management node, multiple first storage nodes, and multiple second storage nodes;

[0048] The management node is used to implement data storage methods such as those described in the first aspect;

[0049] The energy consumption of the first storage node is higher than that of the second storage node. The storage node is used to store data appended and written by the terminal device. Multiple first storage nodes are also used to respond to data read requests sent by the terminal device.

[0050] This application provides a data storage method, apparatus, management node, medium, and data storage system, applied to a management node in a data storage system. The data storage system further includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The multiple first storage nodes are used to respond to data read requests sent by terminal devices. The method includes: receiving a write request sent by a terminal device, the write request carrying first data; determining a first logical storage unit corresponding to the first data according to the type of the first data; wherein the data stored in the logical storage unit is mapped to each storage node; and appending the first data to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node. This method effectively reduces energy loss caused by switching energy-saving levels. Attached Figure Description

[0051] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0052] Figure 1A schematic diagram of the structure of a data storage system provided in this application;

[0053] Figure 2 A schematic flowchart illustrating an embodiment of a data storage method provided in this application;

[0054] Figure 3 A flowchart illustrating a second embodiment of a data storage method provided in this application;

[0055] Figure 4 A flowchart illustrating a data storage method according to a third embodiment of this application;

[0056] Figure 5 A flowchart illustrating a data storage method according to Embodiment 4 of this application;

[0057] Figure 6 A schematic diagram of the data writing process in a data storage method provided in this application;

[0058] Figure 7 A schematic diagram of the data invalidation process in a data storage method provided in this application;

[0059] Figure 8 A schematic diagram of the data migration process in a data storage method provided in this application;

[0060] Figure 9 A schematic diagram of the structure of the data storage device provided in the embodiments of this application;

[0061] Figure 10 This is a schematic diagram of the structure of the management node provided in an embodiment of this application.

[0062] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0063] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0064] While existing data storage systems dynamically allocate hot and cold hard drives and classify energy efficiency levels based on the size of real-time input and output data, enabling rapid responses to data changes, they have significant drawbacks. Because real-time data sizes are random, a hard drive may quickly face new data demands just as it enters a high energy efficiency level, forcing it to revert to a lower level. This frequent switching can actually lead to energy consumption exceeding the energy savings achieved during the high energy efficiency period.

[0065] To address the problems of existing technologies, the inventors recognized that by reducing frequent energy-saving level switching and optimizing data management, the system can maintain energy efficiency while responding to data requests efficiently. Based on this, the inventors proposed a method based on data aggregation and energy-saving level control. This method aggregates append-written data into a dedicated read-only replica set, using read-only replica sets and read-write replica sets to replace the traditional hot and cold data distribution method. By aggregating append-written data in the read-only replica set and keeping a large number of redundant data nodes at a high energy-saving level, while leaving only a small number of nodes at a low energy-saving level to handle read data requests, frequent energy-saving level switching is avoided.

[0066] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0067] Figure 1 This is a schematic diagram of a data storage system provided in this application. Figure 1 As shown, the data storage system includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The multiple first storage nodes are used to respond to data read requests sent by terminal devices.

[0068] In this system, the management node is responsible for managing and coordinating the entire data storage process. It works in conjunction with multiple storage nodes to determine data allocation and manage read / write strategies.

[0069] The first storage node refers to a low-energy-efficiency storage node, which is typically active and used to quickly respond to read requests from end devices. The second storage node is a high-energy-efficiency storage node, primarily used to store redundant data in the system. It is usually kept in a dormant or low-activity state to reduce energy consumption. The purpose of this design is to optimize energy use through a tiered storage structure.

[0070] Figure 2 This is a flowchart illustrating an embodiment of a data storage method provided in this application. Figure 2 As shown, the method includes:

[0071] S201, Receive a write request sent by the terminal device.

[0072] The write request carries first data. In this embodiment, the terminal device can be a client. The client sends a write request to the management node, including the first data to be written. The first data refers to the specific data content sent by the client or terminal device to the data storage system; it is the data object to be written to the data storage system, containing specific data values ​​or data content, i.e., the data object to be appended.

[0073] S202. Determine the first logical storage unit corresponding to the first data according to the type of the first data.

[0074] The data stored in the logical storage unit is mapped to each storage node. It is an object distributed across multiple storage nodes, that is, mapped to multiple storage nodes. This ensures that the data is appended to different nodes, forming a complete redundant storage set, thus realizing the reliability of data redundancy and data distribution.

[0075] In this embodiment, the management node determines the logical storage extent based on the type of the first data. For example, if the first data is log data or time-series data, it may be designated as the type that needs to be appended, thereby selecting the corresponding logical storage extent.

[0076] S203. The first data is appended to the first logical storage unit so that the first logical storage unit appends the first data to the first storage location of each storage node.

[0077] In this embodiment, after the logical storage unit is determined, the terminal device will append data to the logical storage unit. This process is actually carried out at the first storage location of multiple storage nodes.

[0078] Furthermore, the method also includes:

[0079] The mapping relationship is updated based on the first data, the first logical storage unit, and the first storage location. The mapping relationship is used to represent the correspondence between data and stored information. The stored information includes the identifier of the logical storage unit and the storage location.

[0080] In this embodiment, the mapping relationship is used to ensure that the data storage system can quickly and accurately locate the specific location of the data. Specifically, in the data storage system, after each data object (first data) is written, the system associates it with a logical storage unit (e.g., extent) and its specific storage location. When a terminal device appends data, the management node first allocates a logical storage unit (extent) and determines its specific storage location in the storage node (i.e., the first storage location). Then, the system associates the information of the first data (e.g., data identifier, size, offset, etc.) with the logical storage unit and storage location according to the write operation.

[0081] The system then records the relationship between the first data, the logical storage unit, and the storage location in a key-value storage unit (or other type of metadata management unit). The relationship record stores the identifier of the logical storage unit (e.g., the extent ID) and the specific location of the data within that logical storage unit (i.e., the storage offset on the specific storage node, file path, etc.).

[0082] After the system receives a write request and completes the data write operation, the management node updates the mapping relationship in the KV storage unit. For example, it creates a new entry or modifies an existing entry in the KV storage unit, making the identifier of the first data point to the updated logical storage unit and storage location. If the first data is a modification of previous data (such as updating an existing read-only data), the system will not actually change the original read-only data, but will modify the mapping relationship, pointing the data's mapping to the new extent. This process of modifying the mapping relationship will be explained in detail in subsequent embodiments.

[0083] The above embodiments describe a method for data writing and mapping management in a data storage system. By appending the first data sent by the terminal device to the logical storage unit, redundant storage of data is achieved on multiple storage nodes. Then, the mapping relationship between the data and the logical storage unit and its storage location is updated to ensure that the data storage system can quickly locate and manage the data, thereby improving data access and energy efficiency.

[0084] Figure 3 This is a flowchart illustrating a second embodiment of a data storage method provided in this application. Figure 3 As shown, this method describes the data reading process, including:

[0085] S301. Obtain the first data read request sent by the terminal device.

[0086] The first data read request carries an identifier for the second data. In this embodiment, when the terminal device sends a data read request, the request carries the identifier of the data to be read (i.e., the identifier of the second data). The management node receives this request and begins querying and locating the data based on the identifier of the second data.

[0087] Furthermore, after obtaining the first data read request sent by the terminal device, the method also includes:

[0088] Determine the number of first-level storage nodes that are in operation in the data storage system;

[0089] If the number of active storage nodes in the storage node cluster is less than the preset number of nodes, a read failure message is returned to the terminal device.

[0090] This embodiment describes how, when processing a data read request, the system determines whether it is capable of reading data by monitoring the number of low-energy-efficiency nodes in the system. If the number of working nodes is insufficient, the system will prioritize energy efficiency and stability, returning a read failure message to the terminal device to avoid unnecessary energy consumption and system resource waste.

[0091] For example, when a mobile terminal issues a data read request, the system first receives the request and prepares to perform data location and read operations. Then, it needs to determine how many primary storage nodes (low-energy-efficiency level nodes) are currently active and available. This step is typically performed by the management node, which monitors and manages the health and operational status (e.g., online, offline, or dormant) of all nodes. Next, it determines if the number of primary storage nodes is sufficient: the system compares the number of currently active primary storage nodes with a preset number of nodes. This preset number is a threshold set to ensure the stability of data reads and is usually configured based on system size and load. If the number of active primary storage nodes is insufficient to meet the requirement of direct wake-up and operation, the system considers the current health of the read-only replica set insufficient to complete the read operation. Finally, it returns a read failure message: when the system detects an insufficient number of primary storage nodes, the management node will not continue sending read requests but will directly return a read failure message to the terminal device. This avoids unnecessary resource waste (e.g., attempting reads that fail) and informs the terminal device of the current system status.

[0092] S302. Determine the second storage information corresponding to the second data based on the identifier and mapping relationship of the second data.

[0093] In this embodiment, the data storage system utilizes the mapping relationship maintained in the KV storage to locate and determine the second storage information corresponding to the data based on the "identifier of the second data". This storage information includes the identifier (extent ID) of the logical storage unit where the data resides and its specific location within the storage node (e.g., file path or offset). This process ensures that the system can quickly and accurately locate the data within the storage node for subsequent read operations.

[0094] S303. Determine the target first storage node from all first storage nodes.

[0095] In this embodiment, the data storage system selects one target node from multiple first storage nodes (low energy efficiency level nodes) as the target node for performing data read operations. This selection is typically based on the current system load and node availability. For example, if there are enough low energy efficiency level storage nodes, the data storage system preferentially selects the target storage node from these low energy efficiency level storage nodes to reduce the number of switching to high energy efficiency level nodes and maintain the energy efficiency of the data storage system.

[0096] Furthermore, before determining the target first storage node from all first storage nodes, the method also includes:

[0097] Obtain the new first data read request resent by the terminal device based on the read failure information;

[0098] Convert a preset number of second storage nodes into first storage nodes.

[0099] The purpose of this embodiment is to ensure that the system has sufficient resources to complete data read operations when the currently online low-energy-efficiency node (first storage node) is insufficient to handle requests. By converting a high-energy-efficiency node (second storage node) to a low-energy-efficiency node (first storage node), the system can minimize the switching frequency of high-energy-efficiency nodes while ensuring read success rates, thus avoiding unnecessary energy consumption.

[0100] For example, when the system detects that the number of active first storage nodes is insufficient and returns a read failure message to the terminal device, the terminal device will resend the data read request. This step is the system receiving the client's second attempt request. At this point, the system determines that the reason for the first read request failure may be the insufficient number of first storage nodes, and therefore further measures are needed to process the new read request.

[0101] To ensure the system can successfully read data after receiving a new read request, the management node selects several nodes at a high energy efficiency level (i.e., secondary storage nodes) and converts them to primary storage nodes. Specifically, the system wakes up these high energy efficiency level storage nodes and transitions their state from high energy efficiency (dormant or inactive) to low energy efficiency (active or online). This transition enables these nodes to respond to data read requests. The number of nodes converted is typically a preset value configured by the system to ensure sufficient nodes can handle potential read I / O loads and maintain a balance between overall system power consumption and performance.

[0102] S304. Send a second data read request to the target first storage node.

[0103] The second data read request includes the identifier of the second data and the second storage information. For example, the management node sends a data read request to the selected target first storage node, which includes the identifier of the second data and its storage information (i.e., extent ID and storage location), thereby instructing the target node to read the data from the specified location in its storage device.

[0104] S305, Obtain the second data sent by the target first storage node.

[0105] The second data is obtained by the target first storage node based on the second data read request. For example, after receiving the read request, the target first storage node retrieves the data from its local storage device according to the storage information in the request and returns the data (the second data) to the management node.

[0106] S306. Send the second data to the terminal device.

[0107] In this implementation, the management node sends the second data obtained from the target first storage node to the terminal device that initiated the request, thus completing the entire data reading process.

[0108] The above embodiments provide a data read request processing mechanism. When a terminal device sends a data read request, the system first determines the data location according to the mapping relationship and responds to the request through available low energy-saving level nodes. If the number of currently online low energy-saving level nodes is insufficient, the system will return a read failure message. After receiving a reread request from the client, the system will wake up and convert a preset number of high energy-saving level nodes into low energy-saving level nodes to ensure the smooth completion of the data read operation.

[0109] Figure 4 This is a flowchart illustrating a third embodiment of a data storage method provided in this application. Figure 4As shown, the storage information also includes data volume and the type of logical storage unit, which includes read-only and read-write. The method also includes:

[0110] S401. Determine the data volume stored in the logical storage unit of the read / write type according to the mapping relationship.

[0111] In this embodiment, the system can obtain all logical storage units of "read / write type" and their corresponding data volumes through mapping relationships. This process is typically performed periodically by the management node to monitor the usage and capacity of each logical storage unit.

[0112] S402. If the data volume of any logical storage unit of read / write type is greater than the preset data volume, then the type of the logical storage unit is updated from read / write to read-only.

[0113] The preset data volume is less than or equal to the maximum storable data volume of the logical storage unit. This preset data volume is a threshold used to determine when to convert the logical storage unit from "read-write" to "read-only". The preset data volume must be less than or equal to the maximum storable data volume of the logical storage unit to ensure that the encapsulation operation is carried out within a safe range.

[0114] For example, when the system detects that the data volume of a logical storage unit of a certain read-write type exceeds a preset data volume, it updates the state of the logical storage unit to "read-only". After being updated to "read-only", the logical storage unit will no longer allow new data to be appended, and the system will also update its state in the mapping relationship and KV storage. The encapsulated read-only logical storage unit can be moved by the system to a high-energy-efficiency storage node, or alternatively, redundant data in the read-only logical storage unit can be moved to a high-energy-efficiency storage node, while a sufficient number of copies are sent to a low-energy-efficiency storage node to receive scheduling requests, thereby reducing energy consumption while maintaining data integrity and redundancy.

[0115] For example, management nodes or storage nodes in a data storage system continuously monitor the data writing status of each logical storage unit. When a terminal device appends the "first data" to a logical storage unit, the management node records the current size and status of that logical storage unit.

[0116] When a logical storage unit reaches a preset threshold or maximum capacity (e.g., a certain file size or data block size), the management node triggers a sealing operation. This preset size is set based on the data storage system design and performance requirements to ensure that the logical storage unit does not expand indefinitely while guaranteeing storage and retrieval efficiency.

[0117] Once a logical storage unit reaches its set size, the data storage system will "encapsulate" it. This process typically includes: closing the write channel: stopping write operations on the logical storage unit to prevent new data from being appended; state transition: changing the state of the logical storage unit from "read-write" to "read-only," meaning that any subsequent write requests will be rejected or redirected to a newer logical storage unit; and the management node updating the state information of the logical storage unit, marking it as "read-only," and associating it with a read-only replica set so that the system knows it no longer accepts write operations and can only be used for reading.

[0118] After a logical storage unit is encapsulated as read-only, the data system migrates the redundant data corresponding to that logical storage unit to a high-energy-efficiency storage node (i.e., a secondary storage node) according to an energy-saving strategy. These nodes remain in a dormant or low-power state in read-only mode to reduce energy consumption.

[0119] Logical storage units packaged as read-only may be scanned and processed subsequently based on the proportion of invalid data. For example, when the proportion of invalid data in these logical storage units is large, the system will migrate and aggregate valid data into new read-only logical storage units to improve storage efficiency. This migration and aggregation process will be explained in detail in subsequent embodiments.

[0120] This application embodiment monitors the data volume of read-write logical storage units. When the data volume exceeds a preset threshold, the unit's status is updated to read-only, thereby controlling the data storage scale and saving resources. This method not only improves the system's data management efficiency but also ensures the effective implementation of energy-saving strategies.

[0121] Figure 5 This is a flowchart illustrating a fourth embodiment of a data storage method provided in this application. Figure 5 As shown, based on Embodiment 3, the stored information also includes data status, which includes invalid and valid data. The method further includes:

[0122] S501, Obtain the data modification request sent by the terminal device.

[0123] The data modification request is used to instruct the replacement of third data with fourth data. Specifically, when the system receives a data modification request from the terminal device, indicating that existing data (third data) needs to be replaced with new data (fourth data), this request may be due to the terminal device needing to update or overwrite previously stored data objects.

[0124] S502. Based on the mapping relationship and the third data, determine the third storage information corresponding to the third data.

[0125] The third storage information includes the identifier of the second logical storage unit. The system locates the specific position and information (third storage information) of the third data within the data storage system by querying the mapping relationship, including the identifier of the logical storage unit (i.e., the second logical storage unit) where the third data resides. This process ensures that the system can pinpoint the current data location for replacement operations.

[0126] S503. The fourth data is appended to the second logical storage unit corresponding to the identifier of the second logical storage unit, so that the second logical storage unit appends the fourth data to the second storage location of each storage node.

[0127] In this embodiment, after determining the location of the third data, the system appends the fourth data (new data) to the corresponding logical storage unit (i.e., the second logical storage unit). This step is performed on different storage nodes of the second logical storage unit, ensuring that the new data is redundantly stored on multiple nodes, meeting the system's data distribution and redundancy requirements. This process maintains the consistency of the logical storage unit, that is, adding the fourth data to the new storage location of the second logical storage unit.

[0128] S504. Update the correspondence between the fourth data, the second logical storage unit, and the second storage location to the mapping relationship.

[0129] After the fourth data is successfully written, the system updates the mapping relationship between the fourth data, the second logical storage unit, and the second storage location. This update ensures that the system can correctly locate the data when subsequently reading or manipulating it. Simultaneously, updating the mapping relationship helps maintain the consistency and accuracy of data access.

[0130] S505. Set the data status corresponding to the third data in the mapping relationship to invalid, and set the data status corresponding to the fourth data to valid.

[0131] In this step, the system sets the data status of the third data to "invalid" through a mapping relationship. This indicates that the third data has been logically replaced; even if the physical data still exists, it is no longer valid. Then, the system sets the status of the fourth data to "valid," indicating that it is the current valid data object and should be accessed normally by the system and clients. Thus, data validity and invalidity are managed through logical status updates rather than the deletion or overwriting of physical data.

[0132] Furthermore, the method also includes:

[0133] Based on the mapping relationship, determine the ratio of invalid data to valid data in the read-only logical storage unit;

[0134] If the proportion of any read-only logical storage unit is greater than the preset proportion, the valid data in the logical storage unit will be migrated to the updated logical storage unit.

[0135] The mapping relationship is updated based on the updated logical storage unit.

[0136] In this embodiment, the system queries the ratio of invalid to valid data in each read-only logical storage unit (i.e., an encapsulated extent that is read-only) through a mapping relationship. Invalid data refers to data that has been marked as invalid through previous data modification or replacement operations, while valid data is the current data that still needs to be retained and accessed.

[0137] The system periodically or according to a policy scans these read-only logical storage units to determine the proportion of invalid and valid data in each extent.

[0138] The system then compares the ratio of invalid to valid data with a preset threshold (preset ratio). If the proportion of invalid data in a read-only logical storage unit is greater than the preset ratio, it indicates that the invalid data ratio in this extent is high, and data optimization is required.

[0139] The preset ratio is usually configured by the system administrator to ensure that the system can make efficient use of storage space without wasting resources storing a large amount of invalid data.

[0140] When the proportion of invalid data in a read-only logical storage unit exceeds a preset value, the system will initiate a data migration operation to extract the valid data in that logical storage unit and transfer it to a newer logical storage unit.

[0141] This updated logical storage unit may be a new read-write or read-only extent, depending on the data migration strategy. For example, if the updated logical storage unit is a read-only extent, the number of the first and second storage nodes can be adjusted according to preset quantity information, that is, changing the number of high-energy-efficiency storage nodes and low-energy-efficiency storage nodes to meet actual work needs. This series of migration operations ensures that only currently needed data is retained and releases invalid data space in the original logical storage unit, helping to reduce redundancy and waste in the storage system and improve storage efficiency.

[0142] After the data migration is complete, the system updates the mapping relationship, making the data entries that originally pointed to the old logical storage units point to the updated logical storage units and their storage locations. This update process ensures that the system can correctly access the migrated valid data, while discarding references to invalid data in the old logical storage units, thus completing a seamless data switchover.

[0143] The above application embodiments describe a method for data updating and optimization. When data is modified, the system writes new data and updates the data status according to the mapping relationship, and marks the old data as invalid. For read-only logical storage units, when the proportion of invalid data exceeds a preset value, the system migrates the valid data to the updated logical storage unit and updates the mapping relationship to release storage space and improve storage efficiency.

[0144] Figure 6 This is a schematic diagram of the data writing process in a data storage method provided in this application.

[0145] Figure 7 This is a schematic diagram of the data invalidation process in a data storage method provided in this application.

[0146] Figure 8 This is a schematic diagram of the data migration process in a data storage method provided in this application.

[0147] Figure 9 This is a schematic diagram of the data storage device provided in an embodiment of this application. Figure 9 As shown, the data storage device 90 includes a receiving module 901, a determining module 902, and a writing module 903. Specifically,

[0148] The receiving module 901 is used to receive a write request sent by the terminal device, the write request carrying first data;

[0149] The determining module 902 is used to determine the first logical storage unit corresponding to the first data based on the type of the first data; wherein the data stored in the logical storage unit is mapped to each storage node;

[0150] The write module 903 is used to append first data to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node.

[0151] Furthermore, the receiving module 901 is specifically used for:

[0152] The mapping relationship is updated based on the first data, the first logical storage unit, and the first storage location. The mapping relationship is used to represent the correspondence between data and stored information. The stored information includes the identifier of the logical storage unit and the storage location.

[0153] Furthermore, the receiving module 901 is specifically used for:

[0154] Obtain the first data read request sent by the terminal device, the first data read request carrying the identifier of the second data;

[0155] Based on the identifier and mapping relationship of the second data, determine the second storage information corresponding to the second data;

[0156] The target first storage node is determined from all first storage nodes;

[0157] Send a second data read request to the target first storage node. The second data read request includes the identifier of the second data and the second storage information.

[0158] Obtain the second data sent by the target first storage node, which is obtained by the target first storage node based on the second data read request;

[0159] The second data is sent to the terminal device.

[0160] Furthermore, the receiving module 901 is specifically used for:

[0161] Determine the number of first-level storage nodes that are in operation in the data storage system;

[0162] If the number of active storage nodes in the storage node cluster is less than the preset number of nodes, a read failure message is returned to the terminal device.

[0163] Furthermore, the receiving module 901 is specifically used for:

[0164] Obtain the new first data read request resent by the terminal device based on the read failure information;

[0165] Convert a preset number of second storage nodes into first storage nodes.

[0166] Furthermore, the receiving module 901 is specifically used for:

[0167] Based on the mapping relationship, determine the data volume stored in the logical storage unit of the read / write type;

[0168] If the data volume of any logical storage unit of read / write type is greater than the preset data volume, the type of the logical storage unit will be updated from read / write to read-only, and the preset data volume will be less than or equal to the maximum storable data volume of the logical storage unit.

[0169] Furthermore, the receiving module 901 is specifically used for:

[0170] Obtain a data modification request sent by the terminal device. The data modification request is used to instruct the third data to be replaced with the fourth data.

[0171] Based on the mapping relationship and the third data, the third storage information corresponding to the third data is determined. The third storage information includes the identifier of the second logical storage unit.

[0172] The fourth data is appended to the second logical storage unit corresponding to the identifier of the second logical storage unit, so that the second logical storage unit appends the fourth data to the second storage location of each storage node;

[0173] Update the mapping relationship between the fourth data, the second logical storage unit, and the second storage location;

[0174] Set the data status corresponding to the third data in the mapping relationship to invalid, and set the data status corresponding to the fourth data to valid.

[0175] Furthermore, the receiving module 901 is specifically used for:

[0176] Based on the mapping relationship, determine the ratio of invalid data to valid data in the read-only logical storage unit;

[0177] If the proportion of any read-only logical storage unit is greater than the preset proportion, the valid data in the logical storage unit will be migrated to the updated logical storage unit.

[0178] The mapping relationship is updated based on the updated logical storage unit.

[0179] Figure 10 This is a schematic diagram of the structure of the management node provided in an embodiment of this application. Figure 10 As shown, the management node 100 includes:

[0180] The system includes a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a communication component 103, and other components. The processor 101, memory 102, and communication component 103 are connected via a bus 104.

[0181] In the specific implementation process, at least one processor 101 executes computer execution instructions stored in memory 102, causing at least one processor 101 to execute the data storage method described above.

[0182] The specific implementation process of processor 101 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0183] In the above Figure 7In the illustrated embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0184] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0185] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0186] In some embodiments, a computer program product is also provided, comprising a computer program or instructions that, when executed by a processor, implement the steps in any of the above data storage methods.

[0187] For details on the implementation of each of the above operations, please refer to the previous examples, which will not be repeated here.

[0188] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be performed by instructions, or by instructions controlling related hardware. These instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0189] Therefore, embodiments of this application provide a computer-readable storage medium storing a plurality of instructions that can be loaded by a processor to execute steps in any of the data storage methods provided in embodiments of this application.

[0190] The storage medium may include: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0191] Since the instructions stored in the storage medium can execute the steps of any of the data storage methods provided in the embodiments of this application, the beneficial effects that any of the data storage methods provided in the embodiments of this application can achieve can be realized. For details, please refer to the previous embodiments, which will not be repeated here.

[0192] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0193] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this application.

[0194] It should be further noted that although the steps in the flowchart are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0195] Furthermore, unless otherwise specified, the functional units / modules in the various embodiments of this application can be integrated into one unit / module, or each unit / module can exist physically separately, or two or more units / modules can be integrated together. The integrated units / modules described above can be implemented in hardware or as software program modules.

[0196] When integrated units / modules are implemented in hardware, the hardware can be digital circuits, analog circuits, etc. The physical implementation of the hardware structure includes, but is not limited to, transistors, memristors, etc. Unless otherwise specified, the processor can be any suitable hardware processor, such as a CPU, GPU, FPGA, DSP, and ASIC, etc. Unless otherwise specified, the storage unit can be any suitable magnetic or magneto-optical storage medium, such as Resistive Random Access Memory (RRAM), Dynamic Random Access Memory (DRAM), Static Random Access Memory (SRAM), Enhanced Dynamic Random Access Memory (EDRAM), High-Bandwidth Memory (HBM), Hybrid Memory Cube (HMC), etc.

[0197] If the integrated unit / module is implemented as a software program module and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0198] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0199] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A data storage method, characterized in that, The management node is applied in a data storage system, which further includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The first storage nodes refer to low-energy-efficiency storage nodes, which represent nodes in an active state. The multiple first storage nodes are used to respond to data read requests sent by terminal devices. The second storage nodes are high-energy-efficiency storage nodes, which are used to store redundant data. The high-energy-efficiency storage nodes are kept in a dormant or low-activity state. The method includes: Receive a write request sent by the terminal device, the write request carrying first data; Based on the type of the first data, determine the first logical storage unit corresponding to the first data; wherein, the data stored in the logical storage unit will be mapped to each storage node; The first data is appended to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node; The method further includes: Obtain a first data read request sent by the terminal device, wherein the first data read request carries an identifier of the second data; Determine the number of first storage nodes in the data storage system that are in working state; if the number of first storage nodes in the storage node cluster that are in working state is less than the preset number of nodes, return a read failure message to the terminal device; Based on the identifier and mapping relationship of the second data, determine the second storage information corresponding to the second data; Obtain the new first data read request resent by the terminal device based on the read failure information; convert a preset number of second storage nodes into first storage nodes; A target first storage node is determined from all first storage nodes; a second data read request is sent to the target first storage node, the second data read request including the identifier of the second data and the second storage information; the second data sent by the target first storage node is obtained by the target first storage node based on the second data read request; and the second data is sent to the terminal device.

2. The method according to claim 1, characterized in that, The method further includes: The mapping relationship is updated based on the first data, the first logical storage unit, and the first storage location. The mapping relationship is used to represent the correspondence between data and storage information. The storage information includes the identifier of the logical storage unit and the storage location.

3. The method according to claim 2, characterized in that, The storage information also includes data volume and the type of logical storage unit, wherein the type of logical storage unit includes read-only and read-write, and the method further includes: Based on the mapping relationship, determine the data volume stored in the logical storage unit of the read / write type; If the data volume of any logical storage unit of read / write type is greater than the preset data volume, then the type of the logical storage unit is updated from read / write to read-only, and the preset data volume is less than or equal to the maximum storable data volume of the logical storage unit.

4. The method according to claim 3, characterized in that, The storage information also includes data status, which includes invalid and valid data status; the method further includes: Obtain a data modification request sent by the terminal device, the data modification request being used to instruct the third data to be replaced with the fourth data; Based on the mapping relationship and the third data, the third storage information corresponding to the third data is determined, and the third storage information includes the identifier of the second logical storage unit; The fourth data is appended to the second logical storage unit corresponding to the identifier of the second logical storage unit, so that the second logical storage unit appends the fourth data to the second storage location of each storage node; Update the mapping relationship to the correspondence between the fourth data, the second logical storage unit, and the second storage location; Set the data status corresponding to the third data in the mapping relationship to invalid, and set the data status corresponding to the fourth data to valid.

5. The method according to claim 4, characterized in that, The method further includes: Based on the mapping relationship, determine the ratio of invalid data to valid data in the read-only logical storage unit; If the proportion of any read-only logical storage unit is greater than the preset proportion, then the valid data in the logical storage unit will be migrated to the updated logical storage unit. The mapping relationship is updated based on the updated logical storage unit.

6. A data storage device, characterized in that, The management node is used in a data storage system, which also includes multiple first storage nodes and multiple second storage nodes. The energy consumption of the first storage nodes is higher than that of the second storage nodes. The first storage nodes are low-energy-efficiency storage nodes, representing nodes in an active state. The multiple first storage nodes are used to respond to data read requests sent by terminal devices. The second storage nodes are high-energy-efficiency storage nodes used to store redundant data. The high-energy-efficiency storage nodes remain in a dormant or low-activity state. The data storage device includes: A receiving module is used to receive a write request sent by a terminal device, wherein the write request carries first data; The determining module is used to determine the first logical storage unit corresponding to the first data based on the type of the first data; wherein the data stored in the logical storage unit is mapped to each storage node; The write module is used to append the first data to the first logical storage unit, so that the first logical storage unit appends the first data to the first storage location of each storage node; The receiving module is further configured to: acquire a first data read request sent by the terminal device, the first data read request carrying an identifier of second data; determine the number of first storage nodes in the data storage system that are in working state; if the number of first storage nodes in the storage node cluster that are in working state is less than a preset number of nodes, return read failure information to the terminal device; determine the second storage information corresponding to the second data according to the identifier of the second data and the mapping relationship; acquire a new first data read request resent by the terminal device based on the read failure information; convert a preset number of second storage nodes into first storage nodes; determine a target first storage node from all first storage nodes; send a second data read request to the target first storage node, the second data read request including the identifier of the second data and the second storage information; acquire the second data sent by the target first storage node, the second data being acquired by the target first storage node based on the second data read request; and send the second data to the terminal device.

7. A management node, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1 to 5.

9. A data storage system, characterized in that, It includes a management node, multiple primary storage nodes, and multiple secondary storage nodes; The management node is used to implement the data storage method as described in any one of claims 1-5; The energy consumption of the first storage node is higher than that of the second storage node. The storage node is used to store data appended and written by the terminal device. The plurality of first storage nodes are also used to respond to data read requests sent by the terminal device.

Images