Method, device, electronic equipment and storage control chip for data storage

Abstract

The application discloses a data storage method and device, electronic equipment and storage control chip. The method comprises the following steps: obtaining a data processing request issued by a host, wherein the data processing request comprises at least one of a read request and a write request; in the case that the data processing request is the write request, analyzing the write request to determine whether the write request needs to be processed by a disk array; in the case that it is determined that the write request needs to be processed by the disk array, calling the data processing module to process the write request; and in the case that it is determined that the processed write request needs to be cached, calling the cache interface module to write the write request into the external cache module. Through the storage control chip as the core of scheduling, the data packet can be analyzed, then unified scheduling processing and caching are performed, and the data processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computers, and more specifically, to a method, apparatus, electronic device, and storage control chip for data storage. Background Technology

[0002] In today's data center and enterprise storage environments, data processing efficiency is one of the key metrics for measuring storage performance. With the explosive growth of data volume and the increasing complexity of data access patterns, improving data processing efficiency has become a core requirement in storage system design.

[0003] However, the data processing efficiency of storage systems in related technologies is low. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage control chip for data storage, in order to at least solve the problem of low data processing efficiency in related technologies.

[0005] This application provides a data storage method applied to a storage control chip. The storage control chip includes a data processing module and a cache interface module. The cache interface module is used to connect to an external cache module. The method includes: obtaining a data processing request issued by a host, wherein the data processing request includes at least one of the following: a read request and a write request; if the data processing request is a write request, parsing the write request to determine whether the write request needs to be processed by a disk array; if it is determined that the write request needs to be processed by a disk array, invoking the data processing module to process the write request; if it is determined that the processed write request needs to be cached, invoking the cache interface module to write the write request into the external cache module.

[0006] A storage control chip includes: a data processing module for performing disk array processing on data; a cache interface module for connecting to an external cache module; a host interface module for connecting to an external host; and a control module connected to the host interface module, the data processing module, and the cache interface module, respectively, for acquiring data processing requests issued by the host, wherein the data processing request includes at least one of the following: a read request and a write request; if the data processing request is a write request, parsing the write request to determine whether the write request needs to be processed by disk array; if it is determined that the write request needs to be processed by disk array, calling the data processing module to process the write request; and if it is determined that the processed write request needs to be cached, calling the cache interface module to write the write request into the external cache module.

[0007] This application also provides a data storage apparatus, comprising: a request acquisition module, configured to acquire a data processing request issued by a host, wherein the data processing request includes at least one of the following: a read request and a write request;

[0008] The request parsing module is used to parse the write request to determine whether the write request needs to be processed by disk array when the data processing request is the write request.

[0009] The calling module is used to call the data processing module to process the write request when it is determined that the write request needs to be processed by the disk array.

[0010] The write module is used to call the cache interface module to write the write request into the external cache module when it is determined that the processed write request needs to be cached.

[0011] This application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the data storage method described above when executing the computer program.

[0012] This application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-described data storage methods.

[0013] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the above-described data storage methods.

[0014] This application achieves several key improvements. First, it ensures smooth communication between the host and storage through a high-performance PCIe interface and fast request parsing capabilities. Second, the hardware-accelerated RAID data processing module significantly improves data write efficiency and data security. Finally, the integration of the cache interface module with a high-speed external cache not only reduces write latency but also optimizes the data persistence process. The storage controller chip can first determine the data processing needs and then call the corresponding modules to process the data accordingly, improving data processing efficiency. As the core of scheduling, the storage controller chip can analyze data packets and then perform unified scheduling and caching, further enhancing data processing efficiency. Attached Figure Description

[0015] To more clearly illustrate the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a hardware structure block diagram of a server device for a data storage method according to an embodiment of this application;

[0017] Figure 2 This is a flowchart of a data storage method according to an embodiment of this application;

[0018] Figure 3 This is a second flowchart of a data storage method according to an embodiment of this application;

[0019] Figure 4 This is a third flowchart of a data storage method according to an embodiment of this application;

[0020] Figure 5 This is a fourth flowchart of a data storage method according to an embodiment of this application;

[0021] Figure 6 This is the fifth flowchart of a data storage method according to an embodiment of this application;

[0022] Figure 7 This is a flowchart of a data storage method according to an embodiment of this application;

[0023] Figure 8 This is the seventh flowchart of a data storage method according to an embodiment of this application;

[0024] Figure 9 This is the eighth flowchart of a data storage method according to an embodiment of this application;

[0025] Figure 10 This is flowchart nine of a data storage method according to an embodiment of this application;

[0026] Figure 11 This is flowchart ten of a data storage method according to an embodiment of this application;

[0027] Figure 12 This is flowchart eleven of a data storage method according to an embodiment of this application;

[0028] Figure 13 This is flowchart 12 of a data storage method according to an embodiment of this application;

[0029] Figure 14This is a schematic diagram of the structure of a storage control chip according to an embodiment of this application;

[0030] Figure 15 This is a second schematic diagram of the structure of a storage control chip according to an embodiment of this application;

[0031] Figure 16 This is a structural block diagram of a data storage device according to an embodiment of this application. Detailed Implementation

[0032] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of this application.

[0033] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "first," "second," etc., in this application are used to distinguish similar objects and are not used to describe a specific order or sequence.

[0034] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0035] The specific application environment architecture or specific hardware architecture on which the data storage method depends is described here.

[0036] The data storage method embodiments provided in this application can be executed on a server device or a similar computing device. Taking running on a server device as an example, Figure 1 This is a hardware structure block diagram of a server device for a data storage method according to an embodiment of this application. Figure 1 As shown, the server device may include one or more ( Figure 1Only one is shown in the image. A processor 102 (which may include, but is not limited to, a central processing unit (CPU), microprocessor (MCU), or programmable logic device (FPGA), etc.) and a memory 104 for storing data are also shown. The server device may further include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the server equipment described above. For example, the server equipment may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0037] The memory 104 can be used to store computer programs, such as application software programs and modules, like the computer program corresponding to the data storage method in this embodiment. The processor 102 executes various functional applications and data processing by running the computer program stored in the memory 104, thus implementing the aforementioned method. The memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to server devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0038] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the server device. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0039] Embodiments of this application provide a data storage method applied to a storage control chip. The storage control chip includes a data processing module and a cache interface module. The cache interface module is used to connect to an external cache module. The method is described in detail below, in conjunction with its execution flow. Figure 2 As shown, the method includes the following steps S200-230:

[0040] Step S200: Obtain the data processing request sent by the host.

[0041] The data processing request includes at least one of the following: a read request and a write request.

[0042] Specifically, the storage controller (I / O Controller, IOC) receives data processing requests from the host. These requests may include read or write requests, depending on the host's access needs for storage resources. This efficient data processing request receiving mechanism ensures uninterrupted communication between the host and the storage system, reducing request processing wait times and potential communication latency.

[0043] For example, the host system sends data processing requests to the uplink interface of the storage controller chip via the Peripheral Component Interconnect Express (PCIe) bus. The storage controller chip has a high-performance PCIe interface, which can quickly and accurately receive and parse the instructions issued by the host.

[0044] Step S210: If the data processing request is a write request, parse the write request to determine whether the write request needs to be processed by the disk array.

[0045] Specifically, when the storage controller chip receives a write request, the data processing module performs in-depth analysis to determine whether the write operation involves Redundant Array of Independent Disks (RAID). RAID technology provides data protection and performance enhancement through data redundancy and striped distribution. Through an intelligent analysis mechanism, the storage controller chip can quickly decide whether to enable RAID processing based on the specific content of the write request. This not only optimizes the data write process but also ensures data security and consistency.

[0046] For example, the metadata of the write request is parsed to check if it involves a RAID partition or RAID operation. For write requests that require RAID processing, the module further breaks down the request, identifies the specific disk and location to be written to, and calculates the parity data or mirroring data required for RAID.

[0047] Step S220: If it is determined that the write request needs to be processed by the disk array, the data processing module is invoked to process the write request.

[0048] Specifically, if a write request is determined to require disk array processing, the storage controller chip will invoke its built-in RAID data processing module to execute the corresponding RAID algorithm. This process includes data striping, parity calculation, or mirroring, depending on the specific RAID level. Hardware-accelerated RAID algorithms shorten data write processing time and improve the throughput and response speed of the storage system.

[0049] For example, the RAID data processing module employs hardware acceleration technologies, such as parallel computing units and dedicated chips, to efficiently execute RAID algorithms. These algorithms may be simple striping (RAID 0), mirroring (RAID 1), striping with parity (RAID 5), or more complex RAID 6, etc.

[0050] The RAID data processing module is integrated into the storage controller chip and is used to execute RAID algorithms to perform operations such as data striping, parity calculation, or mirroring, thereby improving the security and efficiency of data transmission.

[0051] Step S230: If it is determined that the processed write request needs to be cached, the cache interface module is called to write the write request to the external cache module.

[0052] Specifically, after RAID processing, if it is determined that the processed data needs to be cached, the storage controller chip will call the cache interface module to write the data to a high-speed external cache module, such as a Non-Volatile Memory Express Solid State Drive (NVMe SSD), as a data buffer. This step utilizes the high-speed read and write capabilities of the cache, avoiding the performance bottleneck caused by directly writing to slower back-end storage devices (such as Hard Disk Drives (HDDs) or Serial Advanced Technology Attachment Solid State Drives (SATA SSDs)). By writing data to the high-speed cache first, not only is write latency significantly reduced and the write performance of the storage system improved, but the burden on the back-end storage devices is also reduced.

[0053] For example, the cache interface module conforms to the NVMe protocol specification and writes the RAID-processed data into the NVMe SSD in a suitable format.

[0054] In this embodiment, firstly, the high-performance PCIe interface and fast request parsing capabilities ensure smooth communication between the host and storage. Secondly, the hardware-accelerated RAID data processing module significantly improves data writing efficiency and data security. Finally, the integration of the cache interface module with a high-speed external cache not only reduces write latency but also optimizes the data persistence process. The storage control chip can first determine the data processing needs and then call the corresponding modules to process the data accordingly, improving data processing efficiency. Using the storage control chip as the core of scheduling, it can analyze data packets and then perform unified scheduling and caching, further improving data processing efficiency.

[0055] In one embodiment, such as Figure 3 As shown, in step S210, if the data processing request is a write request, the write request is parsed to determine whether disk array processing is required. This includes steps S300-S330:

[0056] Step S300: Parse the write request to determine the target address and data packet parameters of the write request.

[0057] Specifically, after receiving a write request from the host, the storage controller chip parses the content of the write request to determine the target storage address and the specific parameters of the data packet. This step aims to better understand and process the host's write requirements. Through precise parsing of the write request, the storage controller chip can accurately determine where and how the data should be stored, ensuring correct data writing.

[0058] For example, the data processing module uses predefined parsing rules to extract key parameters, including the target address, data size, and data type, from the write request data packet. These parameters will be used in subsequent decision-making processes. The parsing process typically involves reading and analyzing the packet header to identify this important metadata information.

[0059] The destination address is the physical or logical location where the data to be written will be stored, typically specified by the host in the write request. Data packet parameters are additional information carried in the write request, such as data size, data type, and priority, used to guide the storage controller chip on how to process the write operation.

[0060] Step S310: Query the disk array configuration information of the data processing module to determine whether the target address of the write request is located in the disk array configured by the data processing module.

[0061] And, in step S320, determine whether the write request requires disk array processing based on the data packet parameters.

[0062] The disk array processing requirements include at least one of the following: striping processing requirements, mirroring processing requirements, and verification processing requirements.

[0063] Specifically, after parsing the write request, the storage controller chip will continue to query the disk array configuration information stored in the data processing module to determine whether the target address of the write request is within the currently configured disk array range. This step is crucial for deciding whether to perform RAID operations. Accurately identifying whether a write request is related to RAID can avoid unnecessary RAID processing, reduce data processing overhead, and ensure that when a write request does involve RAID, the corresponding data redundancy or striping operations can be correctly performed, improving data storage security and performance.

[0064] For example, the data processing module calls the built-in database to query the currently configured RAID partitions and structure. If the target address of the write request is within the range of the RAID configuration, the write request will be marked as a RAID-related operation; otherwise, it will be marked as a non-RAID operation.

[0065] The disk array configuration information is stored in the data processing module, describing configuration data such as RAID partition layout, stripe size, and parity algorithm, which guides the execution of RAID operations. RAID partitions are logical partitions within the disk array; depending on the RAID level, data is stored redundantly or striped in these partitions in different ways.

[0066] Specifically, after determining that the target address of the write request is located in the disk array, the storage controller chip further analyzes the data packet parameters to determine whether the write operation requires RAID algorithm operations such as striping, mirroring, or parity processing. This step ensures the necessity and accuracy of RAID processing.

[0067] For example, the data processing module examines the data packet parameters of the write request, such as data size and data type, and, in conjunction with the disk array configuration information, determines whether the write operation triggers the boundary conditions of the RAID operation. For instance, if the written data exceeds the stripe size of the RAID partition or involves updating redundant data, the system will determine that there is a RAID processing requirement.

[0068] The requirements for data striping include: When the amount of data to be written exceeds the RAID stripe size, the data needs to be split and written in parallel to multiple disks to achieve data striping, improving write performance and data read speed. Mirroring requirements: At mirroring levels such as RAID1, data needs to be written to two or more disks simultaneously to achieve redundant data storage, improving data redundancy and fault recovery capabilities. Parity processing requirements: At parity levels such as RAID5 and RAID6, parity information needs to be calculated and stored during data writing for data redundancy and fault detection and recovery.

[0069] Step S330: If it is determined that the target address of the write request is located in the disk array configured by the data processing module and the write request has a disk array processing requirement, then it is determined that the write request needs to be processed by the disk array.

[0070] Specifically, if the data processing module determines that the target address of the write request is located in the configured disk array and there is a disk array processing requirement, it will decide to perform RAID processing on the write request. Through RAID processing, the storage controller chip can ensure the secure storage of data, provide data redundancy, and improve the performance of data writing and reading through striping technology.

[0071] For example, based on the above decision, the data processing module invokes the RAID data processing module to begin executing the RAID algorithm. This may include data striping, redundant storage, or parity calculations, depending on the specific parameters of the write request and the RAID level configuration.

[0072] In this embodiment, efficient management of write operations is achieved through meticulous write request parsing, precise disk array configuration information querying, and intelligent RAID processing requirement judgment. First, write requests are accurately parsed to ensure accurate data location. Second, by querying disk array configuration information, the association between the write operation and RAID is determined, avoiding unnecessary RAID processing and reducing system overhead. Third, RAID processing decisions are made based on data packet parameters to ensure data redundancy and optimized read / write performance. Finally, after confirming RAID processing requirements, the RAID data processing module is invoked to execute the corresponding algorithm, achieving secure data storage and efficient read / write operations.

[0073] In one embodiment, such as Figure 4 As shown, in step S220, if it is determined that the write request requires disk array processing, the data processing module is invoked to process the write request. This includes steps S400-S420:

[0074] Based on the data processing requirements corresponding to the write request, a corresponding data processing strategy is matched for the write request, where:

[0075] In step S400, if the data processing requirement corresponding to the write request is a striping processing requirement, the data packet carried by the write request is fragmented to obtain multiple data fragments.

[0076] In this process, multiple data shards are written to multiple disks respectively.

[0077] Specifically, if the uplink information processing module detects that a write request requires striping, it divides the data packet into multiple data fragments according to a set size. Each data fragment is written to a different disk to achieve parallel data writing, thereby improving write performance. This strategy is applicable to striping modes such as RAID 0, RAID 5, and RAID 6, avoiding the performance bottleneck of a single disk by distributing the data write load. Striping improves write speed and reduces write latency. Furthermore, writing to multiple disks in parallel also enhances the system's fault tolerance and data availability.

[0078] Striping requires that write operations distribute data evenly across multiple disks to achieve parallel reading and writing. Data fragmentation divides the original data packet into smaller components, with each fragment stored independently on a different disk.

[0079] In step S410, if the data processing requirement corresponding to the write request is a mirroring requirement, the data packet carried by the write request is copied to obtain multiple copies of the data packet.

[0080] The data packet and multiple copies of the data packet are written to multiple disks respectively.

[0081] Specifically, if a write request requires mirroring, the uplink information processing module copies the data packets included in the request multiple times, writing each copy to a different disk to achieve redundant data storage. This operation is applicable to mirroring modes such as RAID 1, ensuring high availability and redundancy of data. Even if one disk fails, the same data can be read from the other disks, avoiding data loss. Mirroring significantly enhances data security and reliability, enabling rapid service recovery even in the event of disk failure and reducing the risk of data loss. Simultaneously, read operations in mirroring mode offer excellent performance because data can be read from any mirrored disk.

[0082] Step S420: If the data processing requirement corresponding to the write request is a verification processing requirement, the data packet carried by the write request is fragmented to obtain multiple data fragments, and the corresponding verification data is calculated for each data fragment.

[0083] Multiple data shards and their corresponding verification data are written to multiple disks respectively.

[0084] Specifically, when a write request involves RAID level parity processing, such as RAID 5 or RAID 6, the RAID data processing module divides the data packet into multiple data fragments and calculates corresponding parity data for each fragment. The data fragments and parity data are stored on multiple disks respectively, achieving data redundancy and striping. Even if one or two disks fail, the original data can be recovered using the remaining data and parity information, ensuring data integrity. Parity processing effectively improves the fault tolerance of the storage system. Even in the face of disk failure, lost data can be recovered using the calculated parity information, ensuring data integrity and availability. Parity data is calculated based on a specific algorithm (such as parity check codes) and is used to detect and correct errors in the stored data.

[0085] In this embodiment, upon receiving a write request, the system intelligently analyzes data processing requirements and matches appropriate data processing strategies (striping, mirroring, or verification processing) accordingly, achieving efficient and secure data storage. Striping significantly improves write speed and system throughput while enhancing system fault tolerance. Mirroring ensures high availability and redundancy of data, reducing the risk of data loss. Verification processing not only enhances data security but also enables rapid data recovery in the event of disk failure, maintaining stable system operation.

[0086] In one embodiment, such as Figure 5 As shown, in step S230, before calling the cache interface module to write the write request to the external cache module when it is determined that the processed write request needs to be cached, the method further includes: steps S500-S530:

[0087] Step S500: Determine whether the data packet carried by the write request meets the caching conditions based on the data characteristics of the data packet carried by the write request.

[0088] Data characteristics include data size, data format, and access frequency. Data size (affects the efficiency of cache usage), data format (affects the difficulty of parsing and processing cached data), and access frequency (reflects the importance of the data; high-frequency data is more suitable for caching to reduce access latency).

[0089] Specifically, write requests are analyzed in depth to determine whether data packets need to be cached. Data characteristics such as size, format, and access frequency are key indicators for assessing caching needs.

[0090] For example, when the system receives a write request, it first extracts the characteristic information of the data packet, including but not limited to the data size, data structure format (such as whether it is a continuous, fixed-length data block), and historical access pattern (i.e., access frequency). This information is input into a decision logic engine, which determines whether the data packet should be cached based on preset caching strategy parameters (such as cache threshold and cache hit rate optimization target).

[0091] Step S510: Obtain the status information of the external cache module, the disk load pressure, and the latency requirements of the target application.

[0092] The status information includes cache capacity and cache hit rate.

[0093] Specifically, in order to further ensure the rationality and effectiveness of caching decisions, the system needs to collect the current status of external caching modules, the workload of backend disks, and the IO latency requirements of specific applications.

[0094] For example, when the system periodically checks or is triggered by an event (such as a new write request), it queries information such as the remaining capacity of the external caching module and the current cache hit rate. Disk load is measured by monitoring metrics such as the length of the disk's read / write request queue and the average wait time. Simultaneously, the system obtains the expected latency requirements from the application layer as a reference for formulating caching strategies.

[0095] The status information includes the usage of external cache (such as cache capacity and cache hit rate), disk load pressure (reflecting the pressure on backend storage, such as queue depth and wait time), and the latency requirements of the target application (referring to the specific requirements of the application for data access speed).

[0096] Step S520: Determine whether the storage system has a caching requirement based on the status information of the external cache module, the disk load pressure, and the latency requirements of the target application.

[0097] Specifically, after collecting all relevant information, the system needs to assess whether enabling caching is necessary under the current conditions. Caching requirements refer to the system's need to improve performance by increasing caching operations under specific conditions. Confirming this requirement depends on the real-time status of the cache and disk, as well as the application's latency tolerance. Ensuring dynamic adjustment of the caching strategy allows caching to intervene in a timely manner, especially in scenarios with high disk pressure and application latency sensitivity, effectively improving the storage system's response speed and reliability.

[0098] For example, the status information of the external caching module, disk load pressure, and the latency requirements of the target application can be integrated. Through a series of logical operations (such as comparisons and weighted averaging), it can be determined whether the storage system is currently facing a high latency risk and whether the cache can effectively mitigate this risk. If the disk load is high, the cache has sufficient space, and the application is latency-sensitive, then the system considers that there is a caching requirement.

[0099] Step S530: If it is determined that the storage system has a caching requirement and the data packet carried by the write request meets the caching conditions, it is determined that the write request needs to be cached.

[0100] Specifically, after comprehensively evaluating caching conditions and requirements, the system decides whether to cache the write request.

[0101] For example, if the system determines that there is a caching requirement and the data in a particular write request is suitable for caching, then the write request will be marked as needing to be cached and subsequently processed according to the caching policy.

[0102] In this embodiment, the storage system can intelligently identify and filter appropriate write requests for caching. This not only reduces the number of times data is directly written to the hard drive, thus reducing disk wear and energy consumption, but also significantly improves data access speed and efficiency, especially in high-load and latency-sensitive application scenarios. Furthermore, the dynamic adjustment of the caching strategy ensures the flexibility and adaptability of the storage system, enabling it to respond optimally to real-time conditions and application requirements, thereby reducing overall system latency.

[0103] In one embodiment, such as Figure 6 As shown, in step S230, if it is determined that the processed write request needs to be cached, the cache interface module is called to write the write request to the external cache module. This includes steps S600-S620:

[0104] Step S600: Convert the data format of the data packet carried in the write request into a standard format supported by the caching protocol.

[0105] Specifically, when a write request is received from the host, the data packet carried by the request needs to undergo data format conversion to ensure compatibility with the protocol of external caching modules (such as NVMe SSDs). This conversion process ensures that data can be stored and managed correctly. Through standardized data format conversion, it ensures that data packets can be efficiently recognized and processed by NVMe SSDs, reducing latency caused by protocol incompatibility and improving data write speed and overall efficiency.

[0106] For example, a protocol conversion module is designed inside the IOC chip. This module has flexible configuration capabilities and can identify and parse write request packets from the PCIe interface. Subsequently, the packets are re-encapsulated and converted into a format conforming to the NVMe protocol to meet the storage requirements of the external cache module.

[0107] The caching protocol can be the NVMe protocol, a high-speed caching communication protocol designed for non-volatile storage media such as solid-state drives. The external caching module can be an NVMe SSD connected to the IOC chip, used to store temporary data to improve read and write performance.

[0108] In step S610, the data packet is transmitted to the external cache module through the cache interface module so as to write the data packet into the external cache module.

[0109] Specifically, data packets already written to the external cache module are marked as dirty data.

[0110] Specifically, after data format conversion, the data packets are transmitted to an external cache module (such as an NVMe SSD) via the cache interface module and written there. To manage the data update status, written data packets are marked as "dirty data," indicating that this data needs to be flushed to the hard drive at the appropriate time to ensure data consistency. Using "dirty data" marking helps to achieve efficient cache management, ensuring that in abnormal situations such as sudden power outages, this data can be flushed to disk first to save data, avoiding data loss, thereby enhancing the reliability and security of data processing.

[0111] For example, a high-speed data channel is established between the cache interface module (NVMe interface module) and the external cache module, and data packets are transmitted and written to the NVMe SSD in a converted standard format. At the same time, the cache management module records which data packets have been written to the cache, and these data packets are marked as "dirty data" to distinguish them from the "clean data" that has been synchronized to the hard drive.

[0112] Dirty data refers to data that has been written to the cache but has not yet been synchronized to the backend storage device; its state is not persistent.

[0113] Step S620: After the data packet is written to the external buffer module, a write success signal is sent back to the host.

[0114] Specifically, once a data packet is successfully written to the external cache module, the IOC chip sends a write success signal to the host, confirming that the write operation is complete. This mechanism is crucial for ensuring communication synchronization between the host and the cache system. Through timely write success signal feedback, the host can quickly ascertain the write operation status, avoiding prolonged waiting times and improving the interaction efficiency and response speed between the host and the storage system.

[0115] For example, the cache management module monitors the data write status. Once it confirms that the data packet has been completely written to the NVMe SSD, it immediately sends an acknowledgment signal to the host through the PCIe interface, indicating that the write operation has been successfully completed.

[0116] In this embodiment, firstly, the data packets undergo protocol conversion to ensure compatibility with the high-speed caching module, significantly reducing processing latency caused by format mismatch and improving write speed. Secondly, data is written to an external cache through the caching interface module and marked as "dirty data." This not only optimizes cache management but also improves the reliability and security of data processing, especially in abnormal situations, prioritizing the protection of unsynchronized data. Finally, a write success signal is sent back to the host, ensuring synchronized and efficient communication between the host and the storage system, reducing host waiting time, and optimizing overall interaction performance.

[0117] In one embodiment, such as Figure 7 As shown, the method further includes steps S700-S710:

[0118] Step S700: When the cache interface module is connected to the external cache module, detect the device information of the external cache module.

[0119] The device information includes capacity and protocol version.

[0120] Specifically, when the cache interface module establishes a connection with an external cache module, such as an NVMe SSD, the system first detects and identifies the device information of the external cache module, including but not limited to its capacity and supported protocol versions. Based on the target business requirements, the system then dynamically adjusts the cache partition configuration and cache space size of the external cache module to optimize the data caching strategy.

[0121] For example, during the initialization phase, the caching interface module sends a probe command to the NVMe SSD to obtain its device information. This can be accomplished using the IDENTIFY Controller command in the NVMe protocol, which returns information such as the controller's firmware version, supported commands, capacity, and serial number. The system determines caching strategy parameters, including the number of cache partitions, the size of each partition, and the cache replacement strategy, based on the target business load, type (e.g., read-heavy or write-heavy, latency-sensitive), and performance requirements. Based on the device information and business needs, the system adjusts the cache partition configuration using the NVMe SSD's Admin command set. For example, the Create Namespace command defines the cache partition size, and the Update Namespace command adjusts the partition configuration to meet business requirements. Based on the adjusted cache partitions and sizes, the system implements content-based caching strategies, such as Least Recently Used (LRU), Least Frequently Used (LFU), or custom strategies, to achieve efficient management and access to cached data.

[0122] Step S710: Adjust the cache partition and cache space size in the external cache module according to the target business requirements.

[0123] Specifically, by detecting the device information of the external caching module, the system can intelligently allocate and adjust cache resources to ensure that the cache space matches business needs. Caching strategies are adjusted according to different business characteristics; for example, adding read cache partitions for read-intensive applications improves data access speed and reduces overall system latency. The dynamic configuration capability of the external caching module allows the system to adjust caching strategies in real time according to changes in business needs, improving the flexibility and responsiveness of the storage system.

[0124] In this embodiment, the storage system can dynamically detect and fully utilize the characteristics of external caching modules (such as NVMe SSDs), intelligently adjusting cache partition configuration and space size according to specific business needs. This flexible cache management strategy not only achieves efficient resource allocation and utilization, but also optimizes data access performance for different business characteristics, significantly reduces system latency, and improves overall efficiency and responsiveness.

[0125] In one embodiment, such as Figure 8 As shown, the method further includes steps S800-S810:

[0126] Step S800: Determine the required target cache space based on the target business requirements.

[0127] Specifically, based on the characteristics and needs of the current business, a reasonable cache space size is determined. In practice, the system can dynamically calculate the required target cache space by analyzing historical data access patterns, predicting future business load, and assessing the current hardware resource status. For example, if the business involves a large number of random read / write operations, a larger cache space may be needed to improve the hit rate. If the business load is expected to increase significantly in a short period, the system may need to temporarily increase the cache space to cope with sudden traffic surges. Determining the target cache space helps the storage system optimize resource allocation and improve response speed under different business scenarios.

[0128] Among them, target business requirements refer to the cache space size and data processing capabilities that the storage system must achieve in the current operating environment to meet specific business loads and performance indicators.

[0129] Step S810: If the cache space of the external cache module currently connected to the cache interface module is less than the target cache space, switch the external cache module connected to the cache interface module until the cache space of the external cache module currently connected to the cache interface module is greater than or equal to the target cache space.

[0130] Specifically, to ensure the cache interface module can connect to an external cache module with sufficient cache space to meet target business needs, the system needs to monitor the cache space size of the currently connected external cache module. When insufficient cache space is detected, the system automatically or manually disconnects the current connection and reconnects to an NVMe SSD with a larger cache space. This process can be completed collaboratively by hardware control logic and software management programs. For example, the hardware side can design a configurable cache switching mechanism, while the software side is responsible for monitoring cache usage and the status of the NVMe SSD, and initiating the cache switching process when necessary. By dynamically adjusting the cache space, the storage system can more flexibly respond to different business needs, avoid performance bottlenecks caused by insufficient cache, and also avoid resource waste. Since the external cache module is externally attached, its size can be dynamically adjusted according to actual business needs, offering high flexibility.

[0131] In this embodiment, by dynamically determining the required target cache space based on the target business needs and automatically switching to an external cache module with larger capacity when the cache space is insufficient, the storage system can effectively cope with changes in business load, improve data processing efficiency, and reduce latency. This mechanism ensures flexible expansion and high utilization of cache resources, while improving the overall performance and reliability of the storage system.

[0132] In one embodiment, such as Figure 9As shown, in step S230, if it is determined that the processed write request needs to be cached, the cache interface module is called to write the write request to the external cache module. This includes steps S900-S930:

[0133] Step S900: If it is determined that the processed write request needs to be cached, the hotness level of the data packet is determined based on the access frequency of the data packet carried by the write request.

[0134] Specifically, when a write request is determined to be stored in the cache, the storage control chip analyzes the access frequency of the data packets in the request over a past period to assess the popularity level of the data packets. The popularity level reflects the degree to which the data packets are accessed and is an important basis for caching strategy decisions.

[0135] Access frequency reflects the number of times a data packet is read or written within a specific time period, and is often used to measure the activity level of data. Popularity level is a classification level based on the access frequency of data packets to guide caching strategies.

[0136] Step S910: Determine the cache space corresponding to the data packet based on the data packet's popularity level.

[0137] The cache space includes a first cache space integrated on the storage controller chip and a second cache space in the external cache module.

[0138] Specifically, based on the popularity level of data packets, the storage controller chip determines which cache layer the data packets should be stored in. The first cache space (L1) can be a high-speed cache integrated on the chip, used to store "hot data." The second cache space (L2) can be an external NVMe SSD cache, suitable for storing "warm data" and "cold data." The first cache space (L1) is a high-speed cache inside the storage controller chip, typically based on SRAM technology, offering fast access speeds. The second cache space (L2) is an external cache module, such as an NVMe SSD; although access speeds are slightly slower, its capacity is much larger than L1 and it can be flexibly adjusted and expanded.

[0139] In step S920, if the cache space corresponding to the data packet is the second cache space, or the remaining space of the first cache space is less than the space required by the data packet, the data packet is written into the external cache module.

[0140] Specifically, if a data packet should be stored in the second buffer space (L2), or if the first buffer space (L1) does not have enough space to accommodate a new data packet, the storage control chip will write the data packet into an external buffer module. This avoids buffer overflow, ensures effective storage of data packets, and maintains the continuity of data processing even under high load.

[0141] For example, the current available capacity of the first cache space is checked. If the current available capacity is insufficient, data packets are written to the second cache space via the NVMe interface module. The external cache module has power-loss protection to ensure data integrity in the event of a power outage.

[0142] In this embodiment, by precisely defining the popularity level of data packets and performing hierarchical caching management accordingly, efficient utilization of cache resources is achieved. Specifically, hot data can be accessed quickly in L1, while warm and cold data are properly stored in L2. The data location is dynamically adjusted according to the availability of cache space, ensuring the high performance, high reliability, and high flexibility of the storage system.

[0143] In one embodiment, such as Figure 10 As shown, in step S230, after determining that the processed write request needs to be cached, and calling the cache interface module to write the write request to the external cache module, the method further includes: steps S1000-S1010:

[0144] In step S1000, if the amount of data cached in the external cache module reaches the refresh threshold, the target data packet marked as dirty data is read back from the external cache module through the cache interface module.

[0145] Specifically, when dirty data accumulates to a certain level in the external caching module, a flushing process is triggered to ensure data persistence and consistency. By setting a reasonable flushing threshold, cache space can be effectively managed, cache overflow can be avoided, and data persistence can be ensured. Timely flushing of dirty data reduces the risk of data loss and enhances system reliability. In addition, through efficient scheduling of the cache interface module, the impact of flushing operations on normal read and write performance is reduced, improving overall data processing efficiency.

[0146] For example, the system monitors the usage of the external caching module and sets a cache flushing threshold. When the amount of cached data reaches or exceeds this threshold, the cache management module initiates the flushing process. Through the cache interface module, data packets marked as dirty data are read from the external cache, ready to be written to the disk array.

[0147] Step S1010: Fetch the target data packet to the disk array.

[0148] Specifically, dirty data target packets read from the external cache module are written to the disk array to achieve persistent data storage.

[0149] For example, the cache management module schedules the NVMe interface module and the Serial Attached SCSI (SAS) / Serial Advanced Technology Attachment (SATA) interface module to ensure that dirty data is correctly written to the disk array after necessary RAID processing. The flushing operation can be asynchronous, meaning that data synchronization occurs in the background without affecting current read / write requests. This ensures data consistency without significantly impacting current read / write performance. Flushing data to the disk array achieves persistent data storage, ensuring no data loss even in the event of an unexpected power outage. Furthermore, the asynchronous flushing mechanism (triggered when the amount of data cached in the external cache module reaches a flushing threshold) reduces write amplification, improves write efficiency, and maintains high cache read performance.

[0150] In this embodiment, by monitoring the amount of dirty data in the external cache module and setting a reasonable flush threshold, the cache space is effectively managed, avoiding cache overflow and ensuring timely data flushing, reducing the risk of data loss and enhancing system reliability. Through the efficient collaboration between the cache interface module and the NVMe interface module, high read and write performance is maintained even during data flushing, achieving a win-win situation of data persistence and high throughput. More importantly, the asynchronous flushing of dirty data, combined with RAID redundancy strategies, not only improves data security but also optimizes the overall latency of the storage system, enhancing user experience and overall system performance.

[0151] In one embodiment, such as Figure 11 As shown, step S1010 involves flushing the target data packet to the disk array. This includes steps S1100-S1110:

[0152] Step S1100: Encapsulate the target data packet into a preset transmission format.

[0153] Specifically, before transmitting data to the disk array, the storage controller chip needs to encapsulate the data packets according to a preset transmission protocol (such as SAS or SATA) to ensure that the data can be correctly identified and processed by the disk array. This guarantees the integrity and understandability of the data.

[0154] For example, in the SAS / SATA protocol conversion module of the storage controller chip, the target data packet first undergoes a series of checks. After confirming that the data is error-free, it is formatted according to the requirements of the SAS or SATA protocol, including adding necessary control information, checksums, etc., to form a transmission data packet that conforms to the disk array receiving standard. By encapsulating the data packet according to a specific preset transmission format, the consistency and compatibility of data between different types of disk arrays are ensured. The checksum information added during the encapsulation process helps detect errors in the data during transmission, ensuring data integrity from the storage controller chip to the disk array. The design of the preset transmission format takes into account the optimal implementation of the protocol, which helps to improve the speed and efficiency of data flushing to the disk array.

[0155] Step S1110: The target data packet, encapsulated in a preset transmission format, is downloaded to the disk array through the hard disk interface module of the storage control chip.

[0156] The hard disk interface module is used to connect to an external disk array.

[0157] Specifically, the packaged data packet is sent to the disk array through the hard disk interface module (such as SAS / SATA interface module) of the storage controller chip. This process ensures the physical transfer of data from the chip to the disk and is a key step in persistent data storage.

[0158] For example, the hard drive interface module, based on the encapsulated data packet format, calls the SAS or SATA physical layer transmission interface to send data to the disk array. During this process, technologies such as signal conditioning and clock synchronization may be used to ensure the stability and accuracy of data during high-speed transmission. The hard drive interface module has multi-channel aggregation capabilities, enabling it to handle concurrent read and write operations on multiple disks simultaneously, thus improving data transmission bandwidth.

[0159] In this embodiment, by encapsulating the target data packet into a preset transmission format and flushing it to the disk array, the storage control chip effectively ensures data compatibility, integrity, and transmission efficiency.

[0160] In one embodiment, such as Figure 12 As shown, the method further includes steps S1200-S1230:

[0161] Step S1200: If the data processing request is a read request, parse the read request to determine the target address information of the data to be read.

[0162] Specifically, when a read request is received from the host, the storage controller chip (IOC chip) first parses the request and extracts the target address information of the data to be read specified in the request. The target address information is key to locating the specific data storage location, and it may include logical block address, file name, file ID, etc., depending on the file system or storage protocol used by the system.

[0163] Step S1210: Determine whether the data to be read is stored in the external cache module based on the target address information.

[0164] Specifically, the IOC chip uses the target address information to query the cache directory to determine whether the data to be read has already been cached in an external cache module (such as an NVMe SSD). By querying the cache directory, it can quickly determine whether the data is in the cache, improving the cache hit rate and thus reducing the number of reads from the disk. A cache hit means that the data can be read quickly from the cache, significantly reducing read latency.

[0165] For example, the cache management unit maintains a cache directory or index to record the data in the cache and its address information. A query is performed on the target address information; if a corresponding entry is found in the cache directory, it means the data has been cached. If not found, it means the data is not in the cache.

[0166] Step S1220: If it is determined that the data to be read is stored in the external cache module, the data is read from the external cache module through the cache interface module and the read data is sent back to the host.

[0167] Specifically, when data resides in an external cache module, the IOC chip reads the data directly from the cache through the cache interface module and quickly transmits the data back to the host via the PCIe interface. Leveraging the high-speed characteristics of NVMe SSDs and the PCIe interface, rapid data reading and transmission are achieved, improving the overall read performance of the storage system. Reading data directly from the cache avoids access to the backend disk, significantly improving the system's response speed to read requests.

[0168] In step S1230, if it is determined that the data to be read is not stored in the external cache module, the data is read from the external disk array through the hard disk interface module and the read data is sent back to the host.

[0169] The hard disk interface module is used to connect to an external disk array.

[0170] Specifically, when the required data is not in the external cache module, the IOC chip reads the data from the backend disk array through the hard disk interface module and then sends it back to the host through the PCIe interface. A cache miss means the data is stored in the disk array, and the data is read from the disk array.

[0171] In this embodiment, the IOC chip can intelligently locate and determine whether data exists in the cache based on the content of the read request. When the data is in the cache, high-speed reading and back transmission are achieved, significantly improving read performance and reducing host waiting time. When the data is not in the cache, data can be efficiently read from the disk array, maintaining data integrity and system compatibility, ensuring the stability and reliability of read operations. This dual addressing mechanism of cache and disk not only improves the efficiency and speed of data access but also enhances the overall system performance, providing users with a smoother and faster data access experience.

[0172] In one embodiment, such as Figure 13 As shown, the method further includes steps S1300-S1330:

[0173] Step S1300: Collect historical data access logs from the storage system.

[0174] Specifically, the storage controller chip continuously monitors and records detailed information about all data accesses in the storage system, including access time, access frequency, and accessed data blocks. Historical data access logs are crucial for understanding data access patterns and predicting future demand.

[0175] For example, the logging function of the storage system can be used to collect access records for each data block periodically or in real time.

[0176] Historical data access logs are records of data access over a past period of time, including read and write operations, stored in the storage system.

[0177] Step S1310: Identify hotspot areas for data access based on historical data access logs.

[0178] Specifically, based on collected historical data access logs, the storage controller chip analyzes and identifies areas with high data access frequency, i.e., hotspot areas. Data in these areas may be frequently accessed due to application characteristics, such as database queries and file system access. Identifying hotspot areas helps to more accurately predict subsequent read requests, preload data into the cache, and reduce waiting time.

[0179] Step S1320: Based on the hotspot area and the pre-trained prediction model, predict the target data for the next read request.

[0180] Specifically, the storage control chip uses a pre-trained predictive model to predict the data that may be needed in the next read request based on currently identified hotspot areas. This prediction helps the system prepare data in advance before the read request arrives. Predicting and loading data into the cache in advance reduces direct access to the disk array and significantly reduces read latency.

[0181] For example, the pre-trained prediction model can be a statistical model based on historical data, such as a time series analysis model, or a machine learning model, such as a deep learning model. Using hotspot area information as input, the model predicts data blocks that are likely to be accessed in the future.

[0182] Step S1330: If the target data is stored in an external disk array, transfer the target data from the external disk array to the external cache module.

[0183] Specifically, if the prediction model determines that the target data is not currently in the external cache module but is expected to be accessed, the storage control chip will read this data from the external disk array and preload it into the cache in preparation for the upcoming read request.

[0184] For example, the hard disk interface module initiates a read operation to the disk array to obtain the predicted target data. The read data is converted according to the protocol to adapt to the storage format of the cache module, and then stored in the cache.

[0185] In this embodiment, by collecting historical data access logs and identifying hotspot areas, and combining this with a pre-trained predictive model to predict the target data for the next read request, the storage control chip can intelligently manage the external cache module. When the predicted data is stored on the external disk array, it is pre-transmitted into the cache. This series of operations significantly reduces read latency, improves cache hit rate, and optimizes the overall performance of the storage system. The introduction of the predictive model makes cache management more intelligent, avoids unnecessary disk access, reduces waste of system resources, and improves read speed and user experience by preloading data into the cache. Intelligent data pre-reading is achieved, which improves the response speed of data reading.

[0186] In one embodiment, such as Figure 14 As shown, a storage control chip 100 is provided, including: a data processing module 10, a cache interface module 20, a host interface module 30, and a control module 40, wherein:

[0187] The data processing module 10 is used to perform disk array processing on the data.

[0188] The data processing module 10, which may be a RAID data processing module 10, is a core component of the storage controller chip, specifically responsible for performing data processing tasks related to Redundant Array of Independent Disks (RAID). It can process write requests and determine whether RAID operations are needed, such as data striping (distributing data across multiple disks) or generating parity data (for data redundancy and recovery). When a write request requires RAID processing, the RAID data processing module 10 processes the data according to the preset RAID level using appropriate algorithms, ensuring that the data is written to the RAID array according to a specific redundancy strategy, thereby providing data protection and improving storage performance. The efficient operation of this module is fundamental to achieving high reliability and high throughput in the storage system.

[0189] The cache interface module 20 is used to connect to the external cache module 50.

[0190] The cache interface module 20, which can be an NVMe (Non-Volatile Memory Express) interface module, is used to establish a high-speed data channel with an external high-speed caching device, such as an NVMe SSD (Solid State Drive). When a write request requires caching, the NVMe interface module utilizes the storage capacity of the external cache module 50 (NVMe SSD) to quickly write the RAID-processed data into the cache, ensuring a fast response for subsequent read operations. This caching mechanism significantly improves the read and write performance of the storage system, especially in high-concurrency scenarios. By preventing frequent disk accesses through caching, it reduces system latency and enhances the user experience.

[0191] The host interface module 30 is used to connect to an external host 60.

[0192] The host interface module 30 can be a PCIe (Peripheral Component Interconnect Express) interface module, serving as a communication bridge between the storage controller chip and the external host 60 (such as a personal computer or server). The PCIe interface module is responsible for receiving and parsing data processing requests from the host, including read and write requests, and sending the processing results back to the host. It supports the high-speed PCIe protocol, enabling efficient data transmission and is a key determinant of the data exchange speed between the storage system and the host.

[0193] The control module 40 is connected to the host interface module 30, the data processing module 10, and the cache interface module 20, respectively, and is used to obtain data processing requests issued by the host. These data processing requests include at least one of the following: a read request and a write request. If the data processing request is a write request, the write request is parsed to determine whether disk array processing is required. If disk array processing is required, the data processing module 10 is invoked to process the write request. If the processed write request needs to be cached, the cache interface module 20 is invoked to write the write request to the external cache module 50.

[0194] The control module 40, which can be an I / O information processing module, is responsible for scheduling and controlling the storage control chip. It is closely connected to the host interface module 30, data processing module 10, and cache interface module 20. It is responsible for parsing data processing requests issued by the host, determining the request type, and intelligently scheduling the work of the data processing module 10 and cache interface module 20 based on the specific content of the request and the system status. For write requests, the I / O information processing module can identify whether RAID processing is required and whether caching is needed after processing. It then executes the write operation by calling the corresponding modules (RAID data processing module 10 and NVMe interface module), ensuring efficient data storage and high-reliability protection.

[0195] In this embodiment, the intelligent parsing of the control module 40 ensures accurate processing of write requests and reasonable allocation of resources. The integration of the data processing module 10 enables striping and redundant calculation of data writes, enhancing data reliability and storage system throughput. The combination of the cache interface module 20 and the external NVMe SSD cache provides high-speed data caching capabilities, reducing disk access, lowering system latency, and improving overall read / write performance. The host interface module 30 ensures high-speed data exchange with the host system, forming the basis for rapid response and execution of data processing requests. Through intelligent parsing, efficient RAID processing and cache utilization, and the integration of the high-speed host interface, comprehensive optimization of write requests is achieved, significantly improving the overall performance of data writing and the storage system, while ensuring data security and reliability.

[0196] In one embodiment, please see [link to embodiment]. Figure 14 The cache interface module 20 can be connected to external cache modules 50 of different capacities.

[0197] The cache interface module 20, as part of the storage control chip, has the ability to connect with external cache modules 50 of different capacities. This feature gives the system high flexibility, enabling it to adapt to NVMe SSD cache devices of various capacities, thereby meeting the caching needs in different scenarios. The design of the cache interface module 20 typically includes circuitry and protocol processing logic for efficient data exchange with cache devices, ensuring stable and high-speed data transmission for both small and large cache units. This module's flexibility is crucial for achieving dynamic cache management and optimizing storage system performance.

[0198] The control module 40 is also used to determine the required target cache space based on the target business requirements. If the cache space of the external cache module 50 currently connected to the cache interface module 20 is less than the target cache space, the external cache module 50 connected to the cache interface module 20 is switched until the cache space of the external cache module 50 currently connected to the cache interface module 20 is greater than or equal to the target cache space.

[0199] Specifically, in addition to parsing and scheduling data processing requests, the control module 40 also has the function of determining the required cache space size based on the target business needs. The control module 40 can monitor the current business load of the system in real time, analyze data access patterns, and calculate the appropriate cache space requirements accordingly. When the system detects that the capacity of the currently connected external cache module 50 is insufficient, the control module 40 will automatically initiate a cache switching process to find and connect to a larger external cache module 50, ensuring that the system can provide sufficient cache space to optimize data processing performance. This process demonstrates the control module 40's highly intelligent management and dynamic adjustment capabilities for system resources, which is of great significance for improving the overall efficiency and response speed of the storage system.

[0200] In this embodiment, the control module 40 dynamically determines the cache space size based on the target business requirements, ensuring that the cache capacity closely matches the actual business needs and avoiding resource waste or insufficiency. When the current cache device capacity is insufficient, the cache interface module 20 can automatically switch to an external cache module 50 with sufficient capacity, ensuring that the system maintains optimal cache performance under any business scenario. By dynamically adjusting the cache capacity, the system can process data read and write requests more efficiently, especially in high-concurrency and large-scale data processing scenarios, improving the cache hit rate, reducing direct access to the disk array, thereby reducing latency and improving overall system performance. In summary, the control module 40 analyzes business requirements in real time to determine the required cache space size. If the current cache module's capacity is insufficient to meet this requirement, the control module 40 will trigger the cache interface module 20 to switch cache devices, searching for and connecting to an external cache module 50 with sufficient capacity to cover the target cache space. This automatic cache capacity adjustment mechanism based on business requirements ensures that the storage system receives optimal cache support when processing data requests, thereby improving the system's response speed and data processing efficiency.

[0201] In one embodiment, such as Figure 15 As shown, the storage control chip also includes an internal cache module 70.

[0202] The control module 40 is also used to determine the popularity level of a data packet based on the access frequency of the data packet carried by the write request when it is determined that the processed write request needs to be cached. The corresponding cache space for the data packet is determined based on its popularity level. This cache space includes a first cache space in the internal cache module 70 integrated on the storage control chip and a second cache space in the external cache module 50. If the cache space corresponding to the data packet is the second cache space, or if the remaining space in the first cache space is less than the space required by the data packet, the data packet is written to the external cache module 50.

[0203] The internal cache module 70 refers to a cache unit integrated within the memory controller chip, typically existing as on-chip SRAM, used to store frequently accessed or urgently needed data. The internal cache module 70 is characterized by high access speed and low latency, but its capacity is relatively small due to manufacturing process and chip area limitations. Together with the external cache module 50, it forms a multi-level cache system, jointly improving the read / write efficiency and response speed of the memory system.

[0204] Specifically, after processing the write request, the control module 40 not only determines whether the data needs to be cached, but also further determines the data packet's popularity level based on the access frequency of the data packet. This process is based on the control module 40's analysis of historical access logs, which can identify which data packets are frequently accessed hot data, and thus prioritize storing these data packets in the faster first cache space (internal cache module 70). If the data packet's access frequency is low, or the first cache space is insufficient, the control module 40 will store the data packet in the second cache space (external cache module 50), usually an NVMe SSD, to fully utilize its larger storage capacity. Through intelligent judgment and hierarchical storage strategies, the control module 40 optimizes the allocation of data between the inner and outer cache layers, ensuring the effective utilization of the high-speed cache while also taking into account the long-term storage needs of the data.

[0205] In this embodiment, the control module 40 analyzes the access frequency of data packets and classifies them into different popularity levels. This allows the storage system to adjust its caching strategy according to the importance of the data, improving the utilization efficiency of cache resources. Based on the popularity level of the data packets, the control module 40 intelligently selects either the first cache space of the internal cache module 70 or the second cache space of the external cache module 50. For frequently accessed data packets, they are preferentially stored in the first cache space to achieve ultra-low latency data access. For low-frequency or large-capacity data packets, they are stored in the second cache space, balancing the utilization of the high-speed cache and the demand for storage capacity. When the remaining capacity of the first cache space is insufficient, the control module 40 can store the data packets in the second cache space. This not only solves the problem of limited capacity in the internal cache module 70 but also ensures that data packets can be cached efficiently, avoiding direct writing of data to the disk array due to insufficient cache space, reducing unnecessary disk access, and further reducing overall latency.

[0206] In one embodiment, please see [link to embodiment]. Figure 15 As shown, the storage control chip also includes: a protocol conversion module 80 and a hard disk interface module 90, wherein:

[0207] The protocol conversion module 80 is used to encapsulate the data packets carried by the write request into a preset transmission format.

[0208] The protocol conversion module 80 is a key component of the storage controller chip, responsible for converting data packets of various formats into a standard format suitable for a specific transmission protocol, facilitating data transmission between different layers. In this technical solution, the protocol conversion module 80 converts the data packets carried by write requests received from the host into a transmission format compatible with the downstream disk array 91. This process is crucial because different storage devices (such as SATA HDDs, SAS SSDs, or NVMe SSDs) typically employ their own communication protocols, such as SATA, SAS, or NVMe. By uniformly converting upstream data packets into a preset transmission format, the protocol conversion module 80 ensures that data can be successfully delivered to the disk array 91, while also maintaining compatibility with multiple storage devices, enhancing the interconnectivity and versatility of the storage system.

[0209] The hard disk interface module 90 is used to connect to an external disk array 91.

[0210] The hard disk interface module 90 serves as the direct communication channel between the storage control chip and the external disk array 91, handling the transmission and reception of data packets. It supports multiple disk interface standards, such as SAS or SATA, and can establish a stable data transmission link with the disk array 91. When the control module 40 decides to push a data packet to the disk array 91, the hard disk interface module 90 receives the data packet encapsulated in a preset transmission format and writes it into the disk array 91 using the corresponding protocol (SAS, SATA, etc.), ensuring secure data storage and persistence. The design of the hard disk interface module 90 needs to consider data transmission rate and stability, as well as compatibility with various types of disks, to meet the needs of different storage application scenarios.

[0211] The control module 40 is used to download data packets encapsulated in a preset transmission format to the disk array 91 via the hard disk interface module 90.

[0212] Specifically, during the process of data being flushed to disk array 91, control module 40 instructs protocol conversion module 80 to appropriately convert the data packets to conform to a preset transmission format. After conversion, control module 40 further sends the data packets to downstream disk array 91 for storage via hard disk interface module 90. This process ensures the consistency and efficiency of data transmission from the host to the storage device, and also reflects the refined management of the data processing flow by control module 40.

[0213] In this embodiment, the protocol conversion module 80 ensures that data packets of different formats from the upstream host can be converted into a preset transmission format acceptable to the downstream disk array 91, eliminating obstacles caused by protocol incompatibility and improving the smoothness and compatibility of data transmission. The linkage between the control module 40 and the hard disk interface module 90 enables data to be flushed to the disk array 91 in the optimal format and along the fastest path, reducing data transmission overhead and latency and improving the write performance of the storage system. Through the hard disk interface module 90, which supports multiple disk interface standards, the storage control chip can establish connections with different types of disk arrays 91 (such as SATA hard disk arrays, SAS SSD arrays, etc.), enhancing the system's scalability and the diversity of application scenarios. Writing data to the disk array 91 through the hard disk interface module 90 ensures secure data storage and persistence even in unexpected situations such as power outages, enhancing the reliability of the storage system.

[0214] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method.

[0215] Embodiments of this application also provide a data storage device. Figure 16 This is a structural block diagram of a data storage apparatus according to an embodiment of the present application, the apparatus comprising:

[0216] The request acquisition module 1601 is used to acquire data processing requests issued by the host, wherein the data processing requests include at least one of the following: read request and write request.

[0217] The request parsing module 1602 is used to parse the write request to determine whether the write request requires disk array processing when the data processing request is a write request.

[0218] Module 1603 is invoked to process the write request by calling the data processing module when it is determined that the write request requires disk array processing.

[0219] The write module 1604 is used to call the cache interface module to write the write request to the external cache module when it is determined that the processed write request needs to be cached.

[0220] In one exemplary embodiment, the apparatus is further configured to: parse the write request to determine the target address and data packet parameters of the write request; query the disk array configuration information of the data processing module to determine whether the target address of the write request is located in the disk array configured by the data processing module; and determine whether the write request has a disk array processing requirement based on the data packet parameters, wherein the disk array processing requirement includes at least one of the following: striping processing requirement, mirroring processing requirement, and verification processing requirement. If it is determined that the target address of the write request is located in the disk array configured by the data processing module and the write request has a disk array processing requirement, it is determined that the write request needs to undergo disk array processing.

[0221] In one exemplary embodiment, the apparatus is further configured to: match a corresponding data processing strategy to the write request based on the data processing requirements corresponding to the write request, wherein: when the data processing requirement corresponding to the write request is a striping processing requirement, the data packets carried by the write request are fragmented to obtain multiple data fragments, wherein the multiple data fragments are written to multiple disks respectively. When the data processing requirement corresponding to the write request is a mirroring processing requirement, the data packets carried by the write request are copied to obtain multiple copies of the data packets, wherein the data packets and the copies of the multiple data packets are written to multiple disks respectively. When the data processing requirement corresponding to the write request is a verification processing requirement, the data packets carried by the write request are fragmented to obtain multiple data fragments, and corresponding verification data is calculated for each data fragment, wherein the multiple data fragments and the corresponding verification data are written to multiple disks respectively.

[0222] In one exemplary embodiment, the apparatus is further configured to: determine whether the data packet carried by the write request meets caching conditions based on the data characteristics of the data packet carried by the write request, wherein the data characteristics include data size, data format, and access frequency; acquire status information of an external caching module, disk load pressure, and latency requirements of the target application, wherein the status information includes cache capacity and cache hit rate; determine whether the storage system has a caching requirement based on the status information of the external caching module, disk load pressure, and latency requirements of the target application; and, if it is determined that the storage system has a caching requirement and the data packet carried by the write request meets the caching conditions, determine that the write request needs to be cached.

[0223] In one exemplary embodiment, the apparatus is further configured to: convert the data format of the data packet carried in the write request into a standard format supported by the caching protocol; transmit the data packet to an external caching module through the caching interface module to write the data packet into the external caching module, wherein the data packet already written into the external caching module is marked as dirty data; and, upon completion of writing the data packet into the external caching module, send a write success signal to the host.

[0224] In one exemplary embodiment, the apparatus is further configured to: detect device information of the external cache module when the cache interface module is connected to the external cache module, wherein the device information includes capacity and protocol version; and adjust the cache partition and cache space size in the external cache module according to target business requirements.

[0225] In one exemplary embodiment, the apparatus is further configured to: determine the required target cache space based on target business requirements. If the cache space of the external cache module currently connected to the cache interface module is less than the target cache space, switch the external cache module connected to the cache interface module until the cache space of the external cache module currently connected to the cache interface module is greater than or equal to the target cache space.

[0226] In one exemplary embodiment, the apparatus is further configured to: determine the popularity level of a data packet based on the access frequency of the data packet carried by the write request when it is determined that the processed write request needs to be cached; determine the corresponding cache space for the data packet based on the popularity level of the data packet, wherein the cache space includes a first cache space integrated on the storage controller chip and a second cache space in an external cache module; and write the data packet into the external cache module when the cache space corresponding to the data packet is the second cache space, or when the remaining space of the first cache space is less than the space required by the data packet.

[0227] In one exemplary embodiment, the apparatus is further configured to: when the amount of data cached in the external cache module reaches a flush threshold, read back the target data packet marked as dirty data from the external cache module via the cache interface module; and flush the target data packet to the disk array.

[0228] In one exemplary embodiment, the apparatus is further configured to: encapsulate the target data packet into a preset transmission format. The target data packet, encapsulated in the preset transmission format, is then flushed to the disk array via the hard disk interface module of the storage control chip, wherein the hard disk interface module is used to connect to an external disk array.

[0229] In one exemplary embodiment, the apparatus is further configured to: parse the read request to determine target address information of the data to be read when the data processing request is a read request; determine whether the data to be read is stored in an external cache module based on the target address information; if it is determined that the data to be read is stored in an external cache module, read the data from the external cache module through a cache interface module and send the read data back to the host; if it is determined that the data to be read is not stored in an external cache module, read the data from an external disk array through a hard disk interface module and send the read data back to the host, wherein the hard disk interface module is used to connect to the external disk array.

[0230] In one exemplary embodiment, the apparatus is further configured to: collect historical data access logs from the storage system; identify hotspot areas of data access based on the historical data access logs; predict the target data for the next read request based on the hotspot areas and a pre-trained prediction model; and, if the target data is stored in an external disk array, transfer the target data from the external disk array to an external cache module.

[0231] For a description of the features in the embodiment corresponding to the data storage device, please refer to the relevant description in the embodiment corresponding to the data storage method, which will not be repeated here.

[0232] Embodiments of this application also provide an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above-described data storage method embodiments.

[0233] Embodiments of this application also provide a computer-readable storage medium storing a computer program, wherein the computer program is configured to execute the steps in any of the above-described data storage method embodiments when it is run.

[0234] In one exemplary embodiment, the aforementioned computer-readable storage medium may include, but is not limited to, various media capable of storing computer programs, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard disk, magnetic disk, or optical disk.

[0235] Embodiments of this application also provide a computer program product, which includes a computer program that, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0236] Embodiments of this application also provide another computer program product, including a non-volatile computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the above-described data storage method embodiments.

[0237] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0238] The foregoing has provided a detailed description of a data storage method, apparatus, electronic device, computer-readable storage medium, and computer program product provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are merely for the purpose of helping to understand the method and core ideas of this application. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from its principles, and these improvements and modifications also fall within the protection scope of the claims of this application.

Claims

1. A method for data storage, characterized in that, The method, applied to a storage control chip, includes a data processing module and a cache interface module, the cache interface module being used to connect to an external cache module, and comprises: Obtain a data processing request issued by the host, wherein the data processing request includes at least one of the following: a read request and a write request; If the data processing request is a write request, the write request is parsed to determine the target address and data packet parameters of the write request; the disk array configuration information of the data processing module is queried to determine whether the target address of the write request is located in the disk array configured by the data processing module; and, based on the data packet parameters, it is determined whether the write request has a disk array processing requirement, wherein the disk array processing requirement includes at least one of the following: striping processing requirement, mirroring processing requirement, and verification processing requirement; if it is determined that the target address of the write request is located in the disk array configured by the data processing module and the write request has a disk array processing requirement, it is determined that the write request needs to undergo disk array processing. If it is determined that the write request requires disk array processing, the data processing module is invoked to process the write request; Based on the data characteristics of the data packet carried by the write request, determine whether the data packet carried by the write request meets the caching conditions, wherein the data characteristics include data size, data format, and access frequency; Obtain the status information of the external caching module, disk load pressure, and latency requirements of the target application, wherein the status information includes cache capacity and cache hit rate; Based on the status information of the external cache module, disk load pressure, and latency requirements of the target application, determine whether the storage system has a caching requirement; If it is determined that the storage system has a caching requirement and the data packet carried by the write request meets the caching conditions, it is determined that the write request needs to be cached; If it is determined that the processed write request needs to be cached, the cache interface module is invoked to write the write request to the external cache module.

2. The data storage method according to claim 1, characterized in that, The step of invoking the data processing module to process the write request includes: Based on the data processing requirements corresponding to the write request, a corresponding data processing strategy is matched for the write request, wherein: When the data processing requirement corresponding to the write request is the striping processing requirement, the data packet carried by the write request is fragmented to obtain multiple data fragments, wherein the multiple data fragments are written to multiple disks respectively. When the data processing requirement corresponding to the write request is the mirror processing requirement, the data packet carried by the write request is copied to obtain multiple copies of the data packet, wherein the data packet and the multiple copies of the data packet are written to multiple disks respectively. When the data processing requirement corresponding to the write request is the verification processing requirement, the data packet carried by the write request is fragmented to obtain multiple data fragments. For each data fragment, corresponding verification data is calculated. The multiple data fragments and the corresponding verification data are written to multiple disks respectively.

3. The data storage method according to claim 1, characterized in that, The step of calling the cache interface module to write the write request to the external cache module includes: The data format of the data packet carried in the write request is converted into a standard format supported by the caching protocol; The data packet is transmitted to the external cache module through the cache interface module so as to write the data packet into the external cache module, wherein the data packet already written into the external cache module is marked as dirty data; Once the data packet has been written to the external cache module, a write success signal is sent back to the host.

4. The data storage method according to claim 3, characterized in that, The method further includes: When the cache interface module is connected to an external cache module, the device information of the external cache module is detected, wherein the device information includes capacity and protocol version; Adjust the size of the cache partition and cache space in the external cache module according to the target business requirements.

5. The data storage method according to claim 4, characterized in that, The method further includes: Determine the required target cache space based on the target business requirements; If the cache space of the external cache module currently connected to the cache interface module is less than the target cache space, switch the external cache module connected to the cache interface module until the cache space of the external cache module currently connected to the cache interface module is greater than or equal to the target cache space.

6. The data storage method according to claim 1 or 2, characterized in that, When it is determined that the processed write request needs to be cached, the step of calling the cache interface module to write the write request into the external cache module includes: If it is determined that the processed write request needs to be cached, the popularity level of the data packet is determined based on the access frequency of the data packet carried by the write request; The cache space corresponding to the data packet is determined according to the heat level of the data packet, wherein the cache space includes a first cache space integrated on the storage control chip and a second cache space in the external cache module; If the cache space corresponding to the data packet is the second cache space, or if the remaining space of the first cache space is less than the required space of the data packet, the data packet is written into the external cache module.

7. The data storage method according to claim 3, characterized in that, If it is determined that the processed write request needs to be cached, after calling the cache interface module to write the write request into the external cache module, the method further includes: When the amount of data cached in the external cache module reaches the flush threshold, the target data packet marked as dirty data is read back from the external cache module through the cache interface module. The target data packet is flushed to the disk array.

8. The data storage method according to claim 7, characterized in that, The step of flushing the target data packet to the disk array includes: The target data packet is encapsulated into a preset transmission format; The target data packet, encapsulated in the preset transmission format, is flushed to the disk array via the hard disk interface module of the storage control chip. The hard disk interface module is used to connect to an external disk array.

9. The data storage method according to claim 1 or 2, characterized in that, The method further includes: When the data processing request is the read request, the read request is parsed to determine the target address information of the data to be read; Based on the target address information, determine whether the data to be read is stored in the external cache module; If it is determined that the data to be read is stored in the external cache module, the data is read from the external cache module through the cache interface module and the read data is sent back to the host. If it is determined that the data to be read is not stored in the external cache module, the data is read from the external disk array through the hard disk interface module and the read data is sent back to the host. The hard disk interface module is used to connect to the external disk array.

10. The data storage method according to claim 9, characterized in that, The method further includes: Collect historical data access logs from the storage system; Identify hotspots for data access based on the historical data access logs; Based on the hotspot areas and the pre-trained prediction model, predict the target data for the next read request; When the target data is stored in the external disk array, the target data is transferred from the external disk array to the external cache module.

11. A storage control chip, characterized in that, include: The data processing module is used for disk array processing of data; A cache interface module, which is used to connect to an external cache module; A host interface module, which is used to connect to an external host; A control module, connected to the host interface module, the data processing module, and the cache interface module, is used to acquire data processing requests issued by the host. The data processing requests include at least one of the following: a read request and a write request. If the data processing request is a write request, the control module parses the write request to determine the target address and data packet parameters of the write request. It queries the disk array configuration information of the data processing module to determine whether the target address of the write request is located in the disk array configured by the data processing module. Furthermore, it determines whether the write request has a disk array processing requirement based on the data packet parameters. The disk array processing requirement includes at least one of the following: striping processing requirement, mirroring processing requirement, and verification processing requirement. If it is determined that the target address of the write request is located in the disk array configured by the data processing module and the write request has a disk array processing requirement, it determines that the write request needs to perform disk array processing. Disk array processing; if it is determined that the write request requires disk array processing, the data processing module is invoked to process the write request; based on the data characteristics of the data packet carried by the write request, it is determined whether the data packet carried by the write request meets the caching conditions, wherein the data characteristics include data size, data format, and access frequency; the status information of the external caching module, disk load pressure, and latency requirements of the target application are obtained, wherein the status information includes cache capacity and cache hit rate; based on the status information of the external caching module, disk load pressure, and latency requirements of the target application, it is determined whether the storage system has a caching requirement; if it is determined that the storage system has a caching requirement and the data packet carried by the write request meets the caching conditions, it is determined that the write request needs to be cached; if it is determined that the processed write request needs to be cached, the caching interface module is invoked to write the write request into the external caching module.

12. The storage control chip according to claim 11, characterized in that, The cache interface module can be connected to external cache modules of different capacities; The control module is also used to determine the required target cache space according to the target business requirements; if the cache space of the external cache module currently connected to the cache interface module is less than the target cache space, the external cache module connected to the cache interface module is switched until the cache space of the external cache module currently connected to the cache interface module is greater than or equal to the target cache space.

13. The storage control chip according to claim 11, characterized in that, The storage control chip also includes: an internal cache module; The control module is further configured to, when it is determined that the processed write request needs to be cached, determine the popularity level of the data packet carried by the write request based on the access frequency of the data packet; determine the cache space corresponding to the data packet based on the popularity level of the data packet, wherein the cache space includes a first cache space of an internal cache module integrated on the storage control chip and a second cache space in the external cache module; and write the data packet into the external cache module when the cache space corresponding to the data packet is the second cache space, or when the remaining space of the first cache space is less than the space required by the data packet.

14. The storage control chip according to claim 11, characterized in that, The storage control chip also includes: A protocol conversion module is used to encapsulate the data packet carried by the write request into a preset transmission format; A hard disk interface module, which is used to connect to an external disk array; The control module is used to download data packets encapsulated in the preset transmission format to the disk array through the hard disk interface module.

15. A data storage device, characterized in that, include: The request acquisition module is used to acquire data processing requests issued by the host, wherein the data processing requests include at least one of the following: a read request and a write request; The request parsing module is configured to, when the data processing request is a write request, parse the write request to determine the target address and data packet parameters of the write request; query the disk array configuration information of the data processing module to determine whether the target address of the write request is located in the disk array configured by the data processing module; and determine whether the write request has a disk array processing requirement based on the data packet parameters, wherein the disk array processing requirement includes at least one of the following: striping processing requirement, mirroring processing requirement, and verification processing requirement; and determine that the write request needs to undergo disk array processing if it is determined that the target address of the write request is located in the disk array configured by the data processing module and the write request has a disk array processing requirement. The calling module is used to call the data processing module to process the write request when it is determined that the write request needs to be processed by the disk array. The data storage device is further configured to: determine whether the data packet carried by the write request meets caching conditions based on the data characteristics of the data packet carried by the write request, wherein the data characteristics include data size, data format, and access frequency; acquire the status information of the external caching module, disk load pressure, and latency requirements of the target application, wherein the status information includes cache capacity and cache hit rate; determine whether the storage system has caching requirements based on the status information of the external caching module, disk load pressure, and latency requirements of the target application; and determine that the write request needs to be cached if it is determined that the storage system has caching requirements and the data packet carried by the write request meets the caching conditions. The write module is used to call the cache interface module to write the write request into the external cache module when it is determined that the processed write request needs to be cached.

16. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the steps of the method as described in any one of claims 1 to 10.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 10.

18. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 10.

Images