Data transmission method and device, electronic equipment and storage medium

Abstract

The invention discloses a data transmission method and device, electronic equipment and a storage medium, and relates to the technical field of data processing. By dynamically analyzing the characteristic parameters of the to-be-transmitted data, the adaptive transmission configuration is automatically optimized and generated based on the parameter attributes, and the data is efficiently and reliably transmitted to the target storage position by using a multi-channel parallel transmission mechanism. The technical problems of low resource utilization rate and insufficient transmission efficiency caused by fixed parameter configuration in traditional data transmission are solved, and the technical effect of improving the data transmission rate and accuracy is achieved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data transmission method, device, electronic device and storage medium. Background Art

[0002] With the rapid development of cloud computing, the Internet of Things and big data technologies, the efficient transmission of massive data has become a core demand in the field of information technology.

[0003] In related data transmission technologies, single-channel architectures and static parameter configurations are prevalent, making it difficult to adapt to diverse data characteristics and high-concurrency scenario requirements. Traditional methods rely on manual setting of transmission parameters, lacking dynamic analysis of data characteristics and intelligent reuse of historical experience, resulting in low resource utilization and insufficient parallelism. Especially when processing large-scale read and write tasks, fixed strategies cannot automatically optimize transmission paths based on data size, type, or real-time load, which can easily lead to bandwidth waste and efficiency bottlenecks. In addition, the serial transmission mode of a single channel limits the potential of the hardware, making it difficult to effectively increase throughput through multi-channel collaboration. Summary of the Invention

[0004] The present application provides a data transmission method, device, electronic device and storage medium to at least solve the problems of rigid resource configuration and low transmission efficiency under traditional single-channel architecture and static strategy in related technologies.

[0005] The present application provides a data transmission method, including: obtaining read and write data to be transmitted; parsing the read and write data to determine parameter information of the read and write data; optimizing the parameter information based on attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data; and transmitting the read and write data to a target storage location through multiple parallel channels according to the target data transmission parameters.

[0006] In an optional embodiment, the read and write data are parsed to determine parameter information of the read and write data, including: parsing the read and write data to obtain metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters and transmission parameters corresponding to the read and write data; parsing the data command type configuration parameters to determine the target command of the read and write data; obtaining associated parameters corresponding to the target command, and generating an identifier between the target command and the associated parameters; wherein the parameter information includes at least one of the metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, transmission parameters and identifiers.

[0007] In an optional embodiment, the parameter information is optimized based on the attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data, including: determining the data transmission priority of the read and write data based on the detection results of the metadata contained in the read and write data; optimizing the parameter information according to the data transmission priority to generate target data transmission parameters corresponding to the read and write data.

[0008] In an optional embodiment, based on the detection results of metadata contained in the read and write data, the data transmission priority of the read and write data is determined, including: when the read and write data includes metadata, determining the first transmission priority of the read and write data, the first transmission priority is used to characterize that the data transmission accuracy of the read and write data is better than the data transmission rate; when the read and write data does not include metadata, determining the second transmission priority of the read and write data, the second transmission priority is used to characterize that the data transmission rate of the read and write data is better than the data transmission accuracy; wherein the data transmission priority includes the first transmission priority and the second transmission priority.

[0009] In an optional embodiment, the parameter information includes a base address corresponding to the read and write data; according to the target data transmission parameters, the read and write data are transmitted to the target storage location through multiple parallel channels, including: parsing the base address, determining the number of random access memories corresponding to the read and write data, and determining the number of random access memories as the number of parallel channels; according to the target data transmission parameters, the read and write data are written to multiple target random access memories through multiple parallel channels; wherein the multiple target random access memories are used to represent the target storage location.

[0010] In an optional embodiment, the target random access memory includes multiple memory banks; according to the target data transmission parameters, the read and write data are written to the multiple target random access memories through multiple parallel channels, including: obtaining a first order of the target random access memories and a second order of writing the read and write data to the memory banks; through multiple parallel channels, the read and write data are distributed to the multiple target random access memories according to the first order; for any target random access memory, the read and write data are written to the memory banks in sequence according to the second order.

[0011] In an optional embodiment, the transmission rate and transmission error rate of read and write data transmitted to a target storage location through multiple parallel channels are detected; the transmission rate, transmission error rate and identifier are associated to generate associated data; when the associated data accumulates to a preset threshold, the associated data is used to optimize the parameter information to obtain optimized parameter information.

[0012] The present application also provides a data transmission device, including: an acquisition module for acquiring read and write data to be transmitted; a parsing module for parsing the read and write data and determining parameter information of the read and write data; a first optimization module for optimizing the parameter information based on the attribute characteristics of the parameter information and generating target data transmission parameters corresponding to the read and write data; a transmission module for transmitting the read and write data to a target storage location through multiple parallel channels according to the target data transmission parameters.

[0013] The present application also provides an electronic device, comprising: a memory for storing a computer program; and a processor for implementing the steps of any of the above-mentioned data transmission methods when executing the computer program.

[0014] The present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned data transmission methods are implemented.

[0015] The present application also provides a computer program product, including a computer program, which implements the steps of any of the above-mentioned data transmission methods when executed by a processor.

[0016] Through this application, by dynamically generating optimized target data transmission parameters based on attribute characteristics of parameter information and combining it with a multi-channel parallel transmission mechanism, it is possible to solve the technical problems of low resource utilization and insufficient transmission efficiency caused by fixed parameter configuration in traditional data transmission, thereby achieving the technical effect of improving data transmission rate and accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 A flowchart of a data transmission method provided in an embodiment of the present application;

[0019] Figure 2 A flowchart of another data transmission method provided in an embodiment of the present application;

[0020] Figure 3 A flowchart of another data transmission method provided in an embodiment of the present application;

[0021] Figure 4 A schematic diagram of the correspondence between the DDR4 transmission address and the AXI bus obtained through training provided in an embodiment of the present application;

[0022] Figure 5 A structural block diagram of a data transmission device provided in an embodiment of the present application;

[0023] Figure 6 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The following will be combined with the accompanying drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] It should be noted that, in the description of this application, the terms "comprises," "includes," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. The terms "first," "second," etc., in this application are used to distinguish similar objects, and are not used to describe a particular order or sequence.

[0026] In order to enable those skilled in the art to better understand the present application, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0027] Driven by high-speed interconnects like Peripheral Component Interconnect Express (PCIe) Gen4 / Gen5 and big data scenarios, solid-state drives (SSDs), as core storage devices, face severe performance challenges. Current DRAM-based cache solutions utilize a single-controller transmission mode, which presents significant bottlenecks. Firstly, the single-controller architecture lacks parallelism, resulting in inefficient data transmission between the Non-Volatile Memory Express (NVMe) module and the cache, and between the cache and the storage unit. Secondly, transmission parameters (such as burst length and burst byte size) must be manually and statically configured, making it impossible to dynamically optimize the speed-error-rate balance based on historical transmission experience. This makes it difficult to adapt to high-concurrency, high-throughput scenarios such as large-scale model training and data centers. Furthermore, while the bandwidth of on-chip static random access memory (SRAM) far exceeds that of off-chip double data rate synchronous dynamic random access memory (DDR) by more than ten times, its megabyte-level capacity falls far short of DDR's gigabyte-level capacity, making it unable to independently handle bidirectional data caching tasks under high concurrency. This ultimately restricts the matching of SSD overall performance with intelligent transmission requirements.

[0028] The defects of related technical solutions are mainly reflected in the rigidity of the architecture and the lack of intelligence. First, the single-controller mode cannot fully utilize the parallelism of the hardware, resulting in limited transmission rate, especially in multi-tasking scenarios where performance drops sharply. Secondly, there is a lack of an adaptive parameter adjustment mechanism, and the transmission parameters are configured based on manual experience. It is impossible to optimize for specific data characteristics (such as size, priority), and it is difficult to extract the optimal strategy through historical transmission data, resulting in resource waste and efficiency loss. In addition, the inherent contradictions of storage media (such as the high bandwidth and low capacity of SRAM and the large capacity and low efficiency of DDR) further aggravate the system bottleneck. These problems together make it difficult for SSDs to meet the urgent needs of the intelligent era for high-speed, low-error, and highly adaptable transmission. Breakthroughs are urgently needed through parallel architecture design and intelligent control algorithms.

[0029] An embodiment of the present application provides a data transmission method, which, combined with the execution process of the data transmission method, aims to solve the problem that the current SSD master control cannot automatically adjust the transmission parameters according to the relevant transmission scenarios, resulting in the transmission efficiency not being able to achieve the optimal level.

[0030] According to an embodiment of the present invention, an embodiment of a data transmission method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0031] In this embodiment, a data transmission method is provided, which can be used in a computer device, such as a controller inside an NVMe module. Figure 1 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 1 As shown, the process includes the following steps:

[0032] Step S101: Acquire read and write data to be transmitted.

[0033] Read and write data refers to data sent from the host to the NVMe module for storage (write operation) or read from the storage unit to the host (read operation), which may include user files, metadata, command information, etc. Specifically, the read and write data to be transmitted is obtained through the interaction process between the host and the NVMe module. When the host initiates a read or write command, the command is submitted to the submission queue (SQ) and notified to the NVMe module through the "doorbell" mechanism. The controller inside NVMe actively obtains the command from the SQ, parses the command type (read or write) and the target storage address. For write operations, the NVMe module reads data directly from the host memory through direct memory access (DMA); for read operations, the data is extracted from the storage unit (such as DDR or flash memory).

[0034] Step S102: parse the read / write data to determine parameter information of the read / write data.

[0035] Parameter information refers to the configuration parameters related to data transmission parsed from the read and write data, such as burst length, data transmission size, DDR base address, and metadata existence. Specifically, key parameter information is extracted from the read and write commands issued by the host. First, the controller must obtain command information from the read and write commands, such as distinguishing the command type (read command / write command), whether to enable automatic parameter adjustment, etc. Secondly, it is necessary to obtain relevant configuration parameters for data transmission, such as data attributes (such as determining whether metadata is included, whether data is stored continuously, etc.), transmission parameters (such as extracting burst length from the command, the amount of data transmitted in a single time, etc.), and other information. These parsing results constitute the parameter information of the read and write data.

[0036] Step S103 : Optimizing the parameter information based on the attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data.

[0037] Attribute characteristics refer to key features extracted from parameter information and used to optimize transmission strategies. Target data transmission parameters refer to the optimized dynamic transmission configuration. Specifically, dynamic optimization is performed based on the parameter information and its attribute characteristics. For example, if the data contains metadata, transmission accuracy is prioritized (for example, reducing the error rate weight), while for standard data, transmission speed is prioritized. Furthermore, adaptation strategies are developed based on the physical characteristics of DDR and the data size (continuous large blocks or scattered data).

[0038] The optimization process relies on Convolutional Neural Networks (CNN) to train historical data sets (such as transmission rate, error rate, parameter configuration, etc.), extract features and optimize the loss function through multi-layer convolution, and finally output dynamically adjusted target data transmission parameters. For example, the burst length (Burstlength, the number of single transmissions) and the amount of data transmitted per single transmission (Axsize) are adjusted to match the bank group distribution of DDR, or the cyclic redundancy check (CRC) strategy is selected based on the existence of metadata.

[0039] Step S104 : transmitting the read and write data to the target storage location through multiple parallel channels according to the target data transmission parameters.

[0040] The target storage location refers to the physical storage unit where the data is ultimately written or read, which can be, for example, memory (such as DDR), flash memory, and other storage media. Specifically, data transmission is carried out through multiple parallel channels to improve transmission efficiency and optimize bandwidth utilization. By adjusting the target data transmission parameters, the controller can perform fine-grained optimization based on different types of data, transmission scale, and physical characteristics of the hardware, making data transmission more stable and efficient. During the transmission process, the controller adjusts the details of the data transmission, such as the burst length and data volume of the transmission, according to the target data transmission parameters, to ensure that when distributing data among multiple channels, the transmission can optimally match the characteristics of the storage medium, thereby improving the performance and stability of the entire transmission process. Ultimately, the read and write data will be transferred to the target storage location and stored or read according to the set rules.

[0041] The data transmission method provided by the embodiment of the present invention obtains key parameter information by parsing the read and write data, dynamically optimizes and generates matching target transmission parameters based on parameter attributes, realizes precise adaptation of the transmission strategy, utilizes multiple parallel channels for data transmission, significantly improves hardware resource utilization and overall transmission efficiency, effectively reduces latency, and ensures that the data transmission process is both efficient and targeted by combining parameter optimization with parallel transmission to meet the diverse needs of different storage media.

[0042] In this embodiment, a data transmission method is provided, which can be used in a computer device, such as a controller inside an NVMe module. Figure 2 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 2 As shown, the process includes the following steps:

[0043] Step S201: Get the read and write data to be transmitted. Figure 1 Step S101 of the illustrated embodiment will not be described in detail here.

[0044] Step S202: parse the read / write data to determine parameter information of the read / write data.

[0045] Specifically, the above step S202 includes:

[0046] Step S2021: parse the read / write data to obtain metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, and transmission parameters corresponding to the read / write data.

[0047] Metadata configuration parameters refer to metadata configuration information carried in read and write data, such as the data integrity protection type (such as metadata in DIX or DIF transmission). This parameter is used to identify whether the data contains protection information (such as CRC checksum), thereby determining the priority weighting for accuracy or speed during data transmission.

[0048] Data continuity configuration parameters refer to the continuity of read / write data during storage or transmission. For example, whether read / write data is stored or transmitted as contiguous blocks of addresses. For example, batch transfer optimization can be used for contiguous data, while dynamic transfer parameters must be adjusted for non-contiguous data.

[0049] The data command type configuration parameter refers to the type of the current I / O command (read command / write command). Different command types correspond to different transmission logic and parameter configurations.

[0050] The transmission parameters refer to a set of parameters that directly control the data transmission process, and may include, for example, Burst Length, Axsize, the base address of the target storage unit (DDR Base Address), and Bank Group (BG) allocation strategy.

[0051] Specifically, key information is extracted from the read and write commands sent by the host and the read and write data to be transmitted. First, metadata configuration parameters are obtained by checking flags in the packet header or specific fields, for example, to detect whether DIX (data and metadata are transmitted separately) or DIF (data and metadata are transmitted together) protection information is included. If metadata is present, its type and checksum (such as CRC value) are further analyzed. Second, data continuity configuration parameters are determined by analyzing the address sequence of the data block, for example, checking whether the logical block addresses (LBAs) are continuous or follow a certain pattern. Data command type configuration parameters are determined by parsing the command's opcode (Opcode) to distinguish between read, write, and other operation types (such as Trim and Flush). Finally, transmission parameters (such as Burst Length, Axsize, and DDR base address) are directly extracted from the command's configuration fields. For example, the burst length and transfer granularity are obtained from the "Data Transfer Control" field of the NVMe command, or the base address of the target DDR storage is obtained from the address mapping table. This process is implemented through hardware logic or firmware to ensure efficient parsing and parameter separation.

[0052] Step S2022: parse the data command type configuration parameters to determine the target command for reading and writing data.

[0053] The target command refers to the read and write command currently to be processed, and its type and associated parameters need to be obtained by parsing the command field. Specifically, the data command type is determined by the opcode and additional fields in the command. For example, the read command (Opcode 0x02) and write command (Opcode 0x01) defined by the NVMe protocol are directly distinguished by parsing the opcode in the command header. For complex commands (such as write operations with metadata), it is also necessary to check the "attribute" field of the command to confirm whether the DIX / DIF mode is enabled. The final determination of the target command requires a combination of the command type, data flow direction (Host to NVMe or NVMe to storage unit), and transmission mode (single or batch), and the corresponding operation instructions are ultimately generated by the control logic.

[0054] Step S2023: Obtain the associated parameters corresponding to the target command, and generate an identifier between the target command and the associated parameters.

[0055] The parameter information includes at least one of metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, transmission parameters and identifiers.

[0056] Associated parameters refer to additional parameters bound to the target command. An identifier is a unique identifier (ID) assigned to each read / write command or data entry, used to uniquely identify the command and its associated parameters. Specifically, after parsing each target command, its associated parameters are extracted, such as transmission parameters (e.g., Burst Length, Axsize), storage address information (e.g., DDR base address), and data attributes (e.g., whether it is continuous or contains metadata). Each command is assigned a unique identifier (ID) embedded in the highest bit field of the command or data. For example, in the command submission queue (SQ), the controller assigns a command identifier (CID) to each command. This CID is bound to the parsed parameters (e.g., Burst Length, Axsize) and stored in an internal information table. Identifiers are generated using hardware counters or hash algorithms to ensure uniqueness. For example, an incremental counter is used to generate a globally unique identifier (ID) for each command and store it in association with the parameters. Subsequent processes use this ID to quickly retrieve the corresponding parameters, achieving dynamic binding between commands and parameters.

[0057] The data transmission method provided by the embodiments of the present invention achieves accurate identification of the multidimensional characteristics of read and write data by comprehensively analyzing metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, and transmission parameters, ensuring the integrity and pertinence of parameter information. By generating unique identifiers between target commands and associated parameters, the correspondence between data and commands is clarified, enhancing the consistency and traceability of parameter management. It also supports the flexible extraction of key parameters, providing structured input for subsequent optimization and improving the adaptation efficiency of data transmission strategies.

[0058] Step S203 : Optimizing the parameter information based on the attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data.

[0059] Specifically, the above step S203 includes:

[0060] Step S2031 : determining the data transmission priority of the read / write data based on the detection result of the metadata contained in the read / write data.

[0061] The detection result refers to the real-time detection of read and write data attributes. Data transmission priority represents the weight of the transmission policy that is dynamically adjusted based on the detection result. Specifically, the presence of metadata directly affects the data transmission priority. The presence of metadata is determined by using a flag bit or a checksum field (such as the Guard field in DIX) to determine whether the data contains metadata. The data transmission priority of the read and write data is determined based on the detection result.

[0062] In some optional implementations, the above step S2031 includes:

[0063] Step a1: when the read / write data includes metadata, determining the first transmission priority of the read / write data, where the first transmission priority is used to indicate that data transmission accuracy of the read / write data is better than data transmission rate.

[0064] Step a2: when the read / write data does not include metadata, determining a second transmission priority of the read / write data, where the second transmission priority is used to indicate that the data transmission rate of the read / write data is superior to the data transmission accuracy.

[0065] The data transmission priority includes a first transmission priority and a second transmission priority.

[0066] If metadata exists (such as data integrity protection information), priority is given to ensuring transmission accuracy. In this case, the transmission rate weight is reduced and the check frequency is increased (for example, a CRC check is triggered after each burst of data is transmitted). Conversely, if metadata is absent, priority is given to increasing the transmission rate, using a larger burst length and a more relaxed check strategy. The specific quantification of priorities is achieved through preset rules, for example:

[0067] High priority (with metadata): error rate threshold ≤ 0.01%, transmission rate weight set to 30%;

[0068] Low priority (no metadata): the error rate threshold is ≤ 0.1%, and the transmission rate weight is set to 70%.

[0069] In addition, regardless of whether metadata data is transmitted or not, the accuracy and rate of data transmission must meet basic set values.

[0070] Step S2032: Optimize the parameter information according to the data transmission priority to generate target data transmission parameters corresponding to the read and write data.

[0071] By detecting whether the data carries metadata, the transmission parameter weights are dynamically adjusted, and the target data transmission parameters that balance speed and accuracy are generated by combining the training results of the convolutional neural network on historical transmission data. Specifically, when the data contains metadata (such as DIF / DIX format), accuracy is prioritized. At this time, parameter optimization will reduce the transmission rate weight, adopt a smaller burst length, increase the check frequency, and select a high-reliability address mapping strategy. If the data has no metadata protection, the speed is prioritized, a larger burst length is adopted, and redundant checks are reduced. At the same time, based on the convolutional neural network, the optimal parameter combination is learned from historical data sets (such as DDR type, data continuity, command type and other scenario features), the AXI bus parameters (such as Axsize, base address allocation) and the transmission order across the bank group are dynamically adjusted to generate the target data transmission parameters corresponding to the read and write data.

[0072] Step S204: According to the target data transmission parameters, the read and write data are transmitted to the target storage location through multiple parallel channels. Figure 1 Step S104 of the illustrated embodiment will not be described in detail here.

[0073] The data transmission method provided in this embodiment dynamically prioritizes data transmission based on metadata detection results (e.g., prioritizing accuracy when metadata is present, and prioritizing transmission rate when metadata is absent), enabling differentiated parameter configuration. This prioritized approach optimizes transmission parameters to ensure the efficiency and reliability of critical data. A priority-driven parameter adaptation mechanism flexibly balances transmission rate and accuracy requirements, enhancing the intelligence and scenario adaptability of data transmission strategies.

[0074] In this embodiment, a data transmission method is provided, which can be used in a computer device, such as a controller inside an NVMe module. Figure 3 is a flow chart of a data transmission method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0075] Step S301: Get the read and write data to be transmitted. Figure 2 Step S201 of the illustrated embodiment will not be described in detail here.

[0076] Step S302: parse the read / write data to determine the parameter information of the read / write data. Figure 2 Step S202 of the illustrated embodiment will not be described in detail here.

[0077] Step S303: Based on the attribute characteristics of the parameter information, the parameter information is optimized to generate target data transmission parameters corresponding to the read and write data. Figure 2Step S203 of the illustrated embodiment will not be described in detail here.

[0078] Step S304: transmitting the read and write data to the target storage location through multiple parallel channels according to the target data transmission parameters.

[0079] Specifically, the parameter information includes the base address corresponding to the read and write data. The above step S304 includes:

[0080] Step S3041: parse the base address to determine the number of random access memories corresponding to the read and write data, and determine the number of random access memories as the number of parallel channels.

[0081] The base address refers to the starting address of the target storage location (such as DDR memory) for read and write operations during data transmission. The base address is used to determine the specific storage location of data in the storage device, ensuring that data can be written or read correctly. For example, the base address of DDR memory can be specified by the ddr base address parameter in the command. The number of random access memories refers to the number of DRAMs used for parallel transmission. Each DRAM corresponds to an independent parallel channel. For example, if there are four DDR banks, there are four parallel channels, and data can be written to all four DDR banks simultaneously to improve transmission efficiency. Specifically, the upper bits of the base address (such as ddr base address) contain the identification information of the DDR module or bank group (BG). For example, in the DDR4 architecture, the upper bits of a base address can represent different BGs or physical DDR chip numbers. During parsing, specific fields of the base address (such as the top two or three bits) are extracted to map to specific DDR modules. For example, if the upper two bits of the base address are "00" for DDR0 and "01" for DDR1, consecutive data can be distributed to multiple DDR modules to support parallel transmission. After parsing, the system dynamically allocates the same number of parallel channels (each corresponding to a sub-controller) based on the number of identified DDR modules (e.g., four DDR chips), enabling parallel writing of data across multiple DDR modules. This process combines hardware address mapping rules with software configuration logic to ensure that base address parsing accurately matches physical storage resources.

[0082] Step S3042: writing the read and write data into multiple target random access memories through multiple parallel channels according to the target data transmission parameters.

[0083] Among them, multiple target random access memories are used to represent target storage locations.

[0084] The target data transmission parameters define the granularity and method of data transmission. Specifically, read and write data are divided into multiple sub-blocks based on the number of parallel channels. For example, if the target data transmission parameters specify burst length = 8 and the number of parallel channels is 4, each channel is assigned 2 continuous transmission tasks. Each sub-controller independently manages a channel and writes data to the corresponding DDR module through an independent AXI bus. The cross-Bank Group principle is followed during the transmission process: for example, in DDR4, the address range of BG0 is written first, and then looped to BG1 to reduce storage unit conflicts and improve bandwidth utilization. In addition, if the automatic adjustment parameters (such as CNN optimized parameters) are enabled, the sub-controller will dynamically adjust the transmission timing (such as address step size, burst number) according to the DDR type (such as DDR4 / DDR5) to ensure that each channel completes the write with the optimal configuration. Ultimately, data is efficiently written through multi-channel parallel transmission, making full use of hardware resources.

[0085] In some optional implementations, the target random access memory includes multiple memory banks, and step S3042 includes:

[0086] Step b1, obtaining a first order of the target random access memory and a second order of writing read and write data into the memory bank.

[0087] A memory bank is an internal structure within DRAM that improves parallelism in memory access. It's also called a bank group (BG). Each bank can operate independently. For example, DDR4 typically has two banks (BG0 and BG1).

[0088] The first order of data acquisition is based on predefined rules or dynamic configuration of the hardware architecture. Specifically, when there are multiple DDRs (such as DDR0 to DDR3), data allocation follows a round-robin mechanism. For example, the starting DDR number of the data block is determined by parsing the base address in the command or the user-configured DDR binding rules. The data block of each new command is allocated sequentially starting from DDR0, and is written to DDR1, DDR2, DDR3, and then looping back to DDR0.

[0089] Second-order access relies on the access rules defined by the DDR protocol. Taking DDR4 as an example, each DDR chip contains multiple Bank Groups (such as BG0 and BG1). Cross-BG access reduces row activation latency. When writing, writes must be performed sequentially according to the BG order.

[0090] Step b2: Allocate the read and write data to multiple target random access memories in a first order through multiple parallel channels.

[0091] Multiple parallel channels are implemented by binding the sub-controller to the target random access memory (such as multiple DDR chips) one by one. Each sub-controller independently manages a set of AXI buses and is responsible for writing data to the corresponding DDR. The first order determines the data distribution rule among multiple DDRs. For example, if there are 4 DDRs (DDR0-DDR3), the data blocks will be allocated in sequence in a round-robin manner: the first data block is allocated to DDR0, the second to DDR1, and so on, until it returns to DDR0 after DDR3. During allocation, the sub-controller parses the DDR number based on the base address (such as through address high-order mapping) and dynamically selects the target DDR according to the first order. This allocation method fully utilizes the parallelism of multiple channels, ensuring that data is evenly distributed and avoiding access conflicts, thereby improving overall transmission efficiency.

[0092] Step b3: For any target random access memory, write the read and write data into the memory bank in sequence according to the second order.

[0093] Inside a single target random access memory (such as a single DDR chip), the second order controls the write path of data in the memory bank. Taking DDR4 as an example, each DDR contains multiple BGs (such as BG0 and BG1), and the cross-BG order must be followed when writing to reduce the row activation delay. The sub-controller will first write the data block to the specified row of BG0, and then switch to the corresponding row of BG1, and repeat. Specifically, the sub-controller generates a BG selection signal based on the parsed data address (such as row address, column address), and activates the row buffers of different BGs in sequence according to the second order. At the same time, the data is divided into several groups according to the burst transmission (Burst Length) parameter, and each group is written to the continuous storage cells of the same BG in ascending address order. This process is controlled by a hardware state machine or microcode to ensure that the data is written efficiently along the optimal path while meeting the protocol timing requirements.

[0094] The data transmission method provided by the embodiments of the present invention dynamically determines the number of RAMs by parsing the base addresses corresponding to read and write data, and maps this number to the number of parallel channels, achieving efficient matching of storage resources and transmission channels. Using multiple parallel channels to simultaneously write to multiple target RAMs significantly improves the parallelism and overall throughput of data transmission. A base address-driven channel allocation mechanism ensures precise adaptation of data storage locations, avoids resource conflicts, and optimizes hardware utilization.

[0095] Step S305 , detecting the transmission rate and transmission error rate of the read / write data transmitted to the target storage location through the multiple parallel channels.

[0096] The transfer rate is the amount of data successfully transferred per unit time and can be measured in MB / s or GB / s. Specifically, a timer and counter can be used to measure the transfer rate of read and write data through multiple parallel channels to the target storage location. The timer records the time from the start to the completion of the transfer, and the counter counts the total amount of data transferred (e.g., number of bytes). The transfer rate formula is:

[0097]

[0098] The transmission error rate refers to the proportion of data errors during data transmission. Specifically, the verification method is selected based on whether metadata exists. If the data contains metadata (such as the DIF / DIX protection field), the CRC value is directly verified. If no metadata is present, the CRC module is enabled to calculate the number of error bytes. The transmission error rate calculation formula is:

[0099]

[0100] Step S306: Correlate the transmission rate, the transmission error rate, and the identifier to generate correlation data.

[0101] Associated data refers to a data set generated by binding the transmission rate and error rate of each transmission to a unique identifier (ID). This data is stored in the optimization module's information table. Specifically, each data transmission task is assigned a unique identifier (ID) at the initial stage. After the transmission is completed, the measured transmission rate and error rate are bound to this ID to form a structured record. For example, the record format is:

[0102] {ID: 0x1234, transfer rate: 5 GB / s, transfer error rate: 0.01%, DDR type: DDR4, data size: 64 KB}.

[0103] These records are stored in an information table within the NVMe module and maintained by ID index. The association process is automatically completed through hardware logic, ensuring that the performance indicators of each data item are traceable and providing a labeled dataset for subsequent training.

[0104] Step S307: When the associated data accumulates to a preset threshold, the parameter information is optimized using the associated data to obtain optimized parameter information.

[0105] The preset threshold refers to the amount of data accumulation set by the user to trigger parameter optimization. For example, when the associated data accumulates to 2,000 items, the training process is automatically started, and this data is used to optimize the convolutional neural network model and generate more optimal transmission parameters. Specifically, when the associated data accumulates to the preset threshold, the training process is started. The historical data is normalized (for example, the rate and error rate are mapped to the range of 0-1), and key features (such as DDR type, data continuity, and metadata presence) are extracted. The convolutional neural network is used to perform multi-layer convolution and pooling on the features to extract data patterns. For example, for DDR4 continuous write scenarios, the model learns the optimal combination of burst length = 16 and Axsize = 64B to maximize the rate; for data containing metadata, the model prefers burst length = 8 to reduce the error rate. The model weights are optimized through a loss function (such as mean squared error) to generate a new parameter combination (i.e., optimized parameter information) and update it to the information table. Thereafter, the sub-controller chooses whether to apply the new parameters according to the user configuration during transmission to achieve dynamic adaptation. The new parameters are applied to subsequent transmissions, and their effects are continuously monitored and fed back into the training dataset, forming a self-optimizing closed-loop system.

[0106] The data transmission method provided by the embodiments of this invention monitors transmission rates and error rates in real time and associates them with unique identifiers to form a traceable performance dataset, providing an objective basis for parameter optimization. When the associated data accumulates to a preset threshold, an automatic optimization mechanism is triggered to dynamically adjust parameters, enhancing the adaptability and continuous improvement capabilities of the transmission strategy. Combined with identifiers that accurately match historical transmission scenarios, this ensures a highly targeted optimization process, effectively reducing error rates, improving transmission efficiency, and minimizing manual intervention costs.

[0107] In addition, the optimization process further integrates an environmental perception module, collecting hardware operating status (such as temperature and voltage) in real time as training features. Parameter weights are dynamically adjusted using a multi-objective loss function (rate, error rate, energy consumption, and latency). When a storage media upgrade or scenario switch is detected, an incremental learning mechanism is activated to rapidly update the model based on a small amount of new data, improving cross-scenario adaptability. Specifically, the hardware operating status of the NVMe module (such as temperature, supply voltage, and clock stability) is monitored in real time and input into the convolutional neural network training process as additional features. For example, in high-temperature environments, the transmission rate weight is automatically reduced to prevent hardware overheating, while the burst length is adjusted to match timing stability under voltage fluctuations. Secondly, new optimization objectives such as energy consumption, latency, and hardware lifespan are added to the loss function. For example, in mobile device scenarios, reducing transmission energy consumption (such as reducing DDR activation frequency) is prioritized; in data center scenarios, maximizing throughput while maintaining error rate is prioritized. Dynamic weight allocation is used to balance the priorities of rate, accuracy, energy consumption, and hardware loss. Furthermore, when a storage medium replacement (e.g., upgrading from DDR4 to DDR5) or a sudden change in the transmission scenario is detected, incremental learning is initiated based on a small amount of new data (e.g., 100 pieces of related data) to quickly adapt to the new environment. Furthermore, cross-scenario model migration is supported (e.g., migrating parameters trained in the data center to edge computing devices), reducing the cost of repeated training.

[0108] The data transmission method provided by the embodiment of the present invention realizes dynamic parameter adjustment driven by hardware status through environmental perception and multi-objective optimization mechanism, which significantly enhances the system's adaptability to changes in physical environment such as temperature and voltage while improving transmission rate and accuracy. By introducing multi-dimensional optimization targets such as energy consumption, latency and hardware life, it can flexibly meet differentiated scenario requirements such as low power consumption on mobile terminals and high throughput in data centers. Combining incremental learning with cross-scenario migration capabilities, it greatly reduces the cost of model training when upgrading storage media or switching scenarios, improves the generalization and scalability of the technical solution, and provides a more intelligent, robust and sustainable optimization strategy for NVMe data transmission.

[0109] In the following of this embodiment, the above-mentioned data transmission method will be exemplified by using a complete application scenario.

[0110] For writing and reading data from the host, the host first sends the read and write command to the SQ and rings the doorbell. The NVMe module actively retrieves the command. After the command is retrieved and the data is moved through the local central processor, it returns to the completion queue (Completion Queue, CQ). The data is actively taken from the memory by the NVMe module, and the relevant information of the data and the relevant information of the DDR are read, and a unique identifier ID (the highest bit of the command or data) is assigned to this set of information. Then the relevant parameters are read and stored in the information table, including whether the data transmission parameters can be automatically adjusted. The data with the assigned unique identifier ID is allocated in the order of the number of DDRs (for example, if there are four DDRs, the data of each command will be allocated from DDR0 to DDR3 in sequence, and so on), and enters the corresponding DDR sub-controller. The sub-controller will read the user-configured parameters to determine whether to use the automatically adjusted data transmission parameters: if so, match the current DDR information and use the optimal transmission parameters obtained by multiple trainings to transmit data. If not, continue to transmit according to the user-configured transmission parameters. After the data transmission is completed, the transmission rate and error rate of the data will be obtained and stored. The storage location corresponds to the unique identifier ID of the command and data. When the data set is stored to a certain amount, this information will be trained and the optimal transmission parameters corresponding to DDR4 or DDR5 will be obtained. Figure 4 The optimal transmission parameters are prepared to be applied to the next data transmission.

[0111] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus the necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method.

[0112] The embodiment of the present application also provides a data transmission device, such as Figure 5 As shown, including:

[0113] An acquisition module 501 is used to acquire read and write data to be transmitted;

[0114] The parsing module 502 is used to parse the read and write data and determine parameter information of the read and write data;

[0115] A first optimization module 503 is configured to optimize the parameter information based on the attribute characteristics of the parameter information and generate target data transmission parameters corresponding to the read and write data;

[0116] The transmission module 504 is configured to transmit the read and write data to a target storage location through multiple parallel channels according to target data transmission parameters.

[0117] In some optional implementations, the parsing module 502 includes:

[0118] A first parsing submodule is used to parse the read and write data to obtain metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, and transmission parameters corresponding to the read and write data;

[0119] The second parsing submodule is used to parse the data command type configuration parameters and determine the target command for reading and writing data;

[0120] A generation submodule is used to obtain associated parameters corresponding to the target command and generate an identifier between the target command and the associated parameters; wherein the parameter information includes at least one of metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, transmission parameters and identifiers.

[0121] In some optional implementations, the first optimization module 503 includes:

[0122] A determination submodule, configured to determine a data transmission priority of the read / write data based on a detection result of metadata contained in the read / write data;

[0123] The optimization submodule is used to optimize the parameter information according to the data transmission priority and generate the target data transmission parameters corresponding to the read and write data.

[0124] In some optional implementations, the determining submodule includes:

[0125] a first determining unit, configured to determine a first transmission priority of the read / write data when the read / write data includes metadata, the first transmission priority being used to indicate that data transmission accuracy of the read / write data is superior to data transmission rate;

[0126] The second determination unit is used to determine the second transmission priority of the read / write data when the read / write data does not include metadata, and the second transmission priority is used to indicate that the data transmission rate of the read / write data is better than the data transmission accuracy; wherein the data transmission priority includes the first transmission priority and the second transmission priority.

[0127] In some optional implementations, the transmission module 504 includes:

[0128] A third parsing submodule is used to parse the base address, determine the number of random access memories corresponding to the read and write data, and determine the number of random access memories as the number of parallel channels;

[0129] The transmission submodule is used to write the read and write data into multiple target random access memories through multiple parallel channels according to the target data transmission parameters; wherein the multiple target random access memories are used to represent the target storage location.

[0130] In some optional implementations, the transmission submodule includes:

[0131] An acquisition unit, configured to acquire a first sequence of the target random access memory and a second sequence of writing the read and write data into the memory bank;

[0132] A distribution unit, configured to distribute read and write data to a plurality of target random access memories in a first order through a plurality of parallel channels;

[0133] The transmission unit is used to write the read and write data into the memory bank in sequence according to the second order for any target random access memory.

[0134] In some optional implementations, the data transmission device further includes:

[0135] a detection module, configured to detect a transmission rate and a transmission error rate of read and write data transmitted to a target storage location through multiple parallel channels;

[0136] an association module, configured to associate the transmission rate, the transmission error rate, and the identifier to generate association data;

[0137] The second optimization module is configured to optimize the parameter information using the associated data when the associated data accumulates to a preset threshold value, thereby obtaining the optimized parameter information.

[0138] For the description of the features in the embodiment corresponding to the data transmission device, please refer to the relevant description of the embodiment corresponding to the data transmission method, and will not be repeated here.

[0139] The embodiment of the present application also provides an electronic device, such as Figure 6 As shown, it includes a processor 10 and a memory 20, wherein the memory 20 stores a computer program, and the processor 10 is configured to run the computer program to execute the steps in any one of the above-mentioned data transmission method embodiments.

[0140] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is configured to execute the steps of any of the above-mentioned data transmission method embodiments when running.

[0141] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk, or an optical disk.

[0142] An embodiment of the present application further provides a computer program product, which includes a computer program. When the computer program is executed by a processor, the steps in any of the above-mentioned data transmission method embodiments are implemented.

[0143] An embodiment of the present application further provides another computer program product, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in any of the above-mentioned data transmission method embodiments are implemented.

[0144] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the above description has generally described the components and steps of each example according to their functions. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] The above is a detailed introduction to a data transmission method, device, electronic device and storage medium provided by the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method and core ideas of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present application, several improvements and modifications can be made to the present application, and these improvements and modifications also fall within the scope of protection of the claims of the present application.

Claims

1. A data transmission method, characterized in that: The method comprises: Get the read and write data to be transmitted; Parsing the read / write data to determine parameter information of the read / write data; Based on the attribute characteristics of the parameter information, the parameter information is optimized to generate target data transmission parameters corresponding to the read and write data; The read and write data are transmitted to a target storage location through multiple parallel channels according to the target data transmission parameters.

2. The data transmission method according to claim 1, wherein: The parsing of the read / write data to determine parameter information of the read / write data includes: Parsing the read / write data to obtain metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, and transmission parameters corresponding to the read / write data; Parsing the data command type configuration parameter to determine the target command for reading and writing data; Obtaining associated parameters corresponding to the target command, and generating an identifier between the target command and the associated parameters; The parameter information includes at least one of metadata configuration parameters, data continuity configuration parameters, data command type configuration parameters, transmission parameters and identifiers.

3. The data transmission method according to claim 1 or 2, characterized in that: The optimizing the parameter information based on the attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data includes: Determining a data transmission priority of the read / write data based on a detection result of the metadata included in the read / write data; The parameter information is optimized according to the data transmission priority to generate target data transmission parameters corresponding to the read and write data.

4. The data transmission method according to claim 3, wherein: The determining the data transmission priority of the read / write data based on the detection result of the metadata included in the read / write data includes: When the read / write data includes the metadata, determining a first transmission priority of the read / write data, where the first transmission priority is used to indicate that data transmission accuracy of the read / write data is superior to data transmission rate; When the read / write data does not include the metadata, determining a second transmission priority of the read / write data, where the second transmission priority is used to indicate that a data transmission rate of the read / write data is superior to data transmission accuracy; The data transmission priority includes the first transmission priority and the second transmission priority.

5. The data transmission method according to claim 1, wherein: The parameter information includes a base address corresponding to the read / write data; and the step of transmitting the read / write data to a target storage location through multiple parallel channels according to the target data transmission parameter includes: Parsing the base address to determine the number of random access memories corresponding to the read / write data, and determining the number of random access memories as the number of the parallel channels; Writing the read / write data into a plurality of target random access memories through a plurality of the parallel channels according to the target data transmission parameters; Wherein, a plurality of target random access memories are used to represent the target storage location.

6. The data transmission method according to claim 5, characterized in that: The target random access memory includes a plurality of memory banks; and writing the read and write data into the plurality of target random access memories through the plurality of parallel channels according to the target data transmission parameters comprises: Obtaining a first order of the target random access memory and a second order of writing the read and write data into the memory bank; Allocating the read and write data to the plurality of target random access memories according to the first order through the plurality of parallel channels; For any of the target random access memories, the read and write data are written into the memory banks in sequence according to the second order.

7. The data transmission method according to claim 1, wherein: Also includes: detecting a transmission rate and a transmission error rate of the read / write data transmitted to the target storage location through a plurality of parallel channels; Associating the transmission rate, the transmission error rate, and the identifier to generate associated data; When the associated data accumulates to a preset threshold, the parameter information is optimized using the associated data to obtain the optimized parameter information.

8. A data transmission device, characterized in that: The device comprises: An acquisition module, used to obtain read and write data to be transmitted; A parsing module, configured to parse the read / write data and determine parameter information of the read / write data; A first optimization module is configured to optimize the parameter information based on attribute characteristics of the parameter information to generate target data transmission parameters corresponding to the read and write data; The transmission module is used to transmit the read and write data to the target storage location through multiple parallel channels according to the target data transmission parameters.

9. An electronic device, characterized in that: include: memory for storing computer programs; A processor, configured to implement the steps of the data transmission method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the data transmission method according to any one of claims 1 to 7.

Images