DPU high-degree-of-freedom dynamic storage method for index calculation

Abstract

The application relates to the technical field of data storage, and provides a DPU high-freedom dynamic storage method for index calculation, which comprises the following steps: writing a stock code to a system through a PCIe interface, and sequentially transmitting configuration data according to a preset order; processing the stock code according to a mapping rule through a PCIe dynamic configuration module; generating the mapping rule through a dynamic configuration unit, and transmitting the mapping rule to a dynamic data analysis unit and a data storage preprocessing unit; matching a stock data packet according to a pre-stored template through a data preprocessing unit, and analyzing a received stock data packet header; adding a specific mapping flag bit to a head of stock data according to the mapping rule through a dynamic data analysis unit; and selecting a storage unit for data transmission according to the mapping flag bit and the mapping rule through a data storage preprocessing module. The application adopts a two-level mapping mechanism to optimize storage management, and realizes efficient positioning and access of stored data.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data storage, specifically to a high-degree-of-freedom dynamic storage method for index calculation-oriented DPU. BACKGROUND

[0002] Traditional technical solutions usually rely on static rules for mapping when dealing with the storage and processing of index data. For example, specific fields of index data (such as index codes or timestamps) are mapped to fixed storage areas (such as folders or memory blocks), and other fields are mapped to specific storage locations. Such mapping rules are difficult to flexibly respond to dynamically changing data structures and complex query requirements in index calculation. Data is transmitted to storage devices (such as dynamic random storage units or disks) through standard transmission interfaces (such as PCIe, Ethernet, or other bus protocols).

[0003] The preprocessing of data is mainly completed on the host computer, including data parsing, format conversion, and screening, etc. This process lacks dynamic optimization capability, especially in the context of index calculation, which needs to process a large amount of real-time updated data. Subsequently, the preprocessed data is stored in the target storage area according to fixed rules. Such storage architecture is usually single-level (such as globally shared storage devices or partitioned storage devices). When accessing data, the storage address is calculated according to fixed rules and the data is retrieved from the storage device. However, this method is inefficient in the face of high concurrency and high complexity of index calculation tasks, especially in scenarios with strong data dynamics, it cannot meet the performance requirements. In addition, data display and analysis usually rely on host computer processing, further increasing the time delay and system overhead.

[0004] When comparing the traditional storage scheme with the high-degree-of-freedom dynamic mapping storage scheme proposed, the traditional scheme mainly has the following disadvantages and defects:

[0005] 1. Insufficient mapping flexibility: The traditional scheme usually uses fixed allocation for the storage of security codes, lacking a flexible dynamic mapping mechanism. Non-continuous security codes need to occupy a large amount of redundant storage space, resulting in low storage resource utilization.

[0006] 2. Low data processing efficiency: The traditional scheme lacks a unified data parsing mechanism, and security data needs to be preprocessed by the host computer or external devices before storage, resulting in additional processing delay and increased system complexity.

[0007] 3. Single storage structure, poor scalability: Most of the traditional schemes use a single-level storage architecture, lacking a hierarchical design of local storage and global storage, making it difficult to adapt to high-concurrency access or the storage needs of different priority data. SUMMARY

[0008] To help solve the above technical problems, the application provides a DPU high freedom dynamic storage method for index calculation, which adopts the following technical scheme:

[0009] The DPU high freedom dynamic storage method for index calculation is used in a DPU high freedom dynamic storage system for index calculation, wherein,

[0010] The DPU high freedom dynamic storage system for index calculation comprises:

[0011] a PCIe dynamic configuration module, a dynamic configuration unit, a dynamic data analysis unit, a data storage preprocessing unit, a data preprocessing unit, a local storage unit and a global storage unit;

[0012] The DPU high freedom dynamic storage method for index calculation comprises:

[0013] Step S1: securities codes are written to the system through a PCIe interface, and configuration data is transmitted in a preset order one by one, and the securities codes are processed by the PCIe dynamic configuration module according to a mapping rule, the mapping rule comprising: the first n bits of the securities codes are used as an index for first-level mapping to select a corresponding storage unit, and the last m bits of the securities codes are used as a basis for second-level mapping to select a corresponding continuous storage address, n and m are both positive integers;

[0014] Step S2: the mapping rule is generated by the dynamic configuration unit and transmitted to the dynamic data analysis unit and the data storage preprocessing unit;

[0015] Step S3: securities data from different trading markets are transmitted to the data preprocessing unit in the form of streaming data through an Ethernet unit, the securities data packets are matched according to a pre-stored template by the data preprocessing unit to identify the format and source of the securities data packets, and the header of the received securities data packet is analyzed;

[0016] Step S4: a specific mapping flag bit is added to the header of the securities data by the dynamic data analysis unit according to the mapping rule to identify the mapping information of the securities data, and the storage unit is selected for data transmission by the data storage preprocessing module according to the mapping flag bit and the mapping rule;

[0017] Step S5: the storage unit comprises a local storage unit and a global storage unit, and the securities data are stored by the local storage unit and the global storage unit.

[0018] Preferably, the data preprocessing unit, the dynamic data analysis unit and the data storage preprocessing unit are connected in sequence, the PCIe dynamic configuration module is connected with the data storage preprocessing unit and the dynamic data analysis unit respectively, and the data storage preprocessing unit is connected with the local storage unit and the global storage unit respectively.

[0019] Preferably, the DPU high-freedom dynamic storage system for index calculation further comprises an ETH unit connected with the data preprocessing unit, and the ETH unit is configured to receive a security information data stream from the market.

[0020] Preferably, the data storage preprocessing module comprises a data distribution unit configured to distribute the security data to the local storage unit and the global storage unit.

[0021] Preferably, the step S3 comprises: analyzing the received security data packet header, which comprises: identifying and removing irrelevant data fields according to the mapping rule and the template.

[0022] Preferably, the step S5 comprises: the global storage unit is implemented by a register, and the local storage unit is constructed by a dynamic random memory; if the security information corresponding to the security data is global information, the security data is stored in the global storage unit; if the security information corresponding to the security data is local information, the security data is stored in the local storage unit.

[0023] In summary, the DPU high-freedom dynamic storage method for index calculation of the present application significantly improves the storage efficiency and real-time processing capability through the cooperation relationship among the PCIe dynamic configuration module, the Ethernet receiving module, the dynamic random storage unit and the register two-level storage module and the dynamic mapping logic, and meets the demand of high performance, high expansibility and high security in the financial field. In the DPU high-freedom dynamic storage method for index calculation of the present application, the DPU utilizes its parallel processing capability and high-speed response characteristics to realize efficient and accurate control of the data storage process. The present application integrates multiple storage units and control modules, adopts a two-level mapping mechanism to optimize storage management, and realizes efficient positioning and access of stored data. BRIEF DESCRIPTION OF DRAWINGS

[0024] Figure 1 a flowchart of an embodiment of the DPU high-freedom dynamic storage method for index calculation of the present application;

[0025] Figure 2 a schematic block diagram of an embodiment of the DPU high-freedom dynamic storage system for index calculation of the present application. DETAILED DESCRIPTION

[0026] The application will be described further below with reference to the drawings, which are provided by way of explanation only and are not intended to limit the application. It will be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to limit the application.

[0027] A DPU (Data Processing Unit) is a new type of processor specially used to accelerate data processing tasks in a data center. The DPU is a new generation of processor oriented to data-centric computing, which integrates complete data center functions in a single chip, together with CPU and GPU to form the three pillars of new computing. The DPU focuses on performing basic layer application tasks such as network protocol processing, switching and routing computing, encryption and decryption, and data compression, etc. heavy and specific tasks, thereby reducing the burden of CPU and GPU, and improving the overall efficiency and performance of the data center.

[0028] Figure 1 FIG. 1 is a flowchart of an embodiment of a DPU high-freedom dynamic storage method for exponential calculation according to the present application, Figure 2 FIG. 2 is a schematic block diagram of an embodiment of a DPU high-freedom dynamic storage system for exponential calculation according to the present application.

[0029] In combination with Figure 1 And Figure 2 It can be understood that the DPU high-freedom dynamic storage system for exponential calculation includes an ETH unit, a PCIe dynamic configuration module, a dynamic configuration unit, a dynamic data analysis unit, a data storage preprocessing unit, a data preprocessing unit, a local storage unit, and a global storage unit. The ETH unit is connected to the data preprocessing unit, the dynamic data analysis unit, and the data storage preprocessing unit in sequence. The PCIe dynamic configuration module is connected to the data storage preprocessing unit and the dynamic data analysis unit, respectively. The data storage preprocessing unit is connected to the local storage unit and the global storage unit, respectively. The ETH unit is used to receive security information data stream from the market. The PCIe interface is responsible for receiving security codes from the upper computer, and realizes rule configuration of the data storage preprocessing unit through the PCIe dynamic configuration. The data preprocessing unit analyzes the security information from the market, and distributes and schedules the data according to the preset rules by the data storage preprocessing unit. The global storage unit is realized by using a register, and the local storage unit is constructed by using a dynamic random memory. According to the data distribution of the security information, a self-defined two-level address mapping design is performed to improve the data processing efficiency and the flexibility of storage. The system not only improves the parallelism of data processing, but also provides higher flexibility and configurability in the process of data distribution and scheduling, and meets the demand of fast processing and efficient storage of security information.

[0030] The DPU high-degree-of-freedom dynamic storage method for index calculation comprises the following steps: S1: writing a security code to a system through a PCIe interface, and sequentially transmitting configuration data in a preset order, processing the security code through a PCIe dynamic configuration module according to a mapping rule, wherein the mapping rule comprises performing first-level mapping on the first n bits of the security code as an index to select a corresponding storage unit, and performing second-level mapping on the last m bits of the security code as a basis to select a corresponding continuous storage address, wherein n and m are positive integers.

[0031] S2: generating the mapping rule through a dynamic configuration unit, and transmitting the mapping rule to a dynamic data analysis unit and a data storage preprocessing unit.

[0032] S3: transmitting security data from different trading markets to a data preprocessing unit in the form of streaming data through an Ethernet unit, matching a security data packet according to a pre-stored template through the data preprocessing unit to identify the format and source of the security data packet, and analyzing a received security data packet header. S3 further comprises: analyzing the received security data packet header comprises: identifying and removing irrelevant data fields according to the mapping rule and the template.

[0033] S4: adding a specific mapping flag bit to the header of the security data according to the mapping rule through a dynamic data analysis unit, wherein the mapping flag bit is used to identify the mapping information of the security data, and selecting a storage unit for data transmission according to the mapping flag bit and the mapping rule through a data storage preprocessing module.

[0034] S5: the storage unit comprises a local storage unit and a global storage unit, and the security data is stored through the local storage unit and the global storage unit. S5 further comprises: the global storage unit is implemented by a register, and the local storage unit is constructed by a dynamic random access memory, if the security information corresponding to the security data is global information, the security data is stored in the global storage unit, and if the security information corresponding to the security data is local information, the security data is stored in the local storage unit.

[0035] Specifically, S1, as Figure 1As shown, before the system starts, data pre-configuration is required, that is, by entering codes through the host computer, the PCIe dynamic configuration module and the dynamic configuration unit are mapped according to the rules. Specifically, the host computer writes the securities codes to the system through the PCIe interface, and transmits the data in a predetermined order. The PCIe dynamic configuration module processes these securities codes according to the pre-set mapping rules. This process first uses the first three digits of the securities code as an index for first-level mapping to determine the first part of the storage address; then uses the last three digits as the basis for second-level mapping to further accurately locate the second part of the storage address. The design of this process ensures the efficiency and accuracy of the mapping operation.

[0036] Wherein, the securities codes configured by the host computer do not need to be input in continuous order. The dynamic configuration unit will convert the non-continuous securities codes into continuous storage addresses according to the mapping rules, realizing flexible and efficient storage space management. This mapping mechanism not only optimizes the use of storage resources, but also effectively improves the response speed and data access efficiency of the system. The innovation of this scheme lies in that through the parallel processing capability of DPU, this complex and dynamic two-level mapping operation can be efficiently realized, adapting to diversified input demands and ensuring the efficiency and security of data access.

[0037] Step S1 enters securities codes through the host computer and writes these codes into the system using the PCIe interface. The user enters securities codes through the host computer (computer or dedicated control device). These securities codes are the basis for subsequent data processing. The host computer writes these securities codes into the system through the PCIe (Peripheral Component Interconnect Express) interface. The PCIe interface is a high-speed serial computer expansion bus standard that can support high-bandwidth data transmission, ensuring that data can be quickly and accurately entered into the system. The system uses a two-level mapping mechanism to determine the storage address of the securities codes.

[0038] First-level mapping: The first three digits of the security code are used as an index for first-level mapping. This step determines the first part of the storage address. For example, if the security code is six digits long, the first three digits may represent a certain classification or identification, which is used to preliminarily locate the storage area. Second-level mapping: The last three digits of the security code are used as the basis for second-level mapping to further precisely locate the second part of the storage address. This step ensures that each security code can be mapped to a unique storage address. These mapping rules are pre-set, and the system processes the security codes according to these rules. Through the two-level mapping mechanism, the system can efficiently and accurately determine the storage address of each security code. This mechanism avoids the problem of storage address conflicts or repetition that may occur in traditional methods, improving the utilization of storage resources. Utilizing the parallel processing capability of the DPU (Data Processing Unit), the system can efficiently handle a large number of security code mapping operations. This parallel processing capability enables the system to handle a large number of data input requests in a short time, improving the overall performance of the system.

[0039] This process supports the conversion of discontinuous security codes into continuous storage addresses. This means that users do not need to input security codes in a specific order, and the system can automatically process these discontinuous codes and map them to continuous storage addresses. This flexibility improves the user experience and facilitates subsequent data processing and management.

[0040] In summary, step S1 inputs security codes through the host computer and writes these codes into the system using the PCIe interface. Through the two-level mapping mechanism, the system can efficiently and accurately determine the storage address of each security code. This mechanism not only improves the efficiency of data access but also ensures the accuracy, flexibility, and utilization of storage resources. At the same time, this process also supports the conversion of discontinuous security codes into continuous storage addresses, further improving the user experience and system performance.

[0041] S2, as shown in Figure 1 , the transmission of rule instructions is required, and the rules generated in step S1 are delivered to two key modules: the dynamic data analysis unit and the data storage preprocessing unit.

[0042] Step S2 details the transmission process of the rule instructions. In step S2, the mapping rules generated in step S1 are delivered to two key modules in the system: the dynamic data parsing unit and the data storage preprocessing unit. In subsequent steps, the rule instructions are transmitted through the internal bus or dedicated interface of the system to ensure that the rules can accurately and timely reach the target modules. After receiving the mapping rules, the dynamic data parsing unit processes the header of the security data according to these rules. The parsing unit adds specific mapping flag bits to the header of the security data. These flag bits are used to identify the mapping information of the data, such as storage address, data type, etc. The addition of flag bits can significantly improve the efficiency of the subsequent data distribution unit. By quickly identifying the flag bits in the data header, the distribution unit can accurately distribute the data to the designated storage unit, ensuring the accurate distribution and positioning of data in the storage unit. The data storage preprocessing unit receives the security data processed by the dynamic data parsing unit.

[0043] The data distribution unit further processes the security data according to the mapping rules. These rules guide the distribution unit on how to accurately distribute the data to the global storage unit and the local storage unit.

[0044] S3, as shown in Figure 1 The system begins to receive security information from the market and performs data protocol analysis. That is, when the security data starts to be sent, the security data from different trading markets (such as the main board and the growth board of the Shenzhen Stock Exchange) is transmitted to the data preprocessing unit in the form of streaming data through the Ethernet unit. Since the format and source of security data may vary, the data preprocessing unit first analyzes the received security data packet header according to the pre-stored template. This analysis process ensures that the system can understand and identify security data of different formats, thereby making correct judgments for subsequent data processing.

[0045] During the analysis process, the data preprocessing unit filters out unnecessary plate data and network information unrelated to security data. Through this filtering mechanism, the system can ensure that only data related to the current application is processed, thereby improving processing efficiency and reducing unnecessary resource consumption. Specifically, the data preprocessing unit automatically identifies and removes irrelevant data fields according to the mapping rules and templates, making the subsequent data processing process more focused and efficient.

[0046] S4, as shown in Figure 1As shown, the dynamic data unit and the pre-processing unit will allocate storage data for the data from step S3, that is, the dynamic data parsing unit will perform flag bit mapping on the arriving securities data according to the mapping rules from the dynamic configuration unit. Specifically, the dynamic data parsing unit will add a flag bit to the header of the securities data to identify the mapping information of the data, thereby ensuring the accurate positioning and classification of the data during the storage process. These flag bit mappings not only facilitate subsequent data access, but also provide more efficient data routing information. The data storage pre-processing module selects the appropriate storage unit for data transmission based on these header flag bits. In specific operations, the module will first use the first three digits of the securities code as an index to select the corresponding local storage unit according to the mapping rules. Through this process, the system can accurately locate a specific storage area, providing flexibility for data allocation.

[0047] Next, the data storage preprocessing module locates the corresponding continuous storage address based on the last three digits of the security data. At this point, the data is written to the selected storage location according to specified rules, ensuring correct storage in the local or global storage location. To ensure efficient system operation, the module avoids bus conflicts when selecting storage locations and ensures that the bus is idle, thereby optimizing data transmission efficiency and system resource utilization.

[0048] S5, such as Figure 1 As shown, the storage unit stores data from step S4 and supports interface expansion, such as PCIe and further calculations by the index calculation unit. This means that the local and global storage units during the storage process not only support internal data transmission and processing but can also be accessed and displayed via the PCIe interface. This design offers a high degree of freedom and configurability, enabling the storage solution to flexibly adapt to different types of securities data and providing strong support for subsequent data calculations and services.

[0049] Specifically, local and global storage units communicate with the host computer or other computing units via the PCIe interface, ensuring fast access and dynamic calculation of index data. This design's high degree of freedom enables the system to efficiently receive index data from various sources while flexibly storing, updating, and querying it, meeting the real-time and highly concurrent processing requirements of index calculations. More importantly, this flexible storage solution supports complex computational tasks in index calculation scenarios, such as multi-dimensional weight calculations, real-time index analysis, and historical data correlation processing, significantly improving computational accuracy and system efficiency.

[0050] In summary, the present application proposes a DPU high degree of freedom dynamic storage scheme for index calculation, which can realize the functions of fast storage, dynamic analysis, data distribution, etc. of securities files. Through the coordination of the PCIe dynamic configuration module, the cooperation relationship between the Ethernet receiving module, the dynamic random storage unit and the register secondary storage module, and the dynamic mapping logic, the present scheme significantly improves the storage efficiency and real-time processing capability of the system, and meets the needs of high performance, high expansibility and high security in the financial field.

[0051] The present application compares the traditional DPU-based data storage scheme and the DPU-based multi-level mapping scheme, and both systems store the same securities data. The space utilization rate difference of the storage data size of the two is as follows:

[0052]

[0053] Through testing, it is found that the storage scheme based on two-level mapping consumes less storage space than the traditional storage scheme. First, due to the adoption of the two-level mapping mechanism, the system can dynamically allocate storage addresses according to the characteristics of the data, avoiding unnecessary space waste. This accurate storage mapping greatly improves the utilization rate of storage space, thereby reducing the storage demand and significantly reducing the overall storage consumption of the system. This is particularly important in large-scale data storage and processing scenarios, which can effectively reduce hardware costs and improve storage efficiency.

[0054] Secondly, with its unique distribution preprocessing module, the present scheme can achieve lower latency during data storage. In the traditional storage scheme, data storage and processing often require multiple steps, which may increase latency. However, the present scheme optimizes the data transmission and storage process through real-time analysis, filtering and marking of data, reducing redundant processing steps and thus reducing latency. This enables the system to respond to external requests more quickly, improving overall system performance, especially in applications requiring high-frequency data updates and real-time data display, providing a better experience.

[0055] In addition, the present scheme utilizes the parallel processing capability of DPU to achieve higher parallelism. Traditional storage schemes usually rely on serial processing, and data processing and storage are limited by processing capability. The two-level mapping scheme, however, performs multiple data operations in parallel, thereby significantly improving data processing throughput and speed. This high parallelism not only improves the overall performance of the system, but also enables the system to maintain high efficiency in large-scale data processing, adapting to more complex computing tasks and diverse data flow requirements.

[0056] The present application has the following advantages:

[0057] 1. Dynamic mapping rule generation and configuration Based on the PCIe dynamic configuration module, the high-efficiency first-level and second-level storage mapping is realized through the segmented mapping mechanism of the securities code (the first three digits of the three-level code). It allows continuous address mapping for discontinuous securities codes, improving storage management efficiency and compatibility.

[0058] 2. Dynamic data analysis and flag mechanism The dynamic data analysis module adds a flag bit in the securities data header according to the mapping rule, which is used to optimize subsequent data distribution and storage path selection. The flag bit design ensures the real-time and scalability of data processing.

[0059] 3. Multi-level storage unit cooperative scheduling Design local storage unit and global storage unit, flexible scheduling and access through PCIe bus. Utilize multi-level storage unit to improve storage and access efficiency during low-latency index calculation.

[0060] 4. Combination of dynamic configuration and real-time storage Seamless collaboration of dynamic configuration unit, data analysis unit and storage distribution unit to ensure low-latency processing and fast access of securities data.

[0061] 5. Optimization design for high real-time financial services Provide real-time update, index calculation and low-latency display functions for securities data, adapt to the demand scenario of low-latency index calculation.

Claims

1. A DPU high-degree-of-freedom dynamic storage method for index calculation, the DPU high-degree-of-freedom dynamic storage method for index calculation being used in a DPU high-degree-of-freedom dynamic storage system for index calculation, and the DPU high-degree-of-freedom dynamic storage system for index calculation comprising: a PCIe dynamic configuration module, a dynamic configuration unit, a dynamic data analysis unit, a data storage preprocessing unit, a data preprocessing unit, a local storage unit, and a global storage unit; and the DPU high-degree-of-freedom dynamic storage method for index calculation comprising: step S1: writing a security code to the system through a PCIe interface, and transmitting configuration data in a preset order in sequence, and processing the security code by the PCIe dynamic configuration module according to a mapping rule, wherein the mapping rule comprises performing first-level mapping on the first n bits of the security code as an index to select a corresponding storage unit, and performing second-level mapping on the last m bits of the security code as a basis to select a corresponding continuous storage address, wherein n and m are positive integers; step S2: generating the mapping rule by the dynamic configuration unit, and transmitting the mapping rule to the dynamic data analysis unit and the data storage preprocessing unit; step S3: transmitting security data from different trading markets to the data preprocessing unit in the form of streaming data through an Ethernet unit, and matching a security data packet according to a pre-stored template by the data preprocessing unit to identify the format and source of the security data packet, and analyzing the header of the received security data packet; step S4: adding a mapping flag bit to the header of the security data according to the mapping rule by the dynamic data analysis unit, wherein the mapping flag bit is used to identify the mapping information of the security data, and selecting a storage unit for data transmission according to the mapping flag bit and the mapping rule by the data storage preprocessing module; and step S5: storing the security data by the local storage unit and the global storage unit, and step S5 comprises: implementing the global storage unit by using a register, and constructing the local storage unit by using a dynamic random access memory, and storing the security data in the global storage unit if the security information corresponding to the security data is global information, and storing the security data in the local storage unit if the security information corresponding to the security data is local information. The data preprocessing unit, the dynamic data analysis unit, and the data storage preprocessing unit are connected in sequence, the PCIe dynamic configuration module is connected to the data storage preprocessing unit and the dynamic data analysis unit respectively, and the data storage preprocessing unit is connected to the local storage unit and the global storage unit respectively. The DPU high-degree-of-freedom dynamic storage system for index calculation further comprises an ETH unit connected to the data preprocessing unit, and the ETH unit is used to receive a security information data stream from the market. The data storage preprocessing module comprises a data distribution unit, and the data distribution unit is used to distribute the security data to the local storage unit and the global storage unit. Step S3 comprises: analyzing the header of the received security data packet, which comprises: identifying and removing irrelevant data fields according to the mapping rule and the template. ​ ​ ​ ​ 2. The DPU high degree of freedom dynamic storage method for index calculation oriented computing according to claim 1, characterized in that, ​ 3. The DPU high degree of freedom dynamic storage method for index calculation oriented computing according to claim 2, characterized in that, ​ 4. The DPU high degree of freedom dynamic storage method for index calculation oriented computing according to claim 3, characterized in that, ​ 5. The DPU high degree of freedom dynamic storage method for index calculation oriented computing according to claim 1, wherein, ​

Images