DPU high-degree-of-freedom dynamic storage method for index calculation

Abstract

The invention relates to the technical field of data storage, and provides an index calculation-oriented DPU high-degree-of-freedom dynamic storage method, which comprises the following steps of: writing security codes into a system through a PCIe interface, sequentially transmitting configuration data according to a preset sequence, and processing the security codes through a PCIe dynamic configuration module according to a mapping rule; generating the mapping rule through a dynamic configuration unit, and transmitting the mapping rule to a dynamic data analysis unit and a data storage preprocessing unit; the method comprises the following steps: matching a security data packet according to a pre-stored template through a data preprocessing unit, and analyzing a received security data packet header; a specific mapping flag bit is added to the head of security data through a dynamic data analysis unit according to a mapping rule, and a storage unit is selected for data transmission through a data storage preprocessing module according to the mapping flag bit and the mapping rule. According to the method, storage management is optimized by adopting a secondary mapping mechanism, and efficient positioning and access to the stored data are realized.

Description

Technical Field

[0001] The present application relates to the technical field of data storage, and in particular to a DPU high-freedom dynamic storage method for exponential calculation. Background Art

[0002] Traditional technical solutions usually rely on static rules for mapping when dealing with the storage and processing of index data. For example, specific fields of index data (such as index code or timestamp) are mapped to fixed storage areas (such as folders or memory blocks), and other fields are mapped to specific storage locations. This mapping rule is difficult to flexibly cope with dynamically changing data structures and complex query requirements in index calculations. Data is transmitted to storage devices (such as dynamic random access memory units or disks) through standard transmission interfaces (such as PCIe, Ethernet or other bus protocols).

[0003] Data preprocessing is mainly completed on the host computer, including data parsing, format conversion and screening. This process lacks dynamic optimization capabilities, especially in exponential computing scenarios, which require processing of large amounts of real-time updated data. Subsequently, the preprocessed data is stored in the target storage area according to fixed rules. This storage architecture is usually single-level (such as a global shared storage device or a partitioned storage device). When accessing data, fixed rules are relied on to calculate the storage address and retrieve data from the storage device. However, this method is inefficient when faced with high-concurrency and high-complexity exponential computing tasks, especially in scenarios where data is highly dynamic and cannot meet performance requirements. In addition, data display and analysis usually rely on host computer processing, further increasing time delays and system overhead.

[0004] When comparing the traditional storage solution with the currently proposed high-degree-of-freedom dynamic mapping storage solution, the traditional solution has the following main disadvantages and defects: 1. Insufficient mapping flexibility: Traditional solutions usually use a fixed allocation method to store securities codes, lacking a flexible dynamic mapping mechanism. Discontinuous securities codes require a large amount of redundant storage space, resulting in low storage resource utilization.

[0005] 2. Low data processing efficiency: Traditional solutions lack a unified data analysis mechanism. Securities data needs to be pre-processed by a host computer or external equipment before storage, resulting in additional processing delays and increased system complexity.

[0006] 3. Single storage structure and poor scalability: Most traditional storage solutions adopt a single-level storage architecture, lacking a hierarchical design of local storage and global storage, and are difficult to adapt to the storage needs of high concurrent access or data of different priorities. Summary of the invention

[0007] In order to help solve the above technical problems, the present application provides a DPU high-freedom dynamic storage method for exponential calculation, which adopts the following technical solutions: A DPU high-degree-of-freedom dynamic storage method for exponential calculation, the DPU high-degree-of-freedom dynamic storage method for exponential calculation is used for a DPU high-degree-of-freedom dynamic storage system for exponential calculation, wherein: The DPU high-degree-of-freedom dynamic storage system for exponential computing includes: PCIe dynamic configuration module, dynamic configuration unit, dynamic data parsing unit, data storage preprocessing unit, data preprocessing unit, local storage unit and global storage unit; The DPU high-degree-of-freedom dynamic storage method for exponential computing includes: Step S1: write the securities code into the system through the PCIe interface, and transmit the configuration data in sequence according to the preset order, and process the securities code according to the mapping rule through the PCIe dynamic configuration module, the mapping rule includes using the first n bits of the securities code as an index for primary mapping to select the corresponding storage unit, and using the last m bits of the securities code as a basis for secondary mapping to select the corresponding continuous storage address, where n and m are both positive integers; Step S2: Generate the mapping rule through the dynamic configuration unit, and transmit the mapping rule to the dynamic data parsing unit and the data storage preprocessing unit; Step S3: Securities data from different trading markets are transmitted to the data preprocessing unit in the form of streaming data through the Ethernet unit. The data preprocessing unit matches the securities data packet according to the pre-stored template to identify the format and source of the securities data packet, and parses the received securities data packet header; Step S4: adding a specific mapping flag to the header of the securities data according to the mapping rule by the dynamic data parsing unit to identify the mapping information of the securities data, and selecting a storage unit for data transmission according to the mapping flag and the mapping rule by the data storage preprocessing module; Step S5: The storage unit includes a local storage unit and a global storage unit, and the securities data is stored through the local storage unit and the global storage unit.

[0008] Preferably, the data preprocessing unit, the dynamic data parsing unit and the data storage preprocessing unit are connected in sequence, the PCIe dynamic configuration module is respectively connected to the data storage preprocessing unit and the dynamic data parsing unit, and the data storage preprocessing unit is respectively connected to the local storage unit and the global storage unit.

[0009] Preferably, the DPU high-freedom dynamic storage system for index calculation also includes an ETH unit connected to the data preprocessing unit, and the ETH unit is used to receive a securities information data stream from the market.

[0010] Preferably, the data storage preprocessing module includes a data distribution unit, and the data distribution unit is used to distribute the securities data to the local storage unit and the global storage unit.

[0011] Preferably, step S3 includes: parsing the received securities data packet header includes: identifying and removing irrelevant data fields according to mapping rules and templates.

[0012] Preferably, step S5 includes: the global storage unit is implemented using a register, and the local storage unit is constructed using a dynamic random access memory. If the securities information corresponding to the securities data is global information, the securities data is stored in the global storage unit; if the securities information corresponding to the securities data is local information, the securities data is stored in the local storage unit.

[0013] In summary, the high-freedom dynamic storage method for DPU for exponential calculation in the present application is coordinated through the PCIe dynamic configuration module, and the collaborative relationship between the Ethernet receiving module, the dynamic random storage unit and the register secondary storage module and the dynamic mapping logic, which significantly improves the storage efficiency and real-time processing capability, and meets the requirements of the financial field for high performance, high scalability and high security. In the high-freedom dynamic storage method for DPU for exponential calculation in the present application, the DPU uses its parallel processing capability and high-speed response characteristics to achieve efficient and precise control of the data storage process. The present application integrates multiple storage units and control modules, and uses a secondary mapping mechanism to optimize storage management, thereby achieving efficient positioning and access to stored data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A flowchart of an embodiment of a DPU high-freedom dynamic storage method for exponential calculation in the present application; Figure 2 A schematic block diagram of an embodiment of a DPU high-freedom dynamic storage system for exponential computing of the present application. DETAILED DESCRIPTION

[0015] The present application is further described below in conjunction with the accompanying drawings, and the structure and principle of the present application are very clear to people in the field. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0016] DPU (Data Processing Unit) is a new type of processor specifically designed to accelerate data processing tasks in data centers. DPU is a new generation of processors for data-centric computing. It integrates complete data center functions into a single chip, and together with CPU and GPU, it forms the three pillars of new computing. DPU focuses on performing basic layer application tasks, such as network protocol processing, switching routing calculations, encryption and decryption, and data compression, which are heavy and specific tasks, thereby reducing the burden on the CPU and GPU and improving the overall efficiency and performance of the data center.

[0017] Figure 1 This is a flow chart of an embodiment of a DPU high-freedom dynamic storage method for exponential calculation in the present application. Figure 2 A schematic block diagram of an embodiment of a DPU high-freedom dynamic storage system for exponential computing of the present application.

[0018] Combination Figure 1 and Figure 2 It can be understood that the DPU high-degree-of-freedom dynamic storage system for index calculation includes an ETH unit, a PCIe dynamic configuration module, a dynamic configuration unit, a dynamic data parsing unit, a data storage preprocessing unit, a data preprocessing unit, a local storage unit and a global storage unit. The ETH unit is connected to the data preprocessing unit, the data preprocessing unit, the dynamic data parsing unit and the data storage preprocessing unit are connected in sequence, the PCIe dynamic configuration module is connected to the data storage preprocessing unit and the dynamic data parsing unit respectively, and the data storage preprocessing unit is connected to the local storage unit and the global storage unit respectively. The ETH unit is used to receive the securities information data stream from the market, the PCIe interface is responsible for receiving the securities code from the host computer, and the rule configuration of the data storage preprocessing unit is realized through the PCIe dynamic configuration. The data preprocessing unit parses the securities information from the market, and the data storage preprocessing unit distributes and schedules the data according to the preset rules. Among them, the global storage unit is implemented by registers, the local storage unit is constructed using dynamic random access memory, and a customized secondary address mapping design is performed according to the data distribution of the securities information to improve data processing efficiency and storage flexibility. The system not only improves the parallelism of data processing, but also provides higher flexibility and configurability in data distribution and scheduling, meeting the needs of fast processing and efficient storage of securities information.

[0019] The DPU high-freedom dynamic storage method for index calculation includes step S1: writing the securities code to the system through the PCIe interface, and transmitting the configuration data in sequence according to the preset order, processing the securities code according to the mapping rule through the PCIe dynamic configuration module, the mapping rule includes using the first n bits of the securities code as the index for primary mapping to select the corresponding storage unit, and using the last m bits of the securities code as the basis for secondary mapping to select the corresponding continuous storage address, and n and m are both positive integers.

[0020] Step S2: Generate the mapping rule through the dynamic configuration unit, and transmit the mapping rule to the dynamic data parsing unit and the data storage preprocessing unit.

[0021] Step S3: Securities data from different trading markets are transmitted to the data preprocessing unit in the form of streaming data through the Ethernet unit, and the data preprocessing unit matches the securities data packet according to the pre-stored template to identify the format and source of the securities data packet, and parses the received securities data packet header. Step S3 also includes: Parsing the received securities data packet header includes: identifying and eliminating irrelevant data fields according to the mapping rules and templates.

[0022] Step S4: A specific mapping flag is added to the header of the securities data according to the mapping rules by the dynamic data parsing unit, and the mapping flag is used to identify the mapping information of the securities data. A storage unit is selected for data transmission according to the mapping flag and the mapping rules by the data storage preprocessing module.

[0023] Step S5: The storage unit includes a local storage unit and a global storage unit, and the securities data is stored in the local storage unit and the global storage unit. Step S5 also includes: the global storage unit is implemented by a register, and the local storage unit is constructed by a dynamic random access memory. If the securities information corresponding to the securities data is global information, the securities data is stored in the global storage unit, and if the securities information corresponding to the securities data is local information, the securities data is stored in the local storage unit.

[0024] Specifically, S1, Figure 1As shown in the figure, before the system starts, data pre-configuration is required, that is, the host computer types in the code to perform regular mapping on the PCIe dynamic configuration module and the dynamic configuration unit. Specifically, the host computer writes the securities code to the system through the PCIe interface and transmits the data in sequence according to the predetermined order. The PCIe dynamic configuration module processes these securities codes according to the pre-set mapping rules. The process first uses the first three digits of the securities code as an index to perform a first-level mapping to determine the first part of the storage address; then the last three digits are used as the basis for the second-level mapping to further accurately locate the second part of the storage address. The design of this process ensures the efficiency and accuracy of the mapping operation.

[0025] Among them, the securities codes configured by the host computer do not need to be entered in a continuous order. The dynamic configuration unit will convert discontinuous securities codes into continuous storage addresses according to the mapping rules to achieve flexible and efficient storage space management. This mapping mechanism not only optimizes the use of storage resources, but also effectively improves the system's response speed and data access efficiency. The innovation of this solution is that through the parallel processing capability of the DPU, this complex and dynamic secondary mapping operation can be efficiently implemented to adapt to diverse input requirements and ensure the efficiency and security of data access.

[0026] Step S1 inputs the securities code through the host computer and writes these codes into the system using the PCIe interface. The user types the securities code through the host computer (computer or dedicated control device). These securities codes are the basis for subsequent data processing. The host computer writes these securities codes into the system through the PCIe (Peripheral Component Interconnect Express) interface. The PCIe interface is a high-speed serial computer expansion bus standard that can support high-bandwidth data transmission and ensure that data can enter the system quickly and accurately. The system uses a two-level mapping mechanism to determine the storage address of the securities code.

[0027] Level 1 mapping: The first three digits of the security code are used as the index for level 1 mapping. This step determines the first part of the storage address. For example, if the security code is six digits, the first three digits may represent a classification or identification, which is used to preliminarily locate the storage area. Level 2 mapping: The last three digits of the security code are used as the basis for level 2 mapping to further accurately locate the second part of the storage address. This step ensures that each security code can be mapped to a unique storage address. These mapping rules are pre-set, and the system processes the security code according to these rules. Through the two-level mapping mechanism, the system can efficiently and accurately determine the storage address of each security code. This mechanism avoids the problems of storage address conflicts or duplications that may occur in traditional methods, and improves the utilization of storage resources. Using the parallel processing capabilities of the DPU (data processing unit), the system can efficiently handle a large number of security code mapping operations. This parallel processing capability enables the system to process a large number of data input requests in a short period of time, improving the overall performance of the system.

[0028] This process supports the conversion of discontinuous securities codes into continuous storage addresses. This means that users do not need to enter securities codes in a specific order. The system can automatically process these discontinuous codes and map them to continuous storage addresses. This flexibility improves the user experience and facilitates subsequent data processing and management.

[0029] In summary, step S1 inputs the securities code through the host computer and writes these codes into the system using the PCIe interface. Through the two-level mapping mechanism, the system can efficiently and accurately determine the storage address of each securities code. This mechanism not only improves the efficiency of data access, but also ensures the accuracy, flexibility and utilization of storage resources. At the same time, the process also supports the conversion of discontinuous securities codes into continuous storage addresses, further improving the user experience and system performance.

[0030] S2, such as Figure 1 As shown, rule instructions need to be transmitted to pass the rules generated in step S1 to two key modules: the dynamic data parsing unit and the data storage preprocessing unit.

[0031] Step S2 describes in detail the transmission process of the rule instructions. In step S2, the mapping rules generated in step S1 are passed to two key modules in the system: the dynamic data parsing unit and the data storage preprocessing unit. In the subsequent steps, the rule instructions are transmitted through the bus or dedicated interface inside the system to ensure that the rules can reach the target module accurately and in a timely manner. After receiving the mapping rules, the dynamic data parsing unit processes the header of the securities data according to these rules. The parsing unit adds specific mapping flags to the header of the securities data. These flags are used to identify the mapping information of the data, such as storage address, data type, etc. The addition of flags can significantly improve the efficiency of subsequent data distribution units. By quickly identifying the flags in the data header, the distribution unit can accurately distribute the data to the specified storage unit to ensure the accurate distribution and positioning of the data in the storage unit. The data storage preprocessing unit receives the securities data processed by the dynamic data parsing unit.

[0032] The data distribution unit further processes the securities data according to the mapping rules. These rules guide the distribution unit on how to accurately distribute the data to the global storage unit and the local storage unit.

[0033] S3, such as Figure 1 As shown, the securities information from the market is received and the data protocol is parsed. That is, when the securities data starts to be sent, the securities data from different trading markets (such as the Shenzhen Stock Exchange Main Board, ChiNext, etc.) will be transmitted to the data preprocessing unit in the form of streaming data through the Ethernet unit. Since the format and source of securities data may vary, the data preprocessing unit first parses the received securities data packet header according to the pre-stored template. This parsing process ensures that the system can understand and identify securities data in different formats, so as to make correct judgments for subsequent data processing.

[0034] During the parsing process, the data preprocessing unit will filter out unnecessary sector data and network information that is not related to securities data. Through this filtering mechanism, the system can ensure that only data related to the current application is processed, thereby improving processing efficiency and reducing unnecessary resource consumption. Specifically, the data preprocessing unit automatically identifies and removes irrelevant data fields based on mapping rules and templates, making the subsequent data processing process more focused and efficient.

[0035] S4, such as Figure 1As shown, the dynamic data unit and the preprocessing unit will allocate storage data for the data from step S3, that is, the dynamic data parsing unit will map the flag bits of the arriving securities data according to the mapping rules from the dynamic configuration unit. Specifically, the dynamic data parsing unit will add a flag bit to the header of the securities data to identify the mapping information of the data, thereby ensuring the accurate positioning and classification of the data during the storage process. These flag bit mappings not only facilitate subsequent data access, but also provide more efficient data routing information. The data storage preprocessing module selects the appropriate storage unit for data transmission based on these header flag bits. In specific operations, the module will first use the first three digits of the securities code as an index to select the corresponding local storage unit according to the mapping rules. Through this process, the system can accurately locate a specific storage area, providing flexibility for data allocation.

[0036] Next, the data storage preprocessing module finds the corresponding continuous storage address based on the last three digits of the securities data. At this point, the data will be written into the selected storage unit according to the specified rules to ensure that the data is correctly stored in the local storage unit or the global storage unit. In order to ensure the efficient operation of the system, the module will avoid bus conflicts when selecting storage units and ensure the idle state of the bus, thereby optimizing the efficiency of data transmission and the use of system resources.

[0037] S5, such as Figure 1 As shown, the storage unit will store data according to the data from step S4, and support interface expansion, such as PCIe and further calculation of the index calculation unit, that is, the local storage unit and the global storage unit in the storage process not only support internal data transmission and processing, but also can be accessed and displayed through the PCIe interface. This design has high freedom and high configurability, which enables the storage solution to flexibly adapt to different types of securities data and provide strong support for subsequent data calculation and services.

[0038] Specifically, the local storage unit and the global storage unit communicate with the host computer or other computing units through the PCIe interface to ensure fast access and dynamic calculation of index data. The high degree of freedom of this design enables the system to efficiently receive index data from different sources, while flexibly performing storage, update and query operations to meet the requirements of index calculation for real-time and high concurrency processing. More importantly, this flexible storage solution can support complex computing tasks in index calculation scenarios, such as multi-dimensional weight calculation, real-time updated index analysis and historical data association processing, thereby significantly improving calculation accuracy and system efficiency.

[0039] In summary, the present invention proposes a DPU high-freedom dynamic storage solution for index calculation, which can realize the functions of fast storage, dynamic analysis, data distribution, etc. of securities files. Through the coordination of the PCIe dynamic configuration module, the collaborative relationship between the Ethernet receiving module, the dynamic random storage unit and the register secondary storage module and the dynamic mapping logic, this solution significantly improves the storage efficiency and real-time processing capability of the system, meeting the requirements of the financial field for high performance, high scalability and high security.

[0040] The present invention compares the traditional data storage solution based on DPU and the multi-level mapping solution based on DPU. Both systems store the same securities data. The difference in space utilization of the storage data size between the two is as follows:

[0041] Through testing, we found that the storage solution based on two-level mapping consumes less storage space than the traditional storage solution. First, due to the use of the two-level mapping mechanism, the system can dynamically allocate storage addresses according to the characteristics of the data, avoiding unnecessary waste of space. This precise storage mapping greatly improves the utilization of storage space, thereby reducing storage requirements and significantly reducing the overall storage consumption of the system. This is particularly important in large-scale data storage and processing scenarios, which can effectively reduce hardware costs and improve storage efficiency.

[0042] Secondly, with its unique distribution preprocessing module, the solution can achieve lower latency when storing data. In traditional storage solutions, data storage and processing often require multiple steps, which may lead to increased latency. However, this solution optimizes the data transmission and storage process by real-time parsing, screening and marking of data, reducing unnecessary processing links and thus reducing latency. This enables the system to respond to external requests more quickly and improves overall system performance, especially in applications that require high-frequency data updates and real-time data display, providing a better experience.

[0043] In addition, this solution uses the parallel processing capabilities of the DPU to achieve higher parallelism. Traditional storage solutions usually rely on serial processing, and data processing and storage are limited by processing power. The two-level mapping solution uses parallel processing to execute multiple data operations in parallel, thereby greatly improving the throughput and speed of data processing. This high parallelism not only improves the overall performance of the system, but also enables the system to maintain a high efficiency when processing large-scale data, and adapt to more complex computing tasks and diverse data flow requirements.

[0044] This application has the following advantages: 1. Dynamic mapping rule generation and configuration Based on the PCIe dynamic configuration module, efficient primary and secondary storage mapping is achieved through the segmented mapping mechanism of the securities code (the first and last three digits of the three-level code). Continuous address mapping is allowed for discontinuous securities codes, improving storage management efficiency and compatibility.

[0045] 2. Dynamic data analysis and flag mechanism The dynamic data analysis module adds a flag to the securities data header according to the mapping rules to optimize subsequent data distribution and storage path selection. The flag design ensures the real-time and scalability of data processing.

[0046] 3. Multi-level storage unit collaborative scheduling Design local storage units and global storage units, and flexibly schedule and access them through the PCIe bus. Use multi-level storage units to improve storage and access efficiency during low-latency index calculation.

[0047] 4. Combining dynamic configuration with real-time storage The dynamic configuration unit, data analysis unit and storage distribution unit work together seamlessly to ensure low-latency processing and fast access to securities data.

[0048] 5. The optimized design for high-real-time financial services provides real-time update, index calculation and low-latency display functions of securities data, adapting to the demand scenarios of low-latency index calculation.

Claims

1. A DPU high-degree-of-freedom dynamic storage method for exponential calculation, the DPU high-degree-of-freedom dynamic storage method for exponential calculation is used for a DPU high-degree-of-freedom dynamic storage system for exponential calculation, characterized in that: The DPU high-degree-of-freedom dynamic storage system for exponential computing includes: PCIe dynamic configuration module, dynamic configuration unit, dynamic data parsing unit, data storage preprocessing unit, data preprocessing unit, local storage unit and global storage unit; The DPU high-degree-of-freedom dynamic storage method for exponential computing includes: Step S1: write the securities code into the system through the PCIe interface, and transmit the configuration data in sequence according to the preset order, and process the securities code according to the mapping rule through the PCIe dynamic configuration module, the mapping rule includes using the first n bits of the securities code as an index for primary mapping to select the corresponding storage unit, and using the last m bits of the securities code as a basis for secondary mapping to select the corresponding continuous storage address, where n and m are both positive integers; Step S2: Generate the mapping rule through the dynamic configuration unit, and transmit the mapping rule to the dynamic data parsing unit and the data storage preprocessing unit; Step S3: Securities data from different trading markets are transmitted to the data preprocessing unit in the form of streaming data through the Ethernet unit. The data preprocessing unit matches the securities data packet according to the pre-stored template to identify the format and source of the securities data packet, and parses the received securities data packet header; Step S4: adding a mapping flag to the header of the securities data according to the mapping rule by the dynamic data parsing unit, wherein the mapping flag is used to identify the mapping information of the securities data, and selecting a storage unit for data transmission according to the mapping flag and the mapping rule by the data storage preprocessing module; Step S5: The storage unit includes a local storage unit and a global storage unit, and the securities data is stored through the local storage unit and the global storage unit.

2. The DPU high-degree-of-freedom dynamic storage method for exponential calculation according to claim 1 is characterized in that: The data preprocessing unit, the dynamic data parsing unit and the data storage preprocessing unit are connected in sequence, the PCIe dynamic configuration module is respectively connected to the data storage preprocessing unit and the dynamic data parsing unit, and the data storage preprocessing unit is respectively connected to the local storage unit and the global storage unit.

3. The DPU high-freedom dynamic storage method for exponential calculation according to claim 2 is characterized in that: The DPU high-freedom dynamic storage system for index calculation also includes an ETH unit connected to the data preprocessing unit, and the ETH unit is used to receive securities information data streams from the market.

4. The DPU high-degree-of-freedom dynamic storage method for exponential calculation according to claim 3 is characterized in that: The data storage preprocessing module includes a data distribution unit, which is used to distribute the securities data to the local storage unit and the global storage unit.

5. The DPU high-degree-of-freedom dynamic storage method for exponential calculation according to claim 1 is characterized in that: Step S3 includes: parsing the received securities data packet header includes: identifying and removing irrelevant data fields according to mapping rules and templates.

6. The DPU high-degree-of-freedom dynamic storage method for exponential calculation according to claim 1, characterized in that: Step S5 includes: the global storage unit is implemented by registers, and the local storage unit is constructed by dynamic random access memory. If the securities information corresponding to the securities data is global information, the securities data is stored in the global storage unit; if the securities information corresponding to the securities data is local information, the securities data is stored in the local storage unit.

Images