A data table generation method based on intrusive container and code generation technology

Abstract

The invention discloses a data table generation method based on an intrusive container and code generation technology. The method comprises memory data table definition, golang template definition and golang template generation, wherein the memory data table definition comprises defining a memory data table data structure and defining a memory data table class, and the golang temple template definition comprises defining a memory data table element structure and defining a memory data table class template. The invention adopts an intrusive container as a memory data engine, combines an object pool, simplifies memory management, improves execution efficiency, and can avoid the overhead and risk brought by shared memory, and supports nested variable-length data types. Through code generation technology, multiple intrusive container types are defined for a memory data table, and multi-dimensional indexing of a single memory data table is realized.

Description

Technical Field

[0001] The invention relates to the technical field of data processing, in particular to a data table generation method based on an intrusive container and code generation technology. Background Art

[0002] The memory data table for the securities trading system is a component in the trading system that provides business reference data loading, storage, maintenance and indexing functions.

[0003] Code generation is a technology that generates standardized computer program code based on certain rules according to the input provided by the user. The main methods of code generation technology currently used include template-based generation, visual UI-based generation, code corpus-based generation, and AI-based generation. Among them, the template-based generation method is the most primitive, simplest, and currently the most widely used code generation method. It forms a template by extracting the common logic of the code, fills the template with relevant information according to the description of the metadata, and generates code for different businesses.

[0004] The generation method based on visual UI is also mainly aimed at non-R&D personnel. Developers develop various basic functional modules and provide these functional modules to users through visual programming. Users can freely combine and build related codes or programs.

[0005] The code corpus-based production method is mainly used in various IDE plug-ins. It analyzes the context information of the code written by the developer, predicts what the developer will enter, and generates and completes the code.

[0006] The AI-based generation method refers to the use of technologies such as computer vision and natural language processing to understand the relevant content drawn or written by designers and other non-research and development personnel, and generate corresponding code snippets. Its current application scope is relatively niche.

[0007] Golang template is a data-driven template for generating text output provided by the golang programming language. According to the defined template, various texts can be produced by filling in data.

[0008] The container here is a collection for storing element objects of the same type. Different organizational structures can be used for the organization of element objects, such as heap, stack, linked list, binary tree, hash table, etc.; for data storage, it can be heap memory, stack memory, shared memory, external memory, etc.

[0009] Non-intrusive containers generally do not need to modify the structure of the elements themselves. They store element objects by copying and cloning them. Intrusive containers do not store the element object data itself, but instead establish connections between element objects by adding additional attribute hooks based on the structure of the elements themselves, thereby realizing different organizational methods of element objects and avoiding the copying and cloning of element objects by the container, reducing the requirements for memory management and providing better performance.

[0010] Boost.Intrusives is an intrusive container library for C++ provided by the Boost library.

[0011] In the existing securities trading system, in order to facilitate other processes to access the trading data maintained during the operation of the trading system, a memory data engine based on shared memory is used, based on which logical codes of memory data tables are generated for different data models.

[0012] In the existing securities trading system, when shared memory is used to enable external programs to access the internal data of the trading system, shared memory does not provide a synchronization mechanism. Although other means can be used to synchronize processes, when multiple processes operate on the same data module in the shared memory at the same time, the efficiency is low, errors are prone to occur, and it is difficult to debug after the errors occur.

[0013] In the prior art, the shared memory uses an offset pointer, cannot use a general pointer and store C++ objects, and does not support nested variable-length data types.

[0014] In the prior art, a data structure in a shared memory is used as an API. When there are too many users, it is difficult to optimize the data structure in order to maintain backward compatibility. Summary of the invention

[0015] The purpose of the present invention is to solve the deficiencies of the prior art and provide a data table generation method based on an intrusive container and code generation technology.

[0016] In order to achieve the above purpose, a data table generation method based on an intrusive container and code generation technology is designed, wherein the memory data table T includes a field F of the memory data table and an index I established on the memory data table, and the index I has index fields, orderliness and uniqueness attributes. The method includes a memory data table definition method, and the specific steps are as follows:

[0017] S1. Define the data structure of the memory data table. 1 Fields and indexes, define the memory data table element structure T 1 Data;

[0018] S11. According to the memory data table T 1 Fields, define element fields;

[0019] S12. According to the memory data table T 1 The index of defines the intrusive container hook type;

[0020] S2. Define the memory data table class;

[0021] S21. According to the memory data table T 1 The hook type defined in the memory data table element structure defines the intrusive container HookOpt type;

[0022] S22. According to the memory data table T 1 The index of the index key structure composed of index fields and the KeyOfValue structure used to obtain the index key structure are defined. The KeyOfValue structure implements the conversion from the memory database data to the index key structure through overloading.

[0023] S23. According to the memory data table T 1 Define the intrusive container type, and specify the hook option and index field for the intrusive container type;

[0024] S24. Define the storage type for storing memory data table data, using a standard container and storing it in the form of a unique pointer;

[0025] S25. Unordered intrusive containers organize data through hashing, so hash buckets need to be defined;

[0026] S26. Define a constructor of the memory data table type, and initialize the defined intrusive container type in the constructor, that is, the hash bucket used by the unordered intrusive container type;

[0027] S27. Define the data adding interface of the memory data table, and add the memory data table T 1 The data is encapsulated and stored in datas with a unique pointer, and the data index information is added through the interface of the intrusive container defined according to the index;

[0028] S28. Define the data deletion interface of the memory data table;

[0029] S29. According to the memory data table T 1 The index of the memory data table is defined to search for data interface, search for data from the intrusive container, return the data object pointer if the data exists, and return a null pointer if the data does not exist.

[0030] The present invention also includes the following preferred technical solutions.

[0031] Furthermore, a template definition method is also included, and the specific steps of the definition method are as follows:

[0032] D1. Define the memory data table element structure template;

[0033] D11. Define the name of the memory data table element structure according to the memory data table name;

[0034] D12. Traverse the memory data table field combination, define the field variable according to the field type and field name, and convert the field name into underline format;

[0035] D13. Traverse the memory data table index set and define the hook variable for the intrusive container according to the index field set and index name;

[0036] D2. Define the memory data table class template;

[0037] D21. Define the memory data table class name according to the memory data table name;

[0038] D22. Traverse the memory data table index set, define the index key structure according to the index field set, where the constructor of the index key structure needs to traverse the index field set;

[0039] D23. Traverse the memory data table index set and define the HookOpt of the intrusive container according to the index name and index order;

[0040] D24. Traverse the memory data table index set, and define the type and variable of the intrusive container according to the index name and index order;

[0041] D25. Define the memory for storing the memory data table data;

[0042] D26. Define the constructor of the memory data table class;

[0043] D27. Define the destructor of the memory data table class, traverse the memory data table index set, and clean up the intrusive container and the memory data table data storage according to the index name;

[0044] D28. Define the data adding interface of the memory data table class, traverse the memory data table index set, add the data into the intrusive container according to the index name, and encapsulate and store it in the data storage device through a unique pointer;

[0045] D29. Define the data deletion interface of the memory data table class, traverse the memory data table index set, and delete the data from all intrusive containers and data storage devices according to the index name;

[0046] D210. Traverse the memory data table index set, define the corresponding memory data table class query data interface for all intrusive containers, query data from the intrusive container according to the corresponding index key structure, return the data pointer if it exists, otherwise return a null pointer.

[0047] Furthermore, it also includes a template generation method, which is to pass the attributes of the memory data table T into the template through the provided interface, so as to generate a correct and usable memory data table structure and memory data table class defined in the memory data table definition.

[0048] Compared with the prior art, the present invention has the following advantages:

[0049] 1. Use an intrusive container as the memory data engine, combined with an object pool, to simplify memory management, improve execution efficiency, and avoid the overhead and risks brought by shared memory.

[0050] 2. In the field of securities trading systems, intrusive containers are used as the basic data structure for carrying memory data tables, so that memory data tables support the storage of C++ objects and nested variable-length data types.

[0051] 3. Through code generation technology, multiple intrusive container types are defined for memory data tables to achieve multi-dimensional indexing of a single memory data table. DETAILED DESCRIPTION

[0052] The present invention is further described below, and the structure and principle of the present invention are very clear to people in the field. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The specific embodiments are described as follows:

[0054] 1. Symbol definition:

[0055] The memory data table is represented by T; the field of the memory data table is represented by F, and the field F has attributes such as type F.type; the index established on the memory data table is represented by I, and the index I has attributes such as index field I.fields, orderliness I.ordered, and uniqueness I.unique.

[0056] 2. Memory data table definition:

[0057] Assume that the data table T 1 There are four fields, namely T 1 .fields={F 1 , F 2 , F 3 , F 4}, where F 1 .type=int, F2 .type=int, F 3 .type=string,F 4 .type=string.

[0058] Data Sheet T 1 The indexes are,T 1 .indexes={I 1 , I 1_2}. According to field F 1 Constructed an ordered unique index I 1 , that is I 1 .fields={F 1}、I 1 .ordered = true, I 1 .unique=true; according to field F 2 , F 3 Constructed an unordered unique index I 1_2 , that is I 1_2 .fields={F 1 , F 2}、I 1_2 .ordered = false, I 1_2 .unique=true. The way to define a memory data table using the intrusive container provided by the Boost.Intrusive library is as follows.

[0059] (1) Define the data structure of the memory data table:

[0060] According to T 1 The fields and indexes define the memory data table element structure T1Data;

[0061] Step 1: According to T 1 Fields, define element fields;

[0062] Step 2: According to T 1 Define the intrusive container hook type for the index. Use "set_member_hook" for ordered indexes and "unordered_set_member_hook" for unordered indexes.

[0063] The final defined T1Data structure is as follows:

[0064]

[0065]

[0066] (2) Define the memory data table class:

[0067] Memory Data Table T 1 The corresponding class name is T1, and the definition steps are as follows.

[0068] Step 1: According to T 1 The hook type defined in the memory data table element structure defines the intrusive container HookOpt type;

[0069] using OrderedUniqueIndexF1HookOpt=intrusive::member_hook<

[0070] T1Data,

[0071] intrusive::set_member_hook<>,

[0072] &T1Data::ordered_unique_index_f1_hook>;

[0073] using UnorderedUniqueIndexF1F2HookOpt=intrusive::member_hook<

[0074] T1Data,

[0075] intrusive::unordered_set_member_hook<>,

[0076] &T1Data::unordered_unique_index_f2_f3_hook>;

[0077] Step 2: According to T 1 The index defines an index key structure consisting of index fields and a KeyOfValue structure used to obtain the index key structure. The KeyOfValue structure implements the conversion from memory database data to index key structure by overloading "operator()". 1 .field={F 1}, so the definition is given by F 1 The index key structure is named F1Key, and the KeyOfValue structure is named F1KeyOfValue; 1_2 .fields={F 1 , F 2}, so the definition is given by F 2 and F 3 The index key structure is named F2F3Key, and the KeyOfValue structure is named F2F3KeyOfValue;

[0078]

[0079] Step 3: According to T 1 Define the intrusive container type, and specify the hookoption and index field for the intrusive container type. 1 The index corresponds to OrderedUniqueIndexF1Container, I 1_2 The index corresponds to UnorderedUniqueIndexF1F2Container, and their variables ordered_unique_index_f1_container and unordered_unique_index_f1_f2_container are defined respectively;

[0080] using OrderedUniqueF1IndexContainer=intrusive::rbtree<

[0081] TData,

[0082] OrderedUniqueF1IndexHookOpt,

[0083] intrusive::key_of_value <f1keyofvalue>>;

[0084] using UnorderedUniqueF2F3IndexContainer=intrusive::unordered_set<

[0085] TData,

[0086] UnorderedUniqueF2F3IndexHookOpt,

[0087] intrusive::key_of_value <f2f3keyofvalue>>

[0088] OrderedUniqueIndexF1Container ordered_unique_index_f1_container;

[0089] UnorderedUniqueIndexF1F2Container unordered_unique_index_f1_f2_container;

[0090] Step 4: Define the storage type for storing the memory data table data, using the standard container std::unordered_set, and store it in the form of a unique pointer. Define the data storage variable datas of the memory data table T1;

[0091] std::unordered_set <std::unique_ptr <t1data>>datas;

[0092] Step 5: The unordered intrusive container organizes data through hash, so hash buckets need to be defined, that is, define hash buckets unordered_unique_index_f1_f2_container_buckets for UnorderedUniqueIndexF1F2Container

[0093] std::vector<UnorderedUniqueIndexF1F2Container::bucket_type> unordered_unique_index_f1_f2_container_buckets;

[0094] Step 6: Define the constructor of the memory data table type, and initialize the defined intrusive container type in the constructor, that is, the hash bucket used by the unordered intrusive container type. Define the memory data table T 1 The constructor T1 of the class initializes the hash bucket unordered_unique_index_f1_f2_container_buckets used by UnorderedUniqueIndexF1F2Container, as well as the intrusive container variables ordered_unique_index_f1_container and unordered_unique_index_f1_f2_container;

[0095]

[0096]

[0097] Step 7: Define the data adding interface of the memory data table. Encapsulate the data of the memory data table T1 with a unique pointer and store it in datas, and add data index information through the interface of the intrusive container defined according to the index.

[0098]

[0099] Step 8: Define the data deletion interface of the memory data table. Remove data from the intrusive containers ordered_unique_index_f1_container and unordered_unique_index_f1_f2_container, and the data storage datas;

[0100]

[0101] Step 9: According to T 1 The index of the memory data table defines the data search interface. Search data from the intrusive container and return the data object pointer if the data exists, otherwise return a null pointer. 1 The interface corresponding to the index is FoundByF1Key, I 1_2 The interface corresponding to the index is FoundByF2F3Key;

[0102]

[0103]

[0104] 3.golang template definition:

[0105] In order to simplify the writing of memory data table code, the common logic of the memory data table T is extracted as metadata for code generation and defined as follows based on the symbol definition.

[0106] The memory data table T has the following attributes: memory data table name T.name, memory data table field set T.fields, memory data table index set T.indexes;

[0107] Field F has the following attributes: field name F.name, field type F.type;

[0108] Index I has the following properties: index name I.name, index field set I.fields, index order I.ordered, and index uniqueness I.unique.

[0109] According to the defined properties and the definition method of the memory database, the memory database code template is extracted as follows.

[0110] (1) Define the memory data table element structure:

[0111] Step 1: Define the name of the memory data table element structure according to T.name;

[0112] struct{{.T.Name}}Data

[0113] Step 2: Traverse T.fields, and according to F.type and defined field variables, F.name is converted to the underscore format of "aa_bb_cc" through ToUnderScoreCase;

[0114] {{range$f_i,$f:=.T.Fields}}

[0115] {{$f.Type}}{{ToUnderScoreCase$f.Name}}_;

[0116] {{end}}

[0117] Step 3: Traverse T.indexes and define the hook variable for the intrusive container according to I.ordered and I.name. When I.ordered = true, use intrusive::set_member_hook, otherwise use intrusive::unordered_set_member_hook;

[0118] {{range$i_i,$i:=.T.Indexes}}

[0119] {{if$i.Ordered}}intrusive::set_member_hook<>{{else}}intrusive::unordered_set_member_hook<>{{end}}{{ToUnderScoreCase$i.Name}}_hook_;

[0120] {{end}}

[0121] (2) Define the memory data table class template:

[0122] Step 1: Define the memory data table class name according to T.name;

[0123] class{{.T.Name}}Table

[0124] Step 2: Traverse T.indexes and define the KeyOfValue structure of the index key structure according to I.fields. The constructor of the index key structure needs to traverse I.fields.

[0125]

[0126] Step 3: Traverse T.indexes, define HookOpt of the intrusive container according to I.name and I.ordered, use set_member_hook when I.ordered = true, otherwise use unordered_set_member_hook;

[0127]

[0128]

[0129] Step 4: Traverse T.indexes, define the intrusive container type and variable according to I.name and I.ordered. Use rbtree when I.ordered = true, otherwise use unordered_set and define a hash bucket array.

[0130]

[0131] Step 5: Define the memory for storing the memory data table data, using std::vector;

[0132] std::unordered_set <std::unique_ptr<{{.T.Name}}Data> >datas_;

[0133] Step 6: Define the constructor of the memory data table class. Traverse T.indexes and initialize I.ordered = false using the unordered_set intrusive container and its hash bucket array;

[0134]

[0135]

[0136] Step 7: Define the destructor of the memory data table class, traverse T.indexes, and clean up the intrusive container and memory data table data storage according to I.name;

[0137]

[0138] Step 8: Define the data adding interface of the memory data table class. Traverse T.indexes, add data into the intrusive container according to I.name, and encapsulate and store it in the data storage device through a unique pointer;

[0139]

[0140] Step 9: Define the data deletion interface of the memory data table class. Traverse T.indexes and delete the data from all intrusive containers and data storage according to I.name;

[0141]

[0142]

[0143] Step 10: Traverse T.indexes, define the query data interface of the corresponding memory data table class for all intrusive containers, query data from the intrusive container according to the corresponding index key structure, return the data pointer if it exists, otherwise return a null pointer;

[0144]

[0145] 4. Golang template generation:

[0146] By passing the attributes of the memory data table T into the template through the interface provided by golang template, the correct and usable memory data table structure and memory data table class defined in the memory data table definition can be generated.

[0147] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A data table generation method based on intrusive container and code generation technology, Features The basic memory data table T includes the field F of the memory data table, the index I established on the memory data table, and the index The method has index fields, order and unique attributes, and includes a memory data table definition method, and the specific steps are as follows: S1. Define the data structure of the memory data table. 1 Fields and indexes, define the memory data table element structure T 1 Data; S11. According to the memory data table T 1 Fields, define element fields; S12. According to the memory data table T 1 The index of defines the intrusive container hook type; S2. Define the memory data table class; S21. According to the memory data table T 1 The hook type defined in the memory data table element structure defines the intrusive container HookOpt type; S22. According to the memory data table T 1 The index of the index key structure composed of index fields and the KeyOfValue structure used to obtain the index key structure are defined. The KeyOfValue structure implements the conversion from the memory database data to the index key structure through overloading. S23. According to the memory data table T 1 The index of the intrusive container type is defined, and the hookoption and index fields are specified for the intrusive container type; S24. Define the storage type for storing memory data table data, use standard containers, store in the form of unique pointers, and define the memory data table T 1 The data storage variable datas; S25. Unordered intrusive containers organize data through hash, so hash buckets need to be defined; S26. Define a constructor of the memory data table type, and initialize the defined intrusive container type in the constructor, that is, the hash bucket used by the unordered intrusive container type; S27. Define the data adding interface of the memory data table, and add the memory data table T 1 The data is encapsulated and stored in datas with a unique pointer, and the data index information is added through the interface of the intrusive container defined according to the index; S28. Define the memory data table delete data interface; S29. According to the memory data table T 1 The index of the memory data table is defined to find the data interface, and the data is found from the intrusive container. If the data exists, the data object pointer is returned, and if the data does not exist, a null pointer is returned.

2. A data table generation method based on an intrusive container and code generation technology as claimed in claim 1, Features The method also includes a template definition method, and the specific steps of the definition method are as follows: D1. Define the memory data table element structure template; D11. Define the name of the memory data table element structure according to the memory data table name; D12. Traverse the memory data table field set, define the field variable according to the field type and field name, and convert the field name into underline format; D13. Traverse the memory data table index set and define the hook variable for the intrusive container according to the index field set and index name; D2. Define the memory data table class template; D21. Define the memory data table class name according to the memory data table name; D22. Traverse the memory data table index set, define the index key structure according to the index field set, where the constructor of the index key structure needs to traverse the index field set; D23. Traverse the memory data table index set and define the HookOpt of the intrusive container according to the index name and index order; D24. Traverse the memory data table index set, and define the type and variable of the intrusive container according to the index name and index order; D25. Define the memory for storing the memory data table data; D26. Define the constructor of the memory data table class; D27. Define the destructor of the memory data table class, traverse the memory data table index set, and clean up the intrusive container and memory data table data storage according to the index name; D28. Define the data adding interface of the memory data table class, traverse the memory data table index set, add the data into the intrusive container according to the index name, and encapsulate and store it in the data storage device through a unique pointer; D29. Define the data deletion interface of the memory data table class, traverse the memory data table index set, and delete the data from all intrusive containers and data storage devices according to the index name; D210. Traverse the memory data table index set, define the corresponding memory data table class query data interface for all intrusive containers, query data from the intrusive container according to the corresponding index key structure, return the data pointer if it exists, otherwise return a null pointer.

3. A data table generation method based on an intrusive container and code generation technology as claimed in claim 1, Features The method also includes a template generation method, which is to pass the attributes of the basic memory data table T into the template through the provided interface, so as to generate a correct and usable memory data table structure and memory data table class defined in the memory data table definition.

Images