Parameter processing method and parameter processing device

Abstract

The invention relates to a parameter processing method and a parameter processing device. The parameter processing method comprises the following steps: a data management step: presetting a configuration file which at least comprises parameter table information, a data reading step: reading data from a data source according to the parameter table information in the configuration file; and a data loading step: loading the read data to a target medium, and independently storing the variable-length field in the read data in the target medium. According to the present invention, it is possible to provide a parameter processing method and a parameter processing device capable of achieving both high efficiency and high maintainability.

Description

Technical Field

[0001] The present invention relates to data processing technology, and in particular to a parameter processing method and a parameter processing device for general parameters. Background Art

[0002] As a core component of the financial industry, the efficient and stable operation of the clearing system is crucial to ensuring the security and accuracy of capital flows. The parameter cache refresh and query mechanism is an indispensable part of the clearing system. It is responsible for ensuring that the system can obtain and process various parameter information in a timely and accurate manner to support the smooth operation of clearing business.

[0003] The technical problems currently existing in this technical field include the following aspects:

[0004] (1) Storage data structure adjustment is complex

[0005] Currently, parameter caching and query functions are mostly developed based on specific table structures. This means that whenever a parameter table and its fields are added or adjusted, the storage data structure needs to be adjusted accordingly and the code needs to be recompiled. Since the storage structure may be used in multiple systems, each adjustment needs to maintain consistency, which greatly increases the complexity and workload of the adjustment.

[0006] (2) Personalized processing is difficult to maintain

[0007] In the existing solution, the processing logic of each parameter table is designed to be highly personalized, resulting in poor code reusability. When the number or fields of parameter tables are adjusted, developers need to deeply understand and modify multiple different code snippets. This not only increases the complexity of maintenance but also increases the risk of errors. At the same time, the modification, compilation, and joint debugging process of multiple codes can easily lead to increased labor costs.

[0008] (3) The query mechanism is not flexible enough

[0009] The current parameter query mechanism is designed for individual tables. Different tables require different structure names, and only one table can be queried at a time. This design cannot achieve automatic multi-level cascade search, making it complex and inconvenient for developers to use.

[0010] (4) Insufficient utilization of shared memory resources

[0011] The parameter table is loaded into the shared memory according to the longest storable length of data, which results in the shared memory resources not being fully utilized. In the clearing system, shared memory is a limited resource, and its efficient use is of great significance to the performance and stability of the system. Summary of the Invention

[0012] In order to solve the above problems in the prior art, the present invention provides a parameter processing method and a parameter processing device that can achieve both high efficiency and high maintainability.

[0013] A parameter processing method according to one aspect of the present invention includes:

[0014] A data management step, presetting a configuration file, wherein the configuration file at least includes parameter table information;

[0015] a data reading step, reading data from a data source according to the parameter table information; and

[0016] The data loading step is to load the read data into a target medium, wherein the variable-length fields in the read data are stored separately in the target medium.

[0017] Optionally, in the data reading step, a structure configuration file is generated according to the parameter table information.

[0018] The structure configuration file includes: field name, field type, field length, and whether it is a primary key.

[0019] Optionally, in the data management step, the target medium is further set in the configuration file to be shared memory or Redis.

[0020] In the data loading step, the read data is loaded into the shared memory or Redis according to the target medium set in the configuration file.

[0021] When data is loaded into a shared memory, the data is loaded into the shared memory of each application node.

[0022] Optionally, in the data management step, filtering rules are further predefined in the configuration file.

[0023] The filtering rules include one or more of the following:

[0024] Database-level filtering rules for reading data from the database serving as the data source; and

[0025] Memory-level filtering rules that read data from memory as a data source.

[0026] Optionally, in the data management step, a segmentation rule for segmenting data is further set in the configuration file.

[0027] Further comprising between the data reading step and the data loading step:

[0028] The segmentation step is to segment the data into multiple logical tables using hash processing according to the segmentation rules in the configuration file.

[0029] In the data loading step, the plurality of logical tables are loaded onto a target medium.

[0030] Optionally, the storing the variable-length fields in the read data separately in the target medium includes:

[0031] Store variable-length fields in a separate variable-length field space.

[0032] A long integer value is stored in the variable-length field space, and the long integer value includes: actual length information of the field and an offset of a storage location.

[0033] Optionally, in the data reading step, the actual length information of the field is read and the position of the data is located in combination with the offset of the storage position.

[0034] Optionally, in the data management step, query parameters are further defined in the configuration file.

[0035] The data management step further includes: constructing a query topology map according to the query parameters.

[0036] Optionally, further comprising:

[0037] The query step reads the query topology map, traverses the query topology map and returns the query result.

[0038] Optionally, constructing a query topology graph according to the query parameters includes:

[0039] Parsing the query parameters to obtain a source table, a target table, and a field association relationship between the source table and the target table; and

[0040] A query topology graph is constructed based on the source table, the target table, and the field association relationship.

[0041] Optionally, traversing the query topology graph includes:

[0042] Starting from the source table, traversing each node of the query topology graph according to the node field association relationship; and

[0043] During the traversal process, it continuously checks whether the current node is the target table. If the target table is reached, the traversal is stopped and the query result is fed back.

[0044] Optionally, in the query step, a breadth-first search algorithm is used for traversal.

[0045] Optionally, the query topology graph adopts a DAG topology graph.

[0046] A parameter processing device according to one aspect of the present invention includes:

[0047] A data management module pre-sets a configuration file, wherein the configuration file at least includes parameter table information;

[0048] a data reading module, reading data from a data source according to the parameter table information in the configuration file; and

[0049] The data loading module loads the read data into a target medium, wherein the variable-length fields in the read data are stored separately in the target medium.

[0050] Optionally, in the data reading module, a structure configuration file is generated according to the parameter table information.

[0051] The structure configuration file includes: field name, field type, field length, and whether it is a primary key.

[0052] Optionally, in the data management module, the target medium is further set in the configuration file to be shared memory or Redis.

[0053] In the data loading module, the read data is loaded into the shared memory or Redis according to the target medium set in the configuration file.

[0054] When data is loaded into a shared memory, the data is loaded into the shared memory of each application node.

[0055] Optionally, in the data management module, filtering rules are further predefined in the configuration file.

[0056] The filtering rules include one or more of the following:

[0057] Database-level filtering rules for reading data from the database serving as the data source; and

[0058] Memory-level filtering rules that read data from memory as a data source.

[0059] Optionally, in the data management module, a segmentation rule for segmenting data is further set in the configuration file.

[0060] Further included between the data reading module and the data loading module:

[0061] The segmentation module uses hash processing to segment the data into multiple logical tables according to the segmentation rules in the configuration file.

[0062] In the data loading module, the plurality of logical tables are loaded onto a target medium.

[0063] Optionally, the storing the variable-length fields in the read data separately in the target medium includes:

[0064] Store variable-length fields in a separate variable-length field space.

[0065] A long integer value is stored in the variable-length field space, and the long integer value includes: actual length information of the field and an offset of a storage location.

[0066] Optionally, in the data reading module, the actual length information of the field is read and the position of the data is located in combination with the offset of the storage position.

[0067] Optionally, in the data management module, query parameters are further defined in the configuration file and a query topology graph is constructed according to the query parameters.

[0068] Optionally, further comprising:

[0069] The query module reads the query topology map, traverses the query topology map and returns the query result.

[0070] Optionally, constructing a query topology graph according to the query parameters includes:

[0071] Parsing the query parameters to obtain a source table, a target table, and a field association relationship between the source table and the target table; and

[0072] A query topology graph is constructed based on the source table, the target table, and the field association relationship.

[0073] Optionally, traversing the query topology graph includes:

[0074] Starting from the source table, traversing each node of the query topology graph according to the node field association relationship; and

[0075] During the traversal process, it continuously checks whether the current node is the target table. If the target table is reached, the traversal is stopped and the query result is fed back.

[0076] Optionally, in the query step, a breadth-first search algorithm is used for traversal.

[0077] Optionally, the query topology graph adopts a DAG topology graph.

[0078] A computer-readable medium according to one aspect of the present invention stores a computer program, which implements the parameter processing method when executed by a processor.

[0079] A computer device according to one aspect of the present invention includes a storage module, a processor, and a computer program stored in the storage module and executable on the processor. When the processor executes the computer program, the parameter processing method is implemented.

[0080] A computer program product according to one aspect of the present invention comprises a computer program, which implements the parameter processing method when executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] The above and other objects and advantages of the present application will become more fully apparent from the following detailed description taken in conjunction with the accompanying drawings, in which the same or similar elements are denoted by the same reference numerals.

[0082] Figure 1 It is a flow chart of a parameter cache refresh and query processing method according to an embodiment of the present invention.

[0083] Figure 2 This is a schematic diagram showing a data recording structure according to one embodiment of the present invention.

[0084] Figure 3 The diagram is a diagram showing an example of implementing multi-table and multi-media queries through an internal routing strategy.

[0085] Figure 4 The figure shows a query topology diagram constructed by an example of the present invention.

[0086] Figure 5 This is an example of a data query process based on a query topology graph.

[0087] Figure 6 A schematic diagram showing a data writing process according to an example of the present invention.

[0088] Figure 7 It is a schematic diagram showing a data query process of an example of the present invention.

[0089] Figure 8 It is a structural diagram of a parameter processing device 100 according to an embodiment of the present invention. DETAILED DESCRIPTION

[0090] The following describes some of the various embodiments of the present invention, which are intended to provide a basic understanding of the present invention, but are not intended to identify the key or decisive elements of the present invention or to limit the scope of protection.

[0091] For the purpose of brevity and illustration, the principles of the present invention are described herein primarily with reference to exemplary embodiments thereof. However, those skilled in the art will readily recognize that the same principles are equally applicable to and may be implemented in all types of parameter processing methods and parameter processing devices, and that any such changes do not depart from the true spirit and scope of the present patent application.

[0092] Moreover, in the following description, reference is made to the accompanying drawings, which illustrate specific exemplary embodiments. Electrical, mechanical, logical, and structural changes may be made to these embodiments without departing from the spirit and scope of the present invention. In addition, although a feature of the present invention is disclosed in conjunction with only one of several embodiments, it may be desirable and / or advantageous to combine this feature with one or more other features of other embodiments as may be desired and / or advantageous for any given or identifiable function. Therefore, the following description should not be regarded in a limiting sense, and the scope of the present invention is defined by the appended claims and their equivalents.

[0093] Terms such as “having” and “including” indicate that in addition to the units (modules) and steps directly and clearly stated in the specification and claims, the technical solution of the present invention does not exclude the situation where it has other units (modules) and steps that are not directly or clearly stated.

[0094] First, some technical terms in the present invention are explained.

[0095] (1) Redis

[0096] Redis, short for Remote Dictionary Server, is a memory-based key-value store that provides high-performance data access services. It is commonly used in caching, message queues, and data storage. Redis supports a variety of data structures, such as strings, hashes, lists, sets, and ordered sets, and can persist data to disk to ensure data security.

[0097] (2) DAG topology

[0098] A DAG topology refers to a directed acyclic graph (DAG), which is a special directed graph structure in which each edge has a direction. Starting from any vertex in the graph and moving along a directed edge, it is impossible to return to that vertex. In other words, there are no cycles in a DAG.

[0099] (3) Shared Memory

[0100] Shared memory is a communication mechanism that allows multiple processes or threads to directly access the same physical memory area. It allows different processes or threads to share the same memory area, thereby achieving fast data exchange.

[0101] (4)BFS algorithm

[0102] The BFS algorithm is the Breadth-First Search algorithm, which is a graph search algorithm. Breadth-first search is a layer-by-layer scanning search strategy. It starts from a specified node in the graph (called the starting node or source node), first visits all adjacent nodes, and then visits the unvisited adjacent nodes of these adjacent nodes, and so on, until all nodes in the graph connected to the starting node have been visited.

[0103] (5) GB18030 code

[0104] GB18030 encoding is part of China's national mandatory standard GB18030 "Information Technology Chinese Coded Character Set", which is used to encode Chinese characters. It supports simplified and traditional Chinese characters, as well as Chinese punctuation marks, some Japanese and Korean characters, etc.

[0105] (6) Key-Value Pair

[0106] A key-value pair is a data representation method in which a key is a unique identifier used to find or access its associated value.

[0107] Next, the technical concept of the parameter processing method and parameter processing device of the present invention is briefly described.

[0108] The present invention provides an efficient universal cache model. By storing variable-length fields separately, the universal cache model can fully utilize the data space of shared memory and deconstruct and configure the memory, which is beneficial to the adjustment of shared memory table fields.

[0109] Furthermore, the present invention also provides a loading method based on a universal cache model, wherein the loading list of data fields is determined according to the configuration, the single record of shared memory and the total size of total memory are dynamically applied according to the configuration, and the loading fields are processed uniformly, which is flexible and efficient.

[0110] Furthermore, the present invention also provides a target model for implementing cascade query using a DAG topology graph, wherein application parameters only need to focus on input and output without having to worry about internal cascade details, thereby improving data query efficiency while optimizing development efficiency.

[0111] Furthermore, the present invention also provides a query method based on a universal cache model, wherein the complexity of the query function can be reduced by configuring the query primary key, query algorithm, and data storage location of the management table.

[0112] First, a parameter processing method according to an embodiment of the present invention will be described.

[0113] The parameter processing method according to one embodiment of the present invention mainly includes the following steps:

[0114] The data management step is to pre-set a configuration file, wherein the configuration file includes parameter table information, filtering rules, overwriting rules, segmentation rules and query parameters,

[0115] A data filtering step, filtering data according to the filtering rules in the configuration file;

[0116] Data coverage step, judging whether the filtered data needs to be covered according to the coverage rules;

[0117] a data encryption step of encrypting the data and writing the encrypted information into the configuration file;

[0118] A data segmentation step, using hash processing to segment the data into multiple logical tables according to the segmentation rules in the configuration file;

[0119] Data loading step, loading the segmented data onto the target medium;

[0120] A topology map construction step, constructing a query topology map according to the query parameters; and

[0121] The query step obtains data from the shared memory or Redis according to the configuration file and queries each table step by step according to the query topology until the final target table is reached and the query result is returned.

[0122] Among them, the data filtering step, the data covering step, the data encryption step and the data segmentation step are preferred steps.

[0123] Furthermore, when the target medium is set as shared memory in the configuration file, in the data loading step, the data is loaded into the shared memory and stored in the shared memory in the following storage method: the variable-length fields are stored separately and the storage location information is recorded in the shared memory.

[0124] The parameter table information includes: field name, field type, field length, and whether it is a primary key. The query parameters include: query primary key, query algorithm, and storage location information.

[0125] Next, a parameter cache refresh and query processing method according to an embodiment of the present invention is described.

[0126] Figure 1 It is a flow chart of a parameter cache refresh and query processing method according to an embodiment of the present invention.

[0127] like Figure 1 As shown, the parameter cache refresh and query processing method of one embodiment of the present invention includes the following steps:

[0128] S100: Obtain the table name according to the configuration file, and export the configuration file for generating the table structure;

[0129] S200: Obtain a table word list to be loaded into the cache from the configuration file, determine whether to use shared memory or Redis as a medium, and read the sorting fields required for sorting;

[0130] S300: Read the parameter library, obtain the source table, and write it into a shared memory block or Redis according to the configuration input segmentation;

[0131] S400: The query component can call the query primary key interface, and only needs to transmit the primary key interface table name and the primary key values ​​of each field in the query table;

[0132] S500: The query primary key automatically obtains the query result according to the configuration and feeds it back to the scheduling application.

[0133] Here, first, the contents related to data management in the present invention are described.

[0134] In the present invention, the shared memory data storage method is adjusted.

[0135] Table a shows the shared memory space storage structure in the prior art, and Table b shows the shared memory storage structure provided by the present invention.

[0136] Table a

[0137]

[0138] Table b

[0139]

[0140] Comparing Table b with Table a, we can see that in the prior art, field lengths are fixed, and memory space is allocated based on the maximum supported length of the field type in the database. Regardless of the actual data length, each field will occupy a predefined maximum length space. In the present invention, however, field lengths are variable. As an example, the actual length of the field is represented by the last four digits of the field value or other methods. This approach allows for flexible processing of data of varying lengths, avoiding wasted memory space. Therefore, the shared memory space storage structure of the present invention can achieve the following technical effects:

[0141] Avoiding waste of memory space, especially when processing a large amount of short data or data with large length differences, the shared memory space storage structure of the present invention can significantly improve memory usage efficiency; and

[0142] The shared memory space storage structure of the present invention can flexibly process data of different lengths without worrying about the data length exceeding the predefined maximum length limit, which makes the present invention more advantageous when processing complex data structures or dynamic data.

[0143] Figure 2 This is a schematic diagram showing a data recording structure according to one embodiment of the present invention.

[0144] Figure 2 The top of represents a record set, including "Record 1" to "Record N". Figure 2 The specific field composition of each record is shown below, that is, each record includes: 1 to N fixed-length fields and a variable-length field space, and the variable-length field space is used to store fields of variable length.

[0145] Here, suppose that fixed-length field total length is 100, and variable-length Chinese name field length is 60, in order to hold uncommon characters, use GB18030 encoding, in prior art, memory block space one record needs 100+60*4=340 bytes, but actual uncommon characters occupies 3 bytes and above less than 5%, therefore takes up space<=100+8+6040.05+60*2*0.95=234 bytes in the present invention, compared to the space that needs 340 bytes in the lower prior art, saving space is about (340-234) / 340=31.1%.Therefore, can effectively utilize memory space by shared memory space storage structure of the present invention, reduce unnecessary waste.

[0146] To achieve universal access to the table structure and parse the table structure through a configuration file, you can use a configuration file to define the table structure and then parse it into key-value pairs for querying and loading data into shared memory. The following is a possible implementation step, including parsing the configuration file and using a map to manage field information.

[0147] A common entry to the internal structure of the table structure is implemented through the configuration file. As an example, the configuration file format is as follows:

[0148] Field name: {type: field type, length: length, is_key: whether it is a primary key}

[0149] Here is an example:

[0150] Key:{type:int,length:8,is_pri_key:true}

[0151] Col1:{type:char,length:16,is_pri_key:false}

[0152] Among them, int represents an integer (Integer) and char represents a character (Character).

[0153] Parse the field name into the following key-value pairs:

[0154] Key: field name;

[0155] Value: field type, memory length, and starting address offset.

[0156] Before querying and loading shared memory tables, the above data is obtained. A map is used to store the starting address offset, field type, and length information for each field. Various data types are converted into a unified binary format and placed in a dedicated table cache block. When executing queries, the map can be used to quickly locate the location of each field.

[0157] Next, the technical content of cache loading of the present invention is described.

[0158] The present invention provides a reusable data reading solution that takes into account various data processing requirements, such as filtering rules, segmentation rules, overwriting rules, ascending and descending order settings, and data encryption, which are described in detail below.

[0159] (1) Filter rule settings

[0160] Database-level filtering rules: Allows users to configure SQL query conditions to extract and filter data directly at the database level. For example, you can configure conditions such as oper_in<>'D' or settle_dt=$(SET_DT_8+ / -N).

[0161] Variable substitution: supports variables such as SET_DT_8+ / -N. Before executing the query, these variables are replaced with the actual settlement date plus or minus N days.

[0162] Memory-level filtering rules: Filter in memory based on a combination of user-set conditions. For example, you can configure it to read only records that exist in a specified table, or exclude certain records.

[0163] (2) Segmentation rule settings

[0164] According to the fields of the configuration table, hash processing is performed on the fields that need to be split, and the data table is split into multiple logical tables. The split logical tables can be stored in shared memory or in distributed caches such as Redis to improve data access efficiency and scalability.

[0165] (3) Override rule settings

[0166] As an example, the overwriting rules include: when writing to Redis / shared memory, if sorting is by an incomplete primary key, the data overwriting rules are configurable, supporting first-read-first, last-read-last, and parallel existence.

[0167] (4) Ascending and descending order settings

[0168] Each field can be set in ascending and descending order.

[0169] (5) Data encryption settings

[0170] Encrypt the entire memory block, such as MD5 encryption, and perform decryption verification when reading to prevent data tampering.

[0171] The following is an example of a specific cache loading process.

[0172] The example of the cache loading process specifically includes the following steps:

[0173] Configure filter conditions: The user configures database-level filter conditions, including SQL query conditions and variable replacement rules.

[0174] Data extraction and filtering steps: According to the configured filtering conditions, complete the extraction and preliminary filtering of data at the database level;

[0175] Data segmentation step: Hash the fields that need to be segmented and divide the data table into multiple logical tables;

[0176] Memory-level filtering step: Load the segmented data into memory and perform further filtering based on the memory-level filtering conditions;

[0177] Overwrite strategy application steps: When writing to cache (such as Redis / shared memory), data overwrite is handled according to the configured overwrite strategy;

[0178] Application steps of ascending and descending order setting: Sort the data in the cache according to the ascending and descending order setting of the field;

[0179] Data encryption step: MD5 encryption is performed on the entire memory block to ensure the security of the data in the cache; and

[0180] Data access verification step: When reading cached data, perform decryption verification to ensure that the data has not been tampered with.

[0181] Next, the data query routing of the present invention is described.

[0182] Figure 3The diagram is a diagram showing an example of implementing multi-table and multi-media queries through an internal routing strategy.

[0183] like Figure 3 As shown, the present invention supports associated queries on multiple tables, such as Figure 3 Tables A, B, and C in the table have clear primary keys and association relationships. For example, the primary key of table A is pri_A and table A is stored in shared memory, the primary key of table B is pri_B and table B is stored in Redis memory, and the primary key of table C is pri_C and table C is stored in shared memory. Table B is associated with table A through column A.col, and table C is associated with table B through column B.col2.

[0184] In this way, by introducing the concepts of common structure types and query routing, queries in Redis memory and shared memory are processed according to unified standards, thus avoiding switching back and forth between data formats and data acquisition methods. The complex logical operations involved in various queries can be completed in a unified manner, reducing the complexity of coding.

[0185] The following is an example to illustrate this. Figure 4 This represents a query topology graph constructed by an example of the present invention. The relevant parameters for constructing the query topology graph are set as follows:

[0186] Source table S, query to obtain fields (s1, s2, s3)

[0187] Query parameters A(a1,a2,a3), B(b1,b2), C(c1,c2,c3), D(d1,d2), E(e1,e2,e3)

[0188] The query field requirements are as follows:

[0189] A.a1=S.s1,

[0190] B.b1=A.a2,C.c1=A.a3

[0191] D.d1=B.b2,

[0192] E.e1=D.d2,E.e2=C.c2

[0193] Target table T.d1 = E.e3.

[0194] Based on the above parameters, we construct Figure 4 The query topology diagram (ie, DAG topology diagram) shown in Figure 4 In the example shown, the query topology starts from the source table S, passes through a series of query parameters A, B, C, D, E, and finally reaches the target table T, forming a complete query process. Figure 4 The query topology diagram shown includes the following processes:

[0195] Step S1: Query and obtain fields (s1, s2, s3) from the query source table S. Use field s1 in the source table S to match field a1 in the query parameter A to obtain the data (a1, a2, a3) in table A. This step establishes a dependency relationship between table S and table A.

[0196] Step S2: Use field a2 in table A to match field b1 in query parameter B to obtain the data (b1, b2) in table B. This step establishes a dependency relationship between table A and table B.

[0197] Step S3: Use field a3 in table A to match field c1 in query parameter C to obtain the data (c1, c2, c3) in table C. This step establishes a dependency relationship between table A and table C.

[0198] Step S4: Use field b2 in table B to match field d1 in query parameter D to obtain the data (d1, d2) in table D. This step establishes a dependency relationship between table B and table D.

[0199] Step S5: Use field d2 in table D to match field e1 in query parameter E to obtain data (e1, e2, e3) in table E. This step establishes a dependency relationship between table D and table E.

[0200] Step S6: Use field c2 in table C to match field e2 in query parameter E. This step establishes a dependency relationship between table C and table E; and

[0201] Step S7: Use field e3 in table E to match field d1 in target table T. This step establishes a dependency relationship between table E and table T.

[0202] In summary, the present invention uses a DAG topology diagram to graphically display the data query path and its dependencies from the source table S to the target table T. Through the above process, a one-time query of multi-level data can be completed for the parameters according to the configuration and the obtained table name, thereby optimizing the query logic and improving the query efficiency.

[0203] Figure 5 This is an example of a data query process based on a query topology graph.

[0204] like Figure 5 As shown, the data query process based on the query topology graph specifically includes the following steps:

[0205] Step S11: Start;

[0206] Step S12: Initialize, read the configuration file, and build a query topology map;

[0207] Step S13: Query the table, topology routing label, field name, and target table and field name based on the input source: Determine the table, routing label, field name, and target table and field name to be queried based on the input parameters. This step is used to determine the query content and target and put the relevant tables into the queue to be queried, for example, (A, a1, S, s1.value, E, e3), and put A (such as table and field) into the queue to be queried. Here, the field names to be queried and the target tables where these fields are located are clearly specified. For example, (A.a1, S.s1.value, E.e3) indicates that field a1 comes from table A, field s1.value comes from table S, and field e3 comes from table E.

[0208] Step S14: traverse the queue using a BFS (breadth-first search) algorithm;

[0209] Step S15: Binary search the shared memory or query Redis based on the key to obtain the values ​​of fields such as A.a2 and A.a3, that is, perform a binary search in the shared memory or Redis to obtain the values ​​of fields a2, a3, etc. in table A;

[0210] Step S16: Determine whether the corresponding data is found. If so, proceed to step S17; otherwise, jump to step S21.

[0211] Step S17: Determine whether it is the target table E. If so, proceed to step S18; otherwise, jump to step S22.

[0212] Step S18: traverse the tables corresponding to all topology nodes executed by Table A and place these tables in the query queue. This step is to further traverse the tables corresponding to all related topology nodes after the query content has been determined, and place these tables in the query queue to ensure that all related tables are processed;

[0213] Step S19: Determine whether the queue is empty. If so, proceed to step S20; otherwise, return to step S14.

[0214] Step S20: Feedback routing configuration error, the route cannot find the target table;

[0215] Step S21: Feedback the table that cannot be queried and the error reason;

[0216] Step S22: Feedback the acquired field E.e3.value and feedback that the query is successful.

[0217] Based on the above process, the BFS algorithm can be used to query each table step by step until the target table is found. If the query encounters unreachable data or the target table, an error message is promptly reported. The topology routing diagram clearly displays the relationships between data parameters. Users only need to focus on the input parameters and the final target field value, without having to understand the details of the internal dependencies in between.

[0218] Next, a data writing process according to an example of the present invention will be described.

[0219] Figure 6 A schematic diagram showing a data writing process according to an example of the present invention.

[0220] like Figure 6 As shown, an example from data reading to writing specifically includes the following steps:

[0221] Step S31: Start; Step S32: Read the configuration to obtain the source table name, parse the data structure of each field in the table, cache the loading field list and the sorting field list, etc., build the query statement and filter data engine, data coverage rules, etc.;

[0222] Step S33: Connect to the parameter library according to the pre-set or configured database connection information, read the parameter table, and read the parameter data line by line;

[0223] Step S34: Determine whether to write according to the filtering rules. If yes, proceed to step S35:, otherwise jump to step S39;

[0224] Step S35: Determine whether overwriting is required according to the overwriting strategy (e.g., if there are identical sorting fields);

[0225] Step S36: encrypt the data information to prevent subsequent written information from being tampered with, and write the secret information into a unified table structure;

[0226] Step S37: Obtain the medium to which the data is written. If it is Redis, write it directly; otherwise, write it to a shared memory file and wait for subsequent unified loading.

[0227] Step S38: Load the shared memory file into the shared memory of each application node and proceed to step S40;

[0228] Step S39: Continue to the next item and return to step S33;

[0229] Step S40: End.

[0230] Among them, the "overwrite" in step S35 generally refers to the process of writing or updating data, when it is found that the newly read data conflicts or duplicates with the existing data, and how to handle the data is determined according to the overwrite strategy. As explained above, the overwrite strategy is a set of predefined rules used to guide how to handle data conflicts or duplications during data writing or updating. These rules can be customized according to specific business needs and application scenarios. Among them, "same sort field" generally refers to the field used to uniquely identify or distinguish records in the data table. For example, if there are one or more fields in the data table used to determine the uniqueness of the record (such as the primary key or unique index field), these fields can be regarded as "sort fields". When the newly read data and the existing data have the same value in the sort field, it is necessary to determine whether the existing data needs to be overwritten according to the overwrite strategy. The purpose of this step is to ensure the accuracy and consistency of the data. By applying the overwrite strategy, data conflicts or duplications can be automatically handled during the data writing or updating process, thereby avoiding problems caused by data inconsistency.

[0231] Among them, in step S37, direct writing when writing to Redis can fully utilize the high performance and real-time advantages of Redis, while unified loading when writing to shared memory can optimize memory management, improve batch processing efficiency and ensure data synchronization consistency. This processing method can meet different needs according to the characteristics of different storage media.

[0232] Next, an exemplary data query process of the present invention (including query, verification and encryption processing) is described.

[0233] Figure 7 It is a schematic diagram showing a data query process of an example of the present invention.

[0234] like Figure 7 The process shown includes the following steps:

[0235] Step S41: Start;

[0236] Step S42: Read the configuration to obtain the source table name, parse the data structure of each field in the table, cache the loaded field list and the sorted field list, and build a map for quickly locating the field offset position;

[0237] Step S43: dispatching the query interface, passing the names of the tables to be queried and the sources of the query fields corresponding to the tables to be queried;

[0238] Step S44: Determine whether the table name and field name are correct based on the configuration of each parameter table. If they are correct, proceed to step S45. If they are incorrect, proceed to step S49.

[0239] Step S45: searching for the corresponding query path according to the acquired query information, and then querying the required parameter tables one by one to obtain the corresponding data information;

[0240] Step S46: parsing the queried data information, wherein encryption verification is performed on the encrypted information and the obtained data;

[0241] Step S47: Determine whether the decrypted and encrypted data information is consistent with the original information. If they are consistent, proceed to step S48; otherwise, jump to step S49;

[0242] Step S48: Return the query result of the query success and the parameter information of the obtained parameter data table;

[0243] Step S49: Return failure and provide the failure reason and the corresponding failure table name.

[0244] Among them, in step S43, scheduling the query interface means starting the query interface and preparing to execute the data query task. This step usually involves initializing the connections and resources required for the query, and then passing the name of the database table to be queried to this system, which means clearly specifying which tables to retrieve data from, and then further clarifying which data fields need to be obtained from each table.

[0245] Finally, a parameter processing device according to an embodiment of the present invention will be described.

[0246] Figure 8 It is a structural diagram of a parameter processing device 100 according to an embodiment of the present invention.

[0247] like Figure 8 As shown, a parameter processing device 100 according to an embodiment of the present invention includes:

[0248] The data management module 110 pre-sets a configuration file, wherein the configuration file includes parameter table information, filtering rules, overwriting rules, segmentation rules and query parameters,

[0249] A data filtering module 120 filters data according to the filtering rules in the configuration file;

[0250] The data covering module 130 determines whether the filtered data needs to be covered according to the covering rules;

[0251] A data encryption module 140 encrypts data and writes the encrypted information into the configuration file;

[0252] The data segmentation module 150 is configured to segment the data into multiple logical tables using hash processing according to the segmentation rules in the configuration file;

[0253] The data loading module 160 loads the segmented data into the target medium;

[0254] A topology map construction module 170 constructs a query topology map according to the query parameters; and

[0255] The query module 180 obtains data from the shared memory or Redis according to the configuration file and queries each table step by step according to the query topology until the final target table is reached and returns the query result.

[0256] As described above, the parameter processing method and processing device according to the present invention can bring the following technical effects:

[0257] By storing variable-length fields separately, the data space of shared memory is fully utilized, memory utilization is improved, and memory waste is reduced. In particular, when processing variable-length fields, memory space can be significantly saved.

[0258] By building a DAG topology, multi-table cascade query is implemented. Application parameters only need to focus on input and output, without having to worry about internal cascade details. This greatly optimizes development efficiency and improves data query efficiency.

[0259] By setting the query primary key, query algorithm, and data storage location in the configuration file, the complexity of the query function is reduced;

[0260] By using filtering, segmentation, and overwriting strategies in cache loading, the flexibility and efficiency of data processing can be improved, and configuration can be performed according to actual task requirements, reducing the need for manual code adjustments.

[0261] By maintaining the internal routing strategy of the query interface, automatic multi-table and multi-media queries are achieved, improving the efficiency and maintainability of the query.

[0262] The above are only specific embodiments of the present application, but the scope of protection of the present application is not limited thereto. Those skilled in the art can think of other feasible changes or replacements based on the technical scope disclosed in this application, and such changes or replacements are all included in the scope of protection of the present application. In the absence of conflict, the embodiments of the present application and the features in the embodiments can also be combined with each other. The scope of protection of the present application shall be based on the description of the claims.

Claims

1. A parameter processing method, characterized in that: include: A data management step, presetting a configuration file, wherein the configuration file at least includes parameter table information; a data reading step, reading data from a data source according to the parameter table information; and The data loading step is to load the read data into a target medium, wherein the variable-length fields in the read data are stored separately in the target medium.

2. The parameter processing method according to claim 1, wherein: In the data reading step, a structure configuration file is generated according to the parameter table information. The structure configuration file includes: field name, field type, field length, and whether it is a primary key.

3. The parameter processing method according to claim 1, wherein: In the data management step, the target medium is further set in the configuration file to be shared memory or Redis. In the data loading step, the read data is loaded into the shared memory or Redis according to the target medium set in the configuration file. When data is loaded into a shared memory, the data is loaded into the shared memory of each application node.

4. The parameter processing method according to claim 1, wherein: In the data management step, filtering rules are further predefined in the configuration file. The filtering rules include one or more of the following: Database-level filtering rules for reading data from the database serving as the data source; and Memory-level filtering rules that read data from memory as a data source.

5. The parameter processing method according to claim 1, wherein: In the data management step, a segmentation rule for segmenting data is further set in the configuration file. Further comprising between the data reading step and the data loading step: The segmentation step is to segment the data into multiple logical tables using hash processing according to the segmentation rules in the configuration file. In the data loading step, the plurality of logical tables are loaded onto a target medium.

6. The parameter processing method according to claim 1, wherein: The storing the variable-length fields in the read data separately in the target medium includes: Store variable-length fields in a separate variable-length field space. A long integer value is stored in the variable-length field space, and the long integer value includes: actual length information of the field and an offset of a storage location.

7. The parameter processing method according to claim 6, wherein: In the data reading step, the actual length information of the field is read and the position of the data is located in combination with the offset of the storage position.

8. The parameter processing method according to claim 7, wherein: In the data management step, query parameters are further defined in the configuration file. The data management step further includes: constructing a query topology map according to the query parameters.

9. The parameter processing method according to claim 8, wherein: Further including: The query step reads the query topology map, traverses the query topology map and returns the query result.

10. The parameter processing method according to claim 8, wherein: The constructing of the query topology graph according to the query parameters includes: Parsing the query parameters to obtain a source table, a target table, and a field association relationship between the source table and the target table; and A query topology graph is constructed based on the source table, the target table, and the field association relationship.

11. The parameter processing method according to claim 10, wherein: The traversing the query topology graph includes: Starting from the source table, traversing each node of the query topology graph according to the node field association relationship; and During the traversal process, it continuously checks whether the current node is the target table. If the target table is reached, the traversal is stopped and the query result is fed back.

12. The parameter processing method according to claim 11, wherein: In the query step, a breadth-first search algorithm is used for traversal.

13. The parameter processing method according to claim 12, wherein: The query topology graph adopts a DAG topology graph.

14. A parameter processing device, characterized in that: include: A data management module pre-sets a configuration file, wherein the configuration file at least includes parameter table information; a data reading module, reading data from a data source according to the parameter table information in the configuration file; and The data loading module loads the read data into a target medium, wherein the variable-length fields in the read data are stored separately in the target medium.

15. The parameter processing device according to claim 14, wherein: In the data reading module, a structure configuration file is generated according to the parameter table information. The structure configuration file includes: field name, field type, field length, and whether it is a primary key.

16. The parameter processing device according to claim 14, wherein: In the data management module, the target medium is further set in the configuration file to be shared memory or Redis. In the data loading module, the read data is loaded into the shared memory or Redis according to the target medium set in the configuration file. When data is loaded into a shared memory, the data is loaded into the shared memory of each application node.

17. The parameter processing device according to claim 14, wherein: In the data management module, filtering rules are further predefined in the configuration file. The filtering rules include one or more of the following: Database-level filtering rules for reading data from the database serving as the data source; and Memory-level filtering rules that read data from memory as a data source.

18. The parameter processing device according to claim 14, wherein: In the data management module, a segmentation rule for segmenting data is further set in the configuration file. Further included between the data reading module and the data loading module: The segmentation module uses hash processing to segment the data into multiple logical tables according to the segmentation rules in the configuration file. In the data loading module, the plurality of logical tables are loaded onto a target medium.

19. The parameter processing device according to claim 14, wherein: The storing the variable-length fields in the read data separately in the target medium includes: Store variable-length fields in a separate variable-length field space. A long integer value is stored in the variable-length field space, and the long integer value includes: actual length information of the field and an offset of a storage location.

20. The parameter processing device according to claim 19, wherein: In the data reading module, the actual length information of the field is read and the position of the data is located in combination with the offset of the storage position.

21. The parameter processing device according to claim 20, wherein: In the data management module, query parameters are further defined in the configuration file and a query topology graph is constructed according to the query parameters.

22. The parameter processing device according to claim 21, wherein: Further including: The query module reads the query topology map, traverses the query topology map and returns the query result.

23. The parameter processing device according to claim 22, wherein: The constructing of the query topology graph according to the query parameters includes: Parsing the query parameters to obtain a source table, a target table, and a field association relationship between the source table and the target table; and A query topology graph is constructed based on the source table, the target table, and the field association relationship.

24. The parameter processing device according to claim 23, wherein: The traversing the query topology graph includes: Starting from the source table, traversing each node of the query topology graph according to the node field association relationship; and During the traversal process, it continuously checks whether the current node is the target table. If the target table is reached, the traversal is stopped and the query result is fed back.

25. The parameter processing device according to claim 24, wherein: In the query step, a breadth-first search algorithm is used for traversal.

26. The parameter processing device according to claim 24, wherein: The query topology graph adopts a DAG topology graph.

27. A computer-readable medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the parameter processing method according to any one of claims 1 to 13 is implemented.

28. A computer device comprising a storage module, a processor, and a computer program stored in the storage module and executable on the processor, wherein: When the processor executes the computer program, the parameter processing method according to any one of claims 1 to 13 is implemented.

29. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the parameter processing method according to any one of claims 1 to 13 is implemented.

Images