Data flattening method based on transition type breadth-first traversal

Abstract

The invention discloses a data flattening method based on transition type breadth-first traversal, and relates to the technical field of computer software. Comprising the following steps: automatically generating and acquiring block configuration data of a current application scene, and converting multi-level nested data capable of being constructed into a tree structure into a plurality of block N-way trees; converting the plurality of block N-way trees into a plurality of N-way trees with specified flat levels through a transition type breadth-first traversal algorithm; and assembling the N-way trees of the specified flat hierarchy into flat data of the specified flat hierarchy. According to the method, the flattening method is packaged, the specified flat hierarchy configuration item is opened to be used by the user, the problem that a large number of codes need to be written during data flattening processing of the specified flat hierarchy in diversified scenes is solved, all data formats capable of being constructed into a tree structure are supported, meanwhile, the memory utilization efficiency and the processing speed are improved, and the user experience is improved. And the flexibility of configuration driving is realized.

Description

Technical Field

[0001] The present invention discloses a method, which relates to the technical field of computer software, and specifically is a data flattening method based on jump-style breadth-first traversal. Background Art

[0002] Currently, when performing data flattening, especially when processing common JSON format data, in order to meet the flattening requirements of a specified flattening level, for example, flattening data with a maximum nested level of 3 into data with a maximum nested level of 2, flattening data with a maximum nested level of 4 into data with a maximum nested level of 3, flattening data with a maximum nested level of 5 into data with a maximum nested level of 3, etc. If independent flattening processing logics are developed for each specified flattening level, the efficiency is low and the code complexity is high. For example, Chinese Patent with publication number CN118426748A discloses a JSON format data flattening processing method based on annotations, including how to split and reorganize multi-level nested JSON format data. During operation, this method parses JSON data through annotations, generates Bean object reflection to set attribute values, and then converts it into JSON data. This method can reduce development time and lower code complexity, but it only supports JSON format data, lacks universality; lacks generalization ability, and specific annotations and logics need to be written for the flattening processing of different multi-level nested data in different application scenarios; when processing large-scale or ultra-large data sets, it may face the problem of excessive resource consumption. The above problems seriously affect the stability of the system and the efficiency of agile iteration.

[0003] In order to solve the above existing problems, it is necessary to design a more efficient and flexible flattening processing method to meet the needs of different application scenarios, improve the efficiency of data flattening processing and the stability of the system. The characteristics that this method should possess are: automatically dividing into blocks for different application scenarios, automatically generating configuration tables for different application scenarios; freely specifying the data flattening of the flattening level; efficiently traversing and constructing flat data with low memory consumption. Summary of the Invention

[0004] Aiming at the problems of the prior art, the present invention provides a data flattening method based on jump-style breadth-first traversal, which solves the problem of writing a large amount of code when performing data flattening with freely specified flattening levels in diverse scenarios, supports all data formats that can be constructed into a tree structure, and at the same time improves the memory utilization efficiency and processing speed, and realizes the flexibility of configuration-driven.

[0005] The specific solution proposed by the present invention is:

[0006] A data flattening method based on jump-style breadth-first traversal, characterized in that a root node with a level of 0 is defined, and the levels of multi-level nested data start counting from the first level. The data flattening method includes the following steps:

[0007] Step 1: Automatically generate and obtain the chunk configuration data of the current application scenario, and convert the multi-level nested data that can be constructed into a tree structure into multiple chunk N-ary trees;

[0008] Step 2: Convert multiple chunk N-ary trees into multiple N-ary trees with specified flat levels through the jump-style breadth-first traversal algorithm;

[0009] Step 3: Assemble multiple N-ary trees with specified flat levels into a flat data with a specified flat level;

[0010] In the first step, it includes the data chunking methods of the configuration table, N-ary tree, and multi-level nested data. The configuration table in the first step includes the application scenario, the number of chunks m, the key of the chunk root node, the path of the chunk root node in the multi-level nested data, and the level of the chunk root node in the multi-level nested data.

[0011] The attributes of the nodes of the N-ary tree in the first step include node name, node data, node path, node level, parent node reference, and child node reference list.

[0012] Through the data chunking method of the multi-level nested data in the first step, determine whether there is a configuration table in the current application scenario:

[0013] If not, use the general large model to automatically generate m non-overlapping chunks, save and obtain the configuration data mapped to the current application scenario in the configuration table;

[0014] If it exists, obtain the configuration data mapped to the current application scenario in the configuration table;

[0015] According to the key of the chunk root node in the configuration table, recursively construct N-ary trees for all chunks in parallel, obtain m N-ary trees, and through the formula

[0016] Node level = level of the chunk root node in the multi-level nested data + level of the current traversed node in the N-ary tree

[0017] Calculate the node level of each node of the N-ary tree, and through the formula

[0018] Chunk maximum depth = max(node levels of all nodes of the N-ary tree)

[0019] Calculate the chunk maximum depth of each N-ary tree; for all nodes in the multi-level nested data that are not chunked, select nodes according to the minimum satisfaction condition level formula and construct 1 N-ary tree;

[0020] In the second step, it includes the configuration table of the first step, the m + 1 N-ary trees of the first step, and the jump-style breadth-first traversal algorithm.

[0021] Converting the m + 1 N-ary trees of the first step through the jump-style breadth-first traversal algorithm of the second step includes the following steps:

[0022] Obtain the m + 1 N-ary trees of the first step, perform jump-style breadth-first traversal in parallel, and obtain m + 1 N-ary trees at the specified flat level;

[0023] In the third step, it includes the configuration table of the first step, the m + 1 N-ary trees at the specified flat level of the second step, and the assembly method.

[0024] The assembly method performs depth-first traversal on the m + 1 N-ary trees at the specified flat level of the second step in parallel to obtain m + 1 flat data blocks at the specified flat level; assemble the m + 1 flat data blocks at the specified flat level to obtain the data at the specified flat level.

[0025] Furthermore, in the second step, the jump-style breadth-first traversal algorithm is an iterative process carried out in the following steps:

[0026] Step 1: Establish a virtual root node and a queue, and enqueue all nodes in the list of child nodes of the root node of the N-ary tree in the first step.

[0027] Step 2: Determine whether the queue is non-empty:

[0028] Step 21: If the queue is non-empty, dequeue the head node of the queue and proceed to Step 3;

[0029] Step 22: If the queue is empty, end the iterative process and obtain the N-ary tree at the specified flat level;

[0030] Step 3: Determine whether the maximum depth of the block where the current traversed node is located is greater than the specified flat level, and whether the node level of the current traversed node satisfies the jump formula:

[0031] Step 31: If the condition is not met, skip the jump and proceed to Step 2;

[0032] Step 32: If the condition is met, proceed to Step 4;

[0033] Step 4: Perform data flattening on the nodes of the N-ary tree, including the following steps:

[0034] Step 41: Add node path data to the node data of the current traversed node, and query whether there is a node belonging to the same parent node as the current traversed node among the child nodes of the virtual root node through the union-find set:

[0035] Step 411: If it exists, merge the node data of this node and the currently traversed node, and merge all the child node reference lists of this node and the currently traversed node;

[0036] Step 412: If it does not exist, add the currently traversed node to the child node reference list of the virtual root node;

[0037] Step 42: Through the parent node reference of the currently traversed node, obtain the parent node, and remove the currently traversed node from the child node list of the parent node;

[0038] Step 43: Determine whether the child node list of the currently traversed node is non-empty:

[0039] Step 431: If the child node list is non-empty, enqueue all the nodes in the child node list and proceed to Step 2;

[0040] Step 432: If the child node list is empty, proceed to Step 2.

[0041] Further, in Step 1 and Step 2, it is defined that:

[0042] The specified flat level for the current application scenario is n, and the number of chunks of the multi-level nested data generated by the large model is m; the set of root nodes of all chunks of the multi-level nested data is R, the set of levels of the chunk root nodes of all chunks of the multi-level nested data in the multi-level nested data is L, and the set of maximum depths of all chunks of the multi-level nested data is H. The relationship among the three is

[0043]

[0044] The set of nodes in the multi-level nested data that have not been chunked is W, and the set of nodes that satisfy the minimum conditions is Z;

[0045] The mathematical expression of the minimum satisfaction condition level formula:

[0046] The set of nodes Z that satisfy the minimum conditions, where the element j satisfies the condition: for each element W in the set W i , there exists j belonging to the set RM i and j is the minimum value that satisfies the condition (the remainder of j divided by n is equal to 0); where the set RM i , the element j of the set satisfies the condition: for each element R in the set R i , there exists an element L in the set L i and an element H in the set H i , such that the parent node of R i belongs to W i , and j is in the closed interval [max{1, L i - 1}, H i, within

[0047] The minimum satisfaction condition level formula is

[0048]

[0049] Among them, RM i The formula expression of

[0050]

[0051] The mathematical expression of the transition formula is:

[0052] S is the set of transition levels of the i-th block in the multi-level nested data, and the element j in it satisfies the condition: j belongs to the closed interval [L i , H i , and the remainder of (j - 1) divided by n is equal to 0,

[0053] Transition formula:

[0054] S = {j | (L i ≤ j ≤ H i ) and ((j - 1) mod n = 0)}.

[0055] The present invention also provides a data flattening device based on transitive breadth-first traversal, which is characterized in that a root node with a level of 0 is defined, and the levels of the multi-level nested data start counting from the first level, including the following modules:

[0056] An acquisition module, a flattening module and an assembly module,

[0057] The acquisition module automatically generates and acquires the block configuration data of the current application scenario, converts the multi-level nested data that can be constructed into a tree structure into multiple block N-ary trees, and executes step one. The flattening module converts the multiple block N-ary trees into multiple N-ary trees with specified flat levels through the transitive breadth-first traversal algorithm, and executes step two. The assembly module assembles the multiple N-ary trees with specified flat levels into a flat data with a specified flat level, and executes step three.

[0058] In the step one, it includes a configuration table, an N-ary tree, and a data block method for multi-level nested data. The configuration table in the step one includes the application scenario, the number of blocks m, the key of the block root node, the path of the block root node in the multi-level nested data, and the level of the block root node in the multi-level nested data.

[0059] The attributes of the nodes of the N-ary tree in the step one include node name, node data, node path, node level, parent node reference, and child node reference list.

[0060] Using the data chunking method for multi-level nested data in the first step, determine whether there is a configuration table in the current application scenario:

[0061] If not, use the general large model to automatically generate m non-overlapping chunks, save and obtain the configuration data mapped to the current application scenario in the configuration table;

[0062] If it exists, obtain the configuration data mapped to the current application scenario in the configuration table;

[0063] According to the key of the chunk root node in the configuration table, recursively construct N-ary trees for all chunks in parallel to obtain m N-ary trees. Through the formula

[0064] Node level = level of the chunk root node in the multi-level nested data + level of the current traversed node in the N-ary tree

[0065] Calculate the node level of each node of the N-ary tree. Through the formula

[0066] Chunk maximum depth = max(node levels of all nodes of the N-ary tree)

[0067] Calculate the chunk maximum depth of each N-ary tree; for all nodes in the multi-level nested data that are not chunked, select nodes according to the minimum satisfaction condition level formula and construct 1 N-ary tree;

[0068] In the second step, it includes the configuration table in the first step, the m + 1 N-ary trees in the first step, and the jump-style breadth-first traversal algorithm.

[0069] Convert the m + 1 N-ary trees in the first step through the jump-style breadth-first traversal algorithm in the second step, including the following steps:

[0070] Obtain the m + 1 N-ary trees in the first step, perform jump-style breadth-first traversal in parallel, and obtain m + 1 N-ary trees at the specified flat level;

[0071] In the third step, it includes the configuration table in the first step, the m + 1 N-ary trees at the specified flat level in the second step, and the assembly method.

[0072] The assembly method performs depth-first traversal on the m + 1 N-ary trees at the specified flat level in the second step in parallel to obtain m + 1 flat data chunks at the specified flat level; assemble the m + 1 flat data chunks at the specified flat level to obtain the data at the specified flat level.

[0073] Furthermore, in the second step of the flattening module in a data flattening device based on jump-style breadth-first traversal, the jump-style breadth-first traversal algorithm is an iterative process carried out in the following steps:

[0074] Step 1: Create a virtual root node and a queue, and enqueue all nodes in the list of child nodes of the root node of the N-ary tree in Step 1.

[0075] Step 2: Determine whether the queue is non-empty:

[0076] Step 21: If the queue is non-empty, dequeue the head node of the queue and proceed to Step 3;

[0077] Step 22: If the queue is empty, end the iterative process and obtain the N-ary tree at the specified flattened level;

[0078] Step 3: Determine whether the maximum depth of the block where the currently traversed node is located is greater than the specified flattened level, and whether the node level of the currently traversed node satisfies the jump formula;

[0079] Step 31: If the condition is not satisfied, skip the jump and proceed to Step 2;

[0080] Step 32: If the condition is satisfied, proceed to Step 4;

[0081] Step 4: Perform data flattening on the nodes of the N-ary tree, including the following steps:

[0082] Step 41: Add node path data to the node data of the currently traversed node, and query whether there is a node in the child nodes of the virtual root node that belongs to the same parent node as the currently traversed node through the union-find set:

[0083] Step 411: If there is, merge the node data of this node and the currently traversed node, and merge the reference lists of all child nodes of this node and the currently traversed node;

[0084] Step 412: If not, add the currently traversed node to the reference list of the child nodes of the virtual root node;

[0085] Step 42: Obtain the parent node through the reference of the parent node of the currently traversed node, and remove the currently traversed node from the list of child nodes of the parent node;

[0086] Step 43: Determine whether the list of child nodes of the currently traversed node is non-empty:

[0087] Step 431: If the list of child nodes is non-empty, enqueue all nodes in the list of child nodes and proceed to Step 2;

[0088] Step 432: If the list of child nodes is empty, proceed to Step 2.

[0089] Further, in Step 1 of the acquisition module and Step 2 of the flattening module in the data flattening device based on jump-style breadth-first traversal, it is defined that:

[0090] The current application scenario specifies that the flat level is n, and the number of chunks of the multi-level nested data generated by the large model is m; the set of root nodes of all chunks of the multi-level nested data is R, the set of levels of the chunk root nodes of all chunks of the multi-level nested data in the hierarchy of the multi-level nested data is L, and the set of maximum depths of all chunks of the multi-level nested data is H. The relationship among the three is

[0091]

[0092] The set of nodes in the multi-level nested data that are not chunked is W, and the set of nodes that minimally meet the conditions is Z;

[0093] The mathematical expression of the formula for the level that minimally meets the conditions:

[0094] For the set of nodes Z that minimally meet the conditions, the element j in it satisfies the condition: for each element w in the set W i , there exists j belonging to the set RM i such that j is the minimum value that satisfies the condition (the remainder of j divided by n is equal to 0); where the set RM i , the element j of the set satisfies the condition: for each element r in the set R i , there exists an element l in the set L i and an element h in the set H i such that the parent node of r i belongs to W i , and j is in the closed interval [max{1, l i - 1}, h i ;

[0095] The mathematical expression of the transition formula is:

[0096] S is the set of transition levels of the i-th chunk in the multi-level nested data, and the element j in it satisfies the condition: j belongs to the closed interval [l i , h i , and the remainder of j - 1 divided by n is equal to 0.

[0097] The advantages of the present invention are:

[0098] In the market and public literature, no solution for data flattening with a specified flat level and related patents have been found. The comprehensive solution proposed here fills the technical gap;

[0099] Through the implementation of the specified flat level configuration item and the introduction of the large model intelligent adaptation application scenario mechanism, data flattening with a freely specified flat level is achieved, and it can adapt to the demand changes in different application scenarios without modifying the code. Users only need to set the parameters of the specified flat level;

[0100] Performance improvement: By adopting the block, parallel, and leapfrog breadth-first traversal algorithms, the amount of data processed in a single instance is significantly reduced, and the processing speed and efficiency are significantly improved.

[0101] Reduced memory occupancy: Through block processing, dynamic tree reconstruction technology, and object sharing mechanism, the memory occupancy during the processing of large-scale and ultra-large datasets is effectively reduced.

[0102] Device scenario diversity: In a multi-device environment, the processing method is selected according to the device resource configuration. When there are multiple devices with limited resources, parallel processing is used to allocate small data blocks; for devices with fewer resources, serial block processing is adopted; high-performance devices can perform parallel processing or batch allocate large data to maximize resource utilization and improve flattening efficiency.

[0103] Strong universality: It can be applied to JSON, XML, YAML, CSV, custom data structures (such as Java objects, Dython dictionaries, etc.), and any data format that can be constructed into a tree structure; specific application scenarios include, but are not limited to, artificial intelligence machine learning, big data preprocessing, data integration, data visualization, configuration management, data warehouses, content management systems, and e-commerce. Description of the Drawings

[0104] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0105] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0106] Figure 2 It is a data transformation schematic diagram of the leapfrog breadth-first traversal algorithm of the data flattening method in the embodiment of the present invention;

[0107] Figure 3 It is a schematic diagram of the flattening result of converting JSON format data with a maximum hierarchical nesting of 6 layers into JSON format data with a specified flat hierarchy of 2;

[0108] Figure 4 It is another schematic diagram of the method flow of the data flattening method in the embodiment of the present invention;

[0109] Figure 5 It is a schematic diagram of an embodiment of the data flattening device in the embodiment of the present invention. Detailed Embodiments

[0110] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, so that those skilled in the art can better understand the present invention and be able to implement it. However, the embodiments described by referring to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be construed as a limitation of the present invention.

[0111] Embodiment 1

[0112] The present invention provides a data flattening method based on jump-style breadth-first traversal, defines a root node with level 0, the levels of multi-level nested data start counting from the 1st layer, the format of the data to be flattened is JSON format, specifies the flattening level as 2, referring to Figure 2 , the data flattening method includes the following steps:

[0113] Step 1: Automatically generate and obtain the block configuration data of the current application scenario, and convert the multi-level nested JSON data into multiple block N-ary trees;

[0114] Step 2: Convert the multiple block N-ary trees into multiple N-ary trees with the specified flattening level through the jump-style breadth-first traversal algorithm;

[0115] Step 3: Assemble the multiple N-ary trees with the specified flattening level into a flat data with the specified flattening level;

[0116] In the first step, it includes a data block method for the configuration table, N-ary tree, and multi-level nested data. The configuration table in the first step includes the application scenario, the number of blocks m, the key of the block root node, the path of the block root node in the multi-level nested data, and the level of the block root node in the multi-level nested data.

[0117] The attributes of the nodes of the N-ary tree in the first step include node name, node data, node path, node level, parent node reference, and child node reference list.

[0118] Through the data block method of the multi-level nested data in the first step, it is judged whether there is a configuration table in the current application scenario:

[0119] If not, referring to Figure 2 of 201, use DeepSeek to generate 3 non-overlapping blocks, save and obtain the configuration data mapped to the current application scenario in the configuration table;

[0120] If it exists, obtain the configuration data mapped to the current application scenario in the configuration table;

[0121] According to the key of the block root node in the configuration table, recursively construct N-ary trees for all blocks in parallel, obtain m N-ary trees, through the formula

[0122] Node level = level of the chunk root node in the multi-level nested data + level of the currently traversed node in the N-ary tree

[0123] Calculate the node level of each node in the N-ary tree through the formula

[0124] Chunk maximum depth = max(node levels of all nodes in the N-ary tree)

[0125] Calculate the chunk maximum depth of each N-ary tree; for all nodes in the multi-level nested JSON data that are not chunked, select nodes according to the minimum satisfaction condition level formula and construct 1 N-ary tree, refer to Figure 2 of 202;

[0126] In the second step, it includes the configuration table of the first step, the 4 N-ary trees of the first step, and the leapfrog breadth-first traversal algorithm

[0127] Convert the 4 N-ary trees of the first step through the leapfrog breadth-first traversal algorithm of the second step, including the following steps:

[0128] Obtain the 4 N-ary trees of the first step, perform leapfrog breadth-first traversal in parallel, and obtain 4 N-ary trees at the specified flat level;

[0129] In the third step, it includes the configuration table of the first step, the 4 N-ary trees at the specified flat level of the second step, and the assembly method

[0130] The assembly method performs a depth-first traversal on the 4 N-ary trees at the specified flat level of the second step in parallel to obtain 4 flat data chunks at the specified flat level; assemble the 4 flat data chunks at the specified flat level to obtain JSON data at the specified flat level.

[0131] Further, in the second step, refer to Figure 2 of 203 and 204, and the leapfrog breadth-first traversal algorithm is an iterative process carried out in the following steps:

[0132] Step 1: Establish a virtual root node and a queue, and enqueue all nodes in the list of child nodes of the root node of the N-ary tree in the first step

[0133] Step 2: Determine whether the queue is non-empty:

[0134] Step 21: If the queue is non-empty, dequeue the head node of the queue and proceed to Step 3;

[0135] Step 22: If the queue is empty, end the iterative process and obtain the N-ary tree at the specified flat level;

[0136] Step 3: Determine that the maximum depth of the block where the currently traversed node is located is greater than the specified flattening level, and the node level of the currently traversed node satisfies the transition formula:

[0137] Step 31: If the condition is not met, skip the transition and proceed to Step 2;

[0138] Step 32: If the condition is met, proceed to Step 4;

[0139] Step 4: Perform data flattening on the nodes of the N-ary tree, including the following steps:

[0140] Step 41: Add node path data to the node data of the currently traversed node. Query whether there is a node in the children of the virtual root node that belongs to the same parent node as the currently traversed node through the union-find set:

[0141] Step 411: If there is, merge the node data of this node and the currently traversed node, and merge all the child node reference lists of this node and the currently traversed node;

[0142] Step 412: If not, add the currently traversed node to the child node reference list of the virtual root node;

[0143] Step 42: Obtain the parent node through the parent node reference of the currently traversed node, and remove the currently traversed node from the child node list of the parent node;

[0144] Step 43: Determine whether the child node list of the currently traversed node is non-empty:

[0145] Step 431: If the child node list is non-empty, enqueue all the nodes in the child node list and proceed to Step 2;

[0146] Step 432: If the child node list is empty, proceed to Step 2.

[0147] The method of the present invention encapsulates the flattening method and opens a specified flattening level configuration item for users to use, solving the problem of writing a large amount of code for data flattening at the specified flattening level in diverse scenarios, supporting all data formats that can be constructed into a tree structure, and at the same time improving memory utilization efficiency and processing speed, realizing the flexibility of configuration-driven.

[0148] Example 2

[0149] Based on Example 1, the implementation process of another embodiment of the present invention that does not adopt the block strategy. The format of the data to be flattened is JSON format, the specified flattening level is 2, and the relationship before and after data flattening can be referred to Figure 3 and the process schematic can be referred to Figure 4 The steps can be referred to as follows:

[0150] Step S1: Convert multi-level nested JSON data into an N-ary tree with a specified flat level through a leapfrog breadth-first traversal algorithm;

[0151] Step S2: Assemble the N-ary tree with the specified flat level into a flat data with the specified flat level;

[0152] In step S1, recursively construct an N-ary tree for the multi-level nested JSON data to obtain the N-ary tree, and convert the obtained N-ary tree through the leapfrog breadth-first traversal algorithm. Execute the following steps:

[0153] Obtain the obtained N-ary tree, perform a leapfrog breadth-first traversal to obtain an N-ary tree with the specified flat level;

[0154] In step S2, perform a depth-first traversal on the N-ary tree with the specified flat level in step S1 to obtain JSON data with the specified flat level.

[0155] The present invention also provides a data flattening device based on leapfrog breadth-first traversal, which can be referred to Figure 5 , and includes the following modules:

[0156] An acquisition module, a flattening module, and an assembly module,

[0157] The acquisition module automatically generates and acquires the chunk configuration data of the current application scenario, converts the multi-level nested data that can be constructed into a tree structure into multiple chunk N-ary trees, and executes step one. The flattening module converts the multiple chunk N-ary trees into multiple N-ary trees with the specified flat level through the leapfrog breadth-first traversal algorithm, and executes step two. The assembly module assembles the multiple N-ary trees with the specified flat level into a flat data with the specified flat level, and executes step three.

[0158] In step one, it includes a data chunking method for a configuration table, an N-ary tree, and multi-level nested data. The configuration table in step one includes the application scenario, the number of chunks m, the key of the chunk root node, the path of the chunk root node in the multi-level nested data, and the level of the chunk root node in the multi-level nested data.

[0159] The attributes of the nodes of the N-ary tree in step one include node name, node data, node path, node level, parent node reference, and child node reference list.

[0160] Through the data chunking method of the multi-level nested data in step one, determine whether there is a configuration table in the current application scenario:

[0161] If not, use DeepSeek to generate m non-overlapping chunks, save and acquire the configuration data mapped to the current application scenario in the configuration table;

[0162] If it exists, obtain the configuration data mapped to the current application scenario in the configuration table;

[0163] According to the key of the chunk root node of the configuration table, recursively construct N-ary trees for all chunks in parallel, obtaining m N-ary trees. Through the formula

[0164] Node level = level of the chunk root node in the multi-level nested data + level of the currently traversed node in the N-ary tree

[0165] Calculate the node levels of each node in the N-ary tree through the formula

[0166] Maximum chunk depth = max(node levels of all nodes in the N-ary tree)

[0167] Calculate the maximum chunk depth of each N-ary tree; for all nodes in the multi-level nested data that are not chunked, select nodes according to the minimum satisfaction condition level formula and construct 1 N-ary tree;

[0168] In the second step, it includes the configuration table of the first step, the m + 1 N-ary trees of the first step, and the jump-style breadth-first traversal algorithm.

[0169] Convert the m + 1 N-ary trees of the first step through the jump-style breadth-first traversal algorithm of the second step, including the following steps:

[0170] Obtain the m + 1 N-ary trees of the first step, perform jump-style breadth-first traversal in parallel, and obtain m + 1 N-ary trees at the specified flat level;

[0171] In the third step, it includes the configuration table of the first step, the m + 1 N-ary trees at the specified flat level of the second step, and the assembly method.

[0172] The assembly method performs depth-first traversal on the m + 1 N-ary trees at the specified flat level of the second step in parallel to obtain m + 1 flat data chunks at the specified flat level; assemble the m + 1 flat data chunks at the specified flat level to obtain the data at the specified flat level.

[0173] Regarding the information interaction of the processor in the above device, the process of executing the conversion program, etc., since it is based on the same concept as the method embodiment of the present invention, the specific content can be referred to the description in the method embodiment of the present invention and will not be elaborated here.

[0174] Similarly, the device of the present invention encapsulates the flattening method, opens the specified flat level configuration item for users to use, solves the problem of writing a large amount of code when performing data flattening processing at the specified flat level in diverse scenarios, supports all data formats that can be constructed into a tree structure, and at the same time improves memory utilization efficiency and processing speed, realizing the flexibility of configuration-driven;

[0175] By extracting the specified flat level, it is compatible with the data flattening requirements of different specified flat levels in subsequent different application scenarios, without modifying the code.

[0176] It should be noted that not all steps and modules in the above processes and device structures are necessary, and some steps or modules can be ignored according to actual needs. The execution order of each step is not fixed and can be adjusted according to needs. The system structure described in the above embodiments can be a physical structure or a logical structure, that is, some modules may be implemented by the same physical entity, or some modules may be implemented by multiple physical entities separately, or some components in multiple independent devices may be jointly implemented.

[0177] The above embodiments are only preferred embodiments given to fully illustrate the present invention, and the protection scope of the present invention is not limited thereto. Equivalent substitutions or transformations made by those skilled in the art on the basis of the present invention are all within the protection scope of the present invention. The protection scope of the present invention is subject to the claims.

Claims

1. A data flattening method based on transition breadth-first traversal, characterized by defining A root node with level 0, the levels of multi-level nested data are counted from level 1, and the data flattening method includes the following steps: Step 1: Automatically generate and obtain the block configuration data of the current application scenario, and convert the multi-level nested data that can be constructed into a tree structure into multiple block N-ary trees; Step 2: convert multiple block N-ary trees into multiple N-ary trees of specified flat levels through a transition breadth-first traversal algorithm; Step 3: Assemble multiple N-ary trees of specified flat levels into flat data of a specified flat level; In the step 1, a data partitioning method including a configuration table, an N-ary tree, and multi-level nested data is provided. The configuration table of the step 1 includes an application scenario, the number of partitions m, a key of a partition root node, a path of a partition root node in multi-level nested data, and a level of a partition root node in multi-level nested data. The attributes of the nodes of the N-ary tree in step 1 include node name, node data, node path, node level, parent node reference and child node reference list. By using the data segmentation method of multi-level nested data in step 1, it is determined whether a configuration table exists in the current application scenario: If it does not exist, use the general large model to automatically generate m non-intersecting blocks, save and obtain the configuration data mapped by the current application scenario in the configuration table; If it exists, obtain the configuration data mapped by the current application scenario in the configuration table; According to the key of the block root node of the configuration table, recursively build N-ary trees for all blocks in parallel to obtain m N-ary trees. Node level = the level of the multi-level nested data of the block root node + the level of the current traversal node in the N-ary tree Calculate the node level of each node of the N-ary tree, through the formula Maximum block depth = max(node ​​level of all nodes in the N-ary tree) Calculate the maximum depth of each N-ary tree; for all unblocked nodes in the multi-level nested data, select nodes according to the minimum level formula that satisfies the conditions, and build an N-ary tree; The step 2 includes the configuration table of step 1, the m+1 N-ary trees of step 1, and the transition breadth-first traversal algorithm. The m+1 N-ary trees of step 1 are converted by the transition breadth-first traversal algorithm of step 2, comprising the following steps: Obtain the m+1 N-ary trees of step 1, and perform transition-type breadth-first traversal in parallel to obtain the m+1 N-ary trees of the specified flat level; The step 3 includes the configuration table of the step 1, the m+1 N-ary trees of the specified flat level of the step 2, and the assembly method. The assembly method performs depth-first traversal in parallel on the m+1 N-ary trees of the specified flat level in step 2 to obtain m+1 flat data blocks of the specified flat level; assembles the m+1 flat data blocks of the specified flat level to obtain data of the specified flat level.

2. The data flattening method based on transition breadth-first traversal according to claim 1 is characterized in that In step 2, the transition breadth-first traversal algorithm is an iterative process performed in the following steps: Step 1: Create a virtual root node and a queue, and put all nodes in the child node list of the root node of the N-ary tree in step 1 into the queue. Step 2: Determine whether the queue is not empty: Step 21: If the queue is not empty, the head node of the queue is dequeued and step 3 is performed; Step 22: If the queue is empty, the iteration process ends and an N-ary tree of a specified flat level is obtained; Step 3: Determine whether the maximum depth of the block where the current traversal node is located is greater than the specified flat level, and the node level of the current traversal node satisfies the transition formula: Step 31: If the condition is not met, the transition is skipped and step 2 is performed; Step 32: If the conditions are met, proceed to step 4; Step 4: The nodes of the N-ary tree are flattened, including the following steps: Step 41: Add node path data to the node data of the currently traversed node, and query whether there is a node with the same parent node as the currently traversed node among the child nodes of the virtual root node through a union-find set: Step 411: If it exists, merge the node data of the node and the currently traversed node, and merge the reference lists of all child nodes of the node and the currently traversed node; Step 412: If it does not exist, add the current traversal node to the child node reference list of the virtual root node; Step 42: Obtain the parent node through the parent node reference of the current traversal node, and remove the current traversal node from the child node list of the parent node; Step 43: Determine whether the child node list of the currently traversed node is not empty: Step 431: If the child node list is not empty, all nodes in the child node list are queued and proceed to step 2; Step 432: If the child node list is empty, proceed to step 2.

3. A data flattening method based on transition-type breadth-first traversal according to claim 1 or 2, characterized in that In the steps 1 and 2, it is defined as: The current application scenario specifies that the flat level is n, and the number of blocks of the multi-level nested data generated by the large model is m; the root node set of all blocks of the multi-level nested data is R, the set of block root nodes of all blocks of the multi-level nested data at the level of the multi-level nested data is L, and the maximum block depth set of all blocks of the multi-level nested data is H. The relationship between the three is as follows: The set of unblocked nodes in the multi-level nested data is W, and the minimum set of nodes that meet the conditions is Z; Mathematical expression of the minimum satisfying condition level formula: The minimum set of nodes Z that satisfies the condition, where element j satisfies the condition: for each element W in the set W i , there exists j that belongs to the set RM i , so that j is the minimum value that satisfies the condition (the remainder of j divided by n is equal to 0); where the set RM i , the element j of the set satisfies the condition: for every element R in the set R i , there is an element L in the set L i and the elements H in the set H i , so that R i The parent node belongs to W i , and j is in the closed interval [max{1, L i -1},H i ]Inside, The minimum satisfying condition level formula is: Among them, RM i The formula is expressed as The mathematical expression of the transition formula is: S is the set of transition levels of the ith block in the multi-level nested data, where element j satisfies the condition: j belongs to the closed interval [L i ,H i ], and the remainder of j-1 divided by n is equal to 0, Transition formula: S={j|(L i ≤j≤H i )and((j-1)mod n=0)}。 4. A data flattening device based on transitional breadth-first traversal, characterized in that Define a root node with level 0. The levels of multi-level nested data are counted starting from level 1, including the following modules: Get modules, flatten modules, and assemble modules, The acquisition module automatically generates and obtains the block configuration data of the current application scenario, converts the multi-level nested data that can be constructed into a tree structure into multiple block N-ary trees, and executes step one. The flattening module converts the multiple block N-ary trees into multiple N-ary trees of specified flat levels through a transition breadth-first traversal algorithm, and executes step two. The assembly module assembles multiple N-ary trees of specified flat levels into flat data of a specified flat level, and executes step three. In the step 1, a data partitioning method including a configuration table, an N-ary tree, and multi-level nested data is provided. The configuration table of the step 1 includes an application scenario, the number of partitions m, a key of a partition root node, a path of a partition root node in multi-level nested data, and a level of a partition root node in multi-level nested data. The attributes of the nodes of the N-ary tree in step 1 include node name, node data, node path, node level, parent node reference and child node reference list. By using the data segmentation method of multi-level nested data in step 1, it is determined whether a configuration table exists in the current application scenario: If it does not exist, use the general large model to automatically generate m non-intersecting blocks, save and obtain the configuration data mapped by the current application scenario in the configuration table; If it exists, obtain the configuration data mapped by the current application scenario in the configuration table; According to the key of the block root node of the configuration table, recursively build N-ary trees for all blocks in parallel to obtain m N-ary trees. Node level = the level of the multi-level nested data of the block root node + the level of the current traversal node in the N-ary tree Calculate the node level of each node of the N-ary tree, through the formula Maximum block depth = max(node ​​level of all nodes in the N-ary tree) Calculate the maximum depth of each N-ary tree; for all unblocked nodes in the multi-level nested data, select nodes according to the minimum level formula that satisfies the conditions, and build an N-ary tree; The step 2 includes the configuration table of step 1, the m+1 N-ary trees of step 1, and the transition breadth-first traversal algorithm. The m+1 N-ary trees of step 1 are converted by the transition breadth-first traversal algorithm of step 2, comprising the following steps: Obtain the m+1 N-ary trees of step 1, and perform transition-type breadth-first traversal in parallel to obtain the m+1 N-ary trees of the specified flat level; The step 3 includes the configuration table of the step 1, the m+1 N-ary trees of the specified flat level of the step 2, and the assembly method. The assembly method performs depth-first traversal in parallel on the m+1 N-ary trees of the specified flat level in step 2 to obtain m+1 flat data blocks of the specified flat level; assembles the m+1 flat data blocks of the specified flat level to obtain data of the specified flat level.

5. The data flattening device based on transitional breadth-first traversal according to claim 4 is characterized in that In step 2 of the flattening module in the data flattening device based on transition-type breadth-first traversal, the transition-type breadth-first traversal algorithm is an iterative process performed in the following steps: Step 1: Create a virtual root node and a queue, and put all nodes in the child node list of the root node of the N-ary tree in step 1 into the queue. Step 2: Determine whether the queue is not empty: Step 21: If the queue is not empty, the head node of the queue is dequeued and step 3 is performed; Step 22: If the queue is empty, the iteration process ends and an N-ary tree of a specified flat level is obtained; Step 3: Determine whether the maximum depth of the block where the current traversal node is located is greater than the specified flat level, and the node level of the current traversal node satisfies the transition formula: Step 31: If the condition is not met, the transition is skipped and step 2 is performed; Step 32: If the conditions are met, proceed to step 4; Step 4: The nodes of the N-ary tree are flattened, including the following steps: Step 41: Add node path data to the node data of the currently traversed node, and query whether there is a node with the same parent node as the currently traversed node among the child nodes of the virtual root node through a union-find set: Step 411: If it exists, merge the node data of the node and the currently traversed node, and merge the reference lists of all child nodes of the node and the currently traversed node; Step 412: If it does not exist, add the current traversal node to the child node reference list of the virtual root node; Step 42: Obtain the parent node through the parent node reference of the current traversal node, and remove the current traversal node from the child node list of the parent node; Step 43: Determine whether the child node list of the currently traversed node is not empty: Step 431: If the child node list is not empty, all nodes in the child node list are queued and proceed to step 2; Step 432: If the child node list is empty, proceed to step 2.

6. A data flattening device based on transitional breadth-first traversal according to claim 4 or 5, characterized in that In the data flattening device based on transition-based breadth-first traversal, in step 1 of the acquisition module and step 2 of the flattening module, it is defined as follows: The current application scenario specifies that the flat level is n, and the number of blocks of the multi-level nested data generated by the large model is m; the root node set of all blocks of the multi-level nested data is R, the set of block root nodes of all blocks of the multi-level nested data at the level of the multi-level nested data is L, and the maximum block depth set of all blocks of the multi-level nested data is H. The relationship between the three is as follows: The set of unblocked nodes in the multi-level nested data is W, and the minimum set of nodes that meet the conditions is Z; Mathematical expression of the minimum satisfying condition level formula: The minimum set of nodes Z that satisfies the condition, where element j satisfies the condition: for each element W in the set W i , there exists j that belongs to the set RM i , so that j is the minimum value that satisfies the condition (the remainder of j divided by n is equal to 0); where the set RM i , the element j of the set satisfies the condition: for every element R in the set R i , there is an element L in the set L i and the elements H in the set H i , so that R i The parent node belongs to W i , and j is in the closed interval [max{1, L i -1},H i ]Inside; The mathematical expression of the transition formula is: S is the set of transition levels of the ith block in the multi-level nested data, where element j satisfies the condition: j belongs to the closed interval [L i ,H i ], and the remainder of j-1 divided by n is equal to 0.

Images