A custom hierarchical processing method and system for multi-source data

Abstract

The present invention discloses a method and system for customized hierarchical processing of multi-source data. The method includes establishing a front-end database to store raw data obtained from various channels; establishing a standard database, setting up multiple standard data sets in the standard database, using a tree structure to create the node position of the data in each standard data set, and encoding the data according to the node position; according to the node position, according to the preset encoding rules, the data after association is updated according to the encoding of the node position after association; obtaining the user's data processing request, and hierarchically processing the node position of the relevant data according to the user's processing request. The present application improves the flexibility of data fusion and reduces the design and development cost of data fusion through customization, multi-level, dynamic adjustment and other methods to adapt to the dynamically changing data environment.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a user-defined hierarchical processing method and system for multi-source data. Background Art

[0002] The statements in this section merely relate to the background technology related to the present invention and do not necessarily constitute the prior art.

[0003] Multi-source data fusion refers to the organic combination and analysis of data information from multiple different sources to improve the accuracy, comprehensiveness and reliability of information processing. Multi-source data fusion can make the obtained information more complete, unleash the potential of various data, improve data processing results, and promote scientific and automated decision-making.

[0004] Multi-source data fusion methods involve multiple disciplines, such as information processing, pattern recognition, artificial intelligence, database technology, etc. The commonly used multi-source data fusion methods are as follows: The first one is based on weighted average method: This method mainly uses different weights to comprehensively process data from multiple sources. Commonly used comprehensive methods include arithmetic mean, weighted mean, geometric mean, etc. By setting weights reasonably, information from different sources can be given corresponding weights to reflect their importance; the second method is based on feature extraction: this method mainly converts data from multiple sources into data in the same feature space through feature extraction, and then fuses them. Generally, feature extraction methods include principal component analysis, wavelet transform, independent component analysis, etc. This method can reduce the heterogeneity of data sources and extract key feature information, thereby improving the effect of data fusion; the third method is based on model: this method mainly models data from different data sources based on a specific model and then fuses them. Commonly used models include neural network models, decision tree models, gray system models, etc. By describing the relationship between data sources through specific models, various data can be better fused and the accuracy of the fusion results can be improved; the fourth method is based on decision rules: this method integrates the results of multiple data sources based on decision rules. Commonly used decision rules include simple majority voting, weighted majority voting, logistic regression, etc. By integrating the results of multiple data sources through reasonable decision rules, the effect of data fusion can be effectively improved.

[0005] However, traditional data fusion methods have the following shortcomings: 1. The fusion model is not flexible. Traditional data fusion methods often require pre-definition of data models and fusion rules, and have poor adaptability to dynamically changing data environments. When the data structure or data type changes, the entire data fusion process needs to be redesigned and developed, making it difficult to quickly respond to current business needs; 2. The business attribute abstraction level of the data structure is low. The traditional tree-shaped data fusion method does not abstract the content between the upper and lower data. Information is only stored on the leaf nodes, and the child nodes in the middle layer do not store information. There are deficiencies in business logic, storage space, and usage patterns; 3. Insufficient functionality. With business changes, when the data set needs to be transformed and adjusted as a whole, the existing data model can only be invalidated, and the data structure design and warehousing work must be re-performed. There is a lack of functions such as overall or partial cropping and deformation of the data set, which affects the overall implementation efficiency. Summary of the Invention

[0006] Therefore, the purpose of the present invention is to provide a customized hierarchical processing method and system for multi-source data, so as to realize the automatic data processing method of hierarchical organization, fine-grained expansion and normalized management of multi-source data.

[0007] To achieve the above-mentioned purpose, the present invention provides a customized hierarchical processing method for multi-source data, comprising the following steps:

[0008] S1. Establish a front-end database to store the original data obtained from various channels;

[0009] S2. Establish a standard database, set up multiple standard data sets in the standard database, create data node positions in each standard data set using a tree structure, and encode the data according to the node positions;

[0010] S3. According to the node positions, the data is associated according to a preset coding rule, and the associated data is coded and updated according to the associated node positions; the preset coding rule includes: the data structure code is composed of data bits and level bits, and the code length M is defined. Taking the maximum number of data levels N in the standard database, the level bit is [Lg(N)]+1, and the data bit is M-([Lg(N)]+1);

[0011] The data bits are divided into layers from left to right according to the number of layers N. The length of each layer of data bits is [(M-([Lg(N)]+1)) / N];

[0012] S4. Obtain the user's data processing request, and perform hierarchical processing on the node positions of the relevant data according to the user's processing request.

[0013] Further preferably, when the front-end database stores the original data, it also stores the data structure between the original data synchronously.

[0014] Further preferably, in S2, when a tree structure is used to create the node position of data in each standard data set, the tree structure is a multi-branch tree, including a root node, an intermediate node and a leaf node; wherein the root node and the intermediate node store the common content of the downward part of the node, and the leaf node stores the individual content of the node; each intermediate node includes 1 parent node and any number of child nodes; the root node is a unique identifier related to the described entity; the root node, the intermediate node, and the leaf node are all independent physical data tables; the number of data items of the root node, the intermediate node, and the leaf node under the same data set is consistent, and the data relationship corresponds one to one.

[0015] Further preferably, in S2, data structure encoding is performed according to the node position, and the encoding follows the following principles:

[0016] In the same standard database, the data structure encoding length is the same, and the encoding bits are in decimal;

[0017] The nodes under the same parent node are sorted from left to right, and grow naturally on the data bits of this layer with a step size of 1.

[0018] The data bits of each node are concatenated from the root node to all nodes in a top-down order, and the data bits below the node level are all set to 0.

[0019] The root node, intermediate node, and leaf node of the multi-branch tree of each standard data set form independent physical data tables based on their association relationships. The number of data items in the root node, intermediate node, and leaf node of the same data set is consistent, and the data relationships are one-to-one corresponding.

[0020] All nodes are represented by vectors, including: using the Tf-idf method to extract the vector features of each node's text attributes and perform vector representation;

[0021] All node vectors are calculated according to the following formula to complete the normalization of all node vectors:

[0022] ;

[0023] Among them, X2 indicates that the current node is a normalized vector, and x1 to xn are the original vectors of the child nodes under the current node.

[0024] Further preferably, in S4, when the user's data processing request is data splicing, N-1 data buffer areas are prepared, a data pre-reading operation is performed, the data of the root node and each intermediate node are respectively placed in the data buffer areas of the corresponding levels, and data splicing is performed to generate each complete data, which specifically includes the following steps:

[0025] S411. Generate a request code according to the data processing request, and perform a search on the physical data table of the standard data set using a pre-order breadth-based traversal method.

[0026] S412, obtaining the root node data content to the data cache;

[0027] S413. Read all direct child nodes under the current node N1 from left to right. When the read node matches the requested code, record the node data content into the data buffer.

[0028] S414: Determine whether the data bit + 1 of the current node exists. If so, proceed to the next step. If not, return to the previous level.

[0029] S415. Repeat the operations of S413-S414, each time taking the latest node recorded in the data buffer as the basis and reading the direct child nodes under it;

[0030] S416. When all N-1 positions in the data buffer are filled, spliced ​​data is generated, and a hash method is used to generate a 256-bit data fingerprint for each piece of data as a unique identifier of the data.

[0031] Further preferably, in S4, when the user's data processing request is data cutting or splicing,

[0032] Connect the lower level node P of the cropped A dataset to the Q node of the B dataset, including:

[0033] 1. Identify the nodes to be cut:

[0034] Identify the nodes to be cut except the root node and all its child nodes;

[0035] If the node to be cut contains multiple child nodes, the vectors of all child nodes are aggregated into the vector of the node to be cut by averaging;

[0036] 2. Select an alternative placement point

[0037] Traverse the entire tree structure and find all nodes at the same level or higher than the node to be pruned as candidate nodes;

[0038] Assume that the node to be cut is A and the candidate node is B. Calculate the similarity score between the node to be cut and the candidate position node;

[0039] Define a similarity threshold, select candidate nodes that exceed the threshold as candidate placement points, and sort them according to the score;

[0040] All intermediate nodes and leaf nodes under the P node are transferred to the Q node along with the P node;

[0041] The relationships between the nodes under the P node remain unchanged; the access rights, security level, encryption level, and desensitization level of each node under the P node meet the definition of the Q node.

[0042] Further preferably, in S4, when the data is spliced ​​and cut, the following steps are further included:

[0043] Verify the legitimacy of the target location. The legitimacy verification of the target location includes the following steps:

[0044] Obtain the target location's level information and compare it with the maximum level. If the target location's level is greater than the maximum level, it is considered illegal.

[0045] Determine whether the target location is a parent node. If the target location is not a root node, query the database for a corresponding parent node record based on the parent node identifier of the target location. If there is no parent node record, it is considered illegal.

[0046] To determine the ID duplication of the target node, a deep learning model is used to map a low-dimensional embedding vector to each node in the tree structure. The embedding vector includes the node structure and the attributes of the node data. The vector similarity between the target node and the original node is calculated to determine whether the node is duplicated and the compatibility of the target node.

[0047] Further preferably, in S4, it is further included to determine whether the tree structure after splicing or cutting satisfies the integrity constraint, including determining whether the tree structure after splicing or cutting satisfies:

[0048] a. Node uniqueness constraint: Search the node identification field in the entire multi-branch tree to ensure that the identification of each node in the tree structure is unique within the entire tree structure;

[0049] b. Parent-child relationship integrity constraint: For each non-root node, the judgment standard database points to its parent node record through a foreign key to ensure that each non-root node has only one parent node;

[0050] c. Hierarchical continuity constraint: Let the hierarchical set be L = {1, 2, 3, ..., k}. When inserting a new node, assume that the node belongs to the set L at the level i where it is planned to be inserted. Query the database to confirm whether its parent node exists. If not, the insertion is blocked.

[0051] The present invention also provides a system for customizing hierarchical processing of multi-source data, which is used to implement the above-mentioned method for customizing hierarchical processing of multi-source data, comprising:

[0052] The data storage module is used to establish a front-end database to store the original data obtained from various channels;

[0053] A database establishment module establishes a standard database, sets up multiple standard data sets in the standard database, creates the node position of the data in each standard data set using a tree structure, and encodes the data according to the node position;

[0054] The data encoding module associates data according to the node position and the preset encoding rules, and the associated data is encoded and updated according to the node position after association; the preset encoding rules include the following: the data structure encoding is composed of the concatenation of data bits and level bits, and the encoding length M is defined. According to the multi-tree hierarchical structure, the maximum number of data levels N in the standard database is taken, and the level bit is [Lg(N)]+1, and the data bit is M-([Lg(N)]+1);

[0055] The data bits are divided into layers from left to right according to the number of layers N. The length of each layer of data bits is [(M-([Lg(N)]+1)) / N];

[0056] The hierarchical processing module obtains the user's data processing request and performs hierarchical processing on the node positions of related data according to the user's processing request.

[0057] The present invention also provides an electronic device, comprising: a memory storing computer program instructions; and a processor, which implements the steps of the above-mentioned method for customized hierarchical processing of multi-source data when the computer program instructions are executed by the processor.

[0058] The present invention also provides a computer-readable storage medium for storing instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of the above-mentioned method for customized hierarchical processing of multi-source data.

[0059] The present invention discloses a method and system for customizing hierarchical processing of multi-source data, which has at least the following advantages over existing technologies:

[0060] 1. Improve the flexibility of data fusion and reduce the design and development costs of data fusion through customization, multi-level, dynamic adjustment, etc. to adapt to the dynamically changing data environment, including: the data model can be customized; the data model can be set to any number of levels, and the levels of each part can be different; the data model can be dynamically adjusted during use; and data model encoding rules are designed for various data operations.

[0061] 2. Improve the abstraction level of data structure. The present invention further abstracts the meaning of data. Each level of node saves the common attributes of all nodes below it, and splices them when needed, thereby improving the flexibility of the data processing system in terms of business logic, storage space, usage mode, etc.

[0062] 3. The functions of the data processing system have been expanded. The customized multi-level data structure can be freely transformed, adjusted, and other operations. At the same time, based on the designed data structure coding system, the overall or partial cutting and deformation of the data can be conveniently performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 This is a structural diagram of the customized hierarchical processing method for multi-source data provided by the present invention.

[0064] Figure 2 Schematic diagram of the data splicing process provided for this application.

[0065] Figure 3 This is a structural diagram of the customized hierarchical processing system for multi-source data provided by the present invention. DETAILED DESCRIPTION

[0066] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0067] like Figure 1 As shown, an embodiment of the present invention provides a method for customizing hierarchical processing of multi-source data, including the following steps:

[0068] S1. Establish a front-end database to store raw data obtained from various channels; mark data attributes according to data categories; when the front-end database stores raw data, it also stores the data structure between the raw data. Through this step, raw data from a wide range of sources and in various formats are unified and aggregated to provide a data foundation for subsequent systematic processing. Regardless of whether the data comes from imported data sources, interface data sources, database extraction, or file import, it can be properly stored in the front-end database. At the same time, when the front-end database stores raw data, it also stores the data structure between the raw data, which helps to understand the internal relationship between the raw data in subsequent processing and provides a reference for data conversion and integration.

[0069] S2. Establish a standard database, set up multiple standard data sets in the standard database, use a tree structure to create the node position of the data in each standard data set, and encode the data according to the node position; the tree structure here is a multi-branch tree, including a root node, intermediate nodes, and leaf nodes; wherein the root node and intermediate nodes store the common content of the node's lower part, and the leaf node stores the individual content of the node; each intermediate node includes a parent node and any number of child nodes; the root node is a unique identifier related to the described entity; the root node, intermediate node, and leaf node are all independent physical data tables; the number of data items in the root node, intermediate node, and leaf node under the same data set is consistent, and the data relationship is one-to-one corresponding. Each node includes several entity attributes, and the entity attributes are expressed in text form, including entity type and entity description.

[0070] The tree structure clearly displays the hierarchical relationships between data, supporting orderly data organization and efficient retrieval. Each node corresponds to specific data, and its position determines the hierarchy and order within the entire data system. The code based on the node position becomes a key identifier for subsequent data association and processing.

[0071] The data structure is encoded according to the node position, and the encoding follows the following principles:

[0072] 1. In the same standard database, the data structure encoding length is the same, and the encoding bits are in decimal;

[0073] 2. The nodes under the same parent node are sorted from left to right, growing naturally on the data bit of this layer with a step size of 1;

[0074] 3. The data bits of each node are concatenated from the root node to all nodes in a top-down order, and the data bits below the node level are all set to 0.

[0075] In addition, the root node, intermediate node, and leaf node of the multi-branch tree of each standard data set form independent physical data tables based on the association relationship; the number of data items in the root node, intermediate node, and leaf node of the same data set is consistent, and the data relationships correspond one to one. This design facilitates data storage, management, and query.

[0076] All nodes are represented by vectors, including: using the Tf-idf method to extract the vector features of each node's text attributes and perform vector representation;

[0077] All node vectors are calculated according to the following formula to complete the normalization of all node vectors:

[0078] ;

[0079] Among them, X2 indicates that the current node is a normalized vector, and x1 to xn are the original vectors of the child nodes under the current node.

[0080] S3. According to the node positions, the data is associated according to the preset coding rules, and the associated data is coded and updated according to the associated node positions; the data structure is coded according to the node positions, and the coding follows the following principles:

[0081] In the same standard database, the data structure encoding length is the same, and the encoding bits are in decimal;

[0082] The data structure code is composed of data bits and level bits. The code length M is defined. Taking the maximum number of data levels N in the standard database, the level bit is [Lg(N)]+1 and the data bit is M-([Lg(N)]+1).

[0083] This design allows the encoding to take into account both data levels and specific identification information. For example, if the maximum number of data levels N is 10, the level bits are [Lg (10)] + 1 = 2. If the code length M is set to 12, the data bits are 12 - 2 = 10.

[0084] The data bits are divided into layers from left to right according to the number of layers N. The length of each layer of data bits is [(M-([Lg(N)]+1)) / N];

[0085] Nodes under the same parent node are sorted from left to right, with a natural increase in the data bit length of that layer, with a step size of 1. This division ensures that each level of data has a corresponding encoding space and is distributed relatively reasonably. In the above example, the data bit length of each layer is (12-2) / 10 = 1. In actual applications, this is adapted by rounding and other methods.

[0086] Further preferably, the data bits of each node are concatenated from the root node to all nodes of the node in a top-down order, and the data bits of the node level lower than the node level are all set to 0.

[0087] S4. Obtain the user's data processing request, and perform hierarchical processing on the node positions of the relevant data according to the user's processing request.

[0088] like Figure 3 As shown in S4, when the user's data processing request is data splicing, N-1 data cache areas are prepared, and data pre-reading operations are performed. The data of the root node and each intermediate node are placed in the data cache areas of the corresponding levels respectively. According to the encoding of each level, the pre-order traversal method is adopted to read the left child node of each node starting from the root node until there is no left child node. Each data in the data table is spliced ​​from top to bottom in the order of the nodes passed through; the hash method is used to generate a 256-bit data fingerprint for each data as the unique identifier of the data.

[0089] like Figure 2 As shown, the specific steps include:

[0090] S411. Generate a request code according to the data processing request, and perform a search using a pre-order breadth-based traversal method on the standard data set structure.

[0091] S412, the root node obtains the data content to the data cache;

[0092] S413. Read all direct child nodes under the current node N1 from left to right. When the read node matches the requested code, record the node data content into the data buffer.

[0093] S414: Determine whether the data bit + 1 of the current node exists. If so, proceed to the next step. If not, return to the previous level.

[0094] S415. Repeat the operations of S413-S414, each time taking the latest node recorded in the data buffer as the basis and reading the direct child nodes under it;

[0095] S416. When all N-1 positions in the data buffer are filled, spliced ​​data is generated, and a hash method is used to generate a 256-bit data fingerprint for each piece of data as a unique identifier of the data.

[0096] Further preferably, in S4, when the user's data processing request is data clipping,

[0097] In the same standard database, the dataset at the lower node P of the trimmed data is connected to the dataset at the upper node Q, and the lower node P of the trimmed dataset A is connected to the Q node of the dataset B;

[0098] Connect the lower level node P of the cropped A dataset to the Q node of the B dataset, including:

[0099] 1. Identify the nodes to be cut:

[0100] Identify the nodes to be cut except the root node and all its child nodes;

[0101] If the node to be cut contains multiple child nodes, the vectors of all child nodes are aggregated into the vector of the node to be cut by averaging;

[0102] 2. Select an alternative placement point

[0103] Traverse the entire tree structure and find all nodes at the same level or higher than the node to be pruned as candidate nodes;

[0104] Assume that the node to be cut is A and the candidate node is B. Calculate the similarity score between the node to be cut and the candidate position node;

[0105] Define a similarity threshold, select candidate nodes that exceed the threshold as candidate placement points, and sort them according to the score;

[0106] All intermediate nodes and leaf nodes under the P node are transferred to the Q node along with the P node;

[0107] The relationships between the nodes under the P node remain unchanged; the access rights, security level, encryption level, and desensitization level of each node under the P node all meet the definition of the Q node.

[0108] After data is cut or spliced, the data association relationship is re-established. At this time, the data tables of the root node, intermediate node and child node in the standard database are automatically updated;

[0109] Obtain the data set and code at the upper node Q of the cropped data, and re-encode the codes of the lower node P and each node under the P node according to the relationship with other nodes at the upper node Q and the preset coding rules.

[0110] Further preferably, when data is spliced ​​and cut, in order to prevent the possible destruction of the integrity of the tree structure, the following measures are taken:

[0111] 1. Establish integrity constraints for the tree structure:

[0112] 1.1 Node Uniqueness Constraint: Let the node set be S. For the node identification field, set a unique index on it to ensure that for any two nodes n1, n2 ∈ S (n1 ≠ n2). At the database design level, set a unique index on the node identification field to ensure that the identifier of each node in the tree structure is unique within the entire structure, avoiding the situation where the same data has multiple different node positions and ensuring data consistency. For example, in a database table with an enterprise organizational structure tree structure, set a unique index on the department number field to prevent two departments from having the same number.

[0113] 1.2 Parent-Child Relationship Integrity: Utilize database foreign key constraints to maintain parent-child relationships. For each non-root node, its record points to its parent node record via a foreign key. Let the node set be S. For any non-root node element n∈S, ensure that each non-root node has exactly one parent node, and the root node has no parent node. Taking a file system tree-structured database as an example, in the table storing folder information, the foreign key field of non-root directory records points to their parent folder records, while the foreign key field of root directory records is left empty. This maintains the hierarchical logic of the tree structure and prevents issues such as dangling nodes or circular references.

[0114] 1.3 Hierarchical Continuity: Add hierarchical continuity checks to the program logic of data insertion or update operations. Assume the hierarchical set L = {1, 2, 3, ..., k}. When inserting a new node, assume that the node belongs to the set L at level i. A database query is performed to confirm whether its parent node exists. If not, the insertion operation is blocked. For example, in a product classification tree, when inserting a specific product classification node on the third level, first check whether the general category node on the second level and the general category node on the first level exist to ensure continuity between the levels in the tree structure.

[0115] 2. Verify the legitimacy of the splicing target location:

[0116] 2.1 Level Range Verification: Set a global variable in the program code to record the maximum level of the tree structure. When performing a data splicing operation involving a target location, obtain the target location's level information and compare it with the maximum level. If the maximum level of the tree structure is 5 and the target location is 6, the result is considered illegal, the splicing operation is terminated, and an error message is returned to the user.

[0117] 2.2 Parent Node Existence Verification: When performing data splicing, for the target location of a non-root node, the parent node identifier of the target location is used to check whether the corresponding parent node record exists in the database. For example, in a family tree structure, if a new member node is to be added to a certain location, the parent node is first checked to see if it exists in the tree database. If it does not exist, an error message is returned to prevent illegal node connections.

[0118] 2.3 Node Type Compatibility Verification: When designing the database, set a type field for each node to record the node type (e.g., category node, product node, etc.). During data concatenation, perform a validity check based on the type fields of the source and target nodes according to predefined rules for connecting different types of nodes. For example, in an e-commerce product classification tree, a category node can be connected to a subcategory node or product node, but a product node cannot be connected to a category node. When the concatenation target involves connecting different types of nodes, the type field is verified to ensure the legality of the operation.

[0119] like Figure 3 As shown, the present invention also provides a customized hierarchical processing system for multi-source data, which is used to implement the customized hierarchical processing method for multi-source data, including:

[0120] The data storage module is used to establish a front-end database to store the original data obtained from various channels;

[0121] A database establishment module establishes a standard database, sets up multiple standard data sets in the standard database, creates the node position of the data in each standard data set using a tree structure, and encodes the data according to the node position;

[0122] The data encoding module associates the data according to the node position and the preset encoding rules, and the associated data is updated according to the encoding of the associated node position;

[0123] The hierarchical processing module obtains the user's data processing request and performs hierarchical processing on the node positions of related data according to the user's processing request.

[0124] The present invention also provides an electronic device, comprising: a memory storing computer program instructions; and a processor, which implements the steps of the above-mentioned method for customized hierarchical processing of multi-source data when the computer program instructions are executed by the processor.

[0125] The present invention also provides a computer-readable storage medium for storing instructions. When the stored instructions are executed on a computer, the computer is caused to execute the steps of the above-mentioned method for customized hierarchical processing of multi-source data.

[0126] Obviously, the above embodiments are merely examples for clarity of explanation and are not intended to limit the implementation methods. Those skilled in the art will readily appreciate that other variations or modifications based on the above descriptions are possible. It is not necessary and impossible to enumerate all implementation methods here. Obvious variations or modifications arising therefrom remain within the scope of protection of the present invention.

Claims

1. A customized hierarchical processing method for multi-source data, characterized in that: The following steps are involved: S1. Establish a front-end database to store the original data obtained from various channels; S2. Establish a standard database, set up multiple standard data sets in the standard database, create data node positions in each standard data set using a tree structure, and encode the data according to the node positions; S3. According to the node position, the data is associated according to the preset coding rules, and the associated data is coded and updated according to the associated node position; The preset coding rules include: The data structure encoding consists of the concatenation of data bits and level bits. The encoding length M is defined. According to the multi-tree hierarchical structure, the maximum number of data levels N in the standard database is taken, then the level bit is [Lg(N)]+1, and the data bit is M-([Lg(N)]+1); The data bits are divided into data bits of each level from left to right according to the number of levels N. The length of each data bit is defined as [(M-([Lg(N)]+1)) / N]; S4. Obtain the user's data processing request and hierarchically process the node positions of the relevant data according to the user's processing request. When the user's data processing request is data clipping, in the same standard database, the data set at the lower node P of the clipped data is connected to the data set at the upper node Q, and the lower node P of the clipped data set is connected to the node below the Q node of the data set. 1). Identify the nodes to be cut Identify the nodes to be cut except the root node and all its child nodes; If the node to be cut contains multiple child nodes, the vectors of all child nodes are aggregated into the vector of the node to be cut by averaging; 2). Select an alternative placement point Traverse the entire tree structure and find all nodes at the same level or higher than the node to be pruned as candidate nodes; Assume that the node to be cut is A and the candidate node is B. Calculate the similarity score between the node to be cut and the candidate position node; Define a similarity threshold, select candidate nodes that exceed the threshold as candidate placement points, and sort them according to the score; All intermediate nodes and leaf nodes under the P node are transferred to the Q node along with the P node; The relationships between the nodes under the P node remain unchanged; the access rights, security level, encryption level, and desensitization level of each node under the P node meet the definition of the Q node.

2. The method for customizing hierarchical processing of multi-source data according to claim 1, characterized in that: When the front-end database stores the original data, the data structure between the original data is stored synchronously.

3. The method for customized hierarchical processing of multi-source data according to claim 1, characterized in that: In S2, when a tree structure is used to create the node position of data in each standard data set, the tree structure is a multi-branch tree, including a root node, an intermediate node and a leaf node; wherein the root node and the intermediate node store the common content of the lower part of the node, and the leaf node stores the individual content of the node; each intermediate node includes 1 parent node and any number of child nodes; the root node is a unique identifier related to the described entity; each node includes several entity attributes, and the entity attributes are expressed in text form including entity type and entity description.

4. The method for customizing hierarchical processing of multi-source data according to claim 1, characterized in that: In S2, data structure encoding is performed according to the node position, and the encoding follows the following principles: In the same standard database, the data structure encoding length is the same, and the encoding bits are in decimal; The nodes under the same parent node are sorted from left to right, and grow naturally on the data bit of this layer with a step size of 1; The data bits of each node are concatenated from the root node to all nodes in a top-down order, and the data bits below the node level are all set to 0.

5. The method for customizing hierarchical processing of multi-source data according to claim 4, characterized in that: The root node, intermediate node, and leaf node of the multi-branch tree of each standard data set form independent physical data tables based on the association relationship; The number of data items in the root node, intermediate node, and leaf node of the same data set is consistent, and the data relationships are one-to-one corresponding; All nodes are represented by vectors, including: using the Tf-idf method to extract the vector features of each node's text attributes and perform vector representation; All node vectors are calculated according to the following formula to complete the normalization of all node vectors: ； in, Indicates that the current node is a normalized vector, x1 to x n is the current node x s The original vector of the child node.

6. The method for customizing hierarchical processing of multi-source data according to claim 5, characterized in that: In S4, when the user's data processing request is data splicing, N-1 data buffers are prepared, data pre-reading operations are performed, and the data of the root node and each intermediate node are placed in the data buffer of the corresponding level. Data splicing is performed to generate each complete data. The specific steps include: S411. Generate a request code according to the data processing request, and perform a search on the physical data table of the standard data set using a pre-order breadth-based traversal method. S412, obtaining the root node data content to the data cache; S413. Read all direct child nodes under the current node N1 from left to right. When the read node matches the requested code, record the node data content into the data buffer. S414: Determine whether the data bit + 1 of the current node exists. If so, proceed to the next step. If not, return to the previous level. S415. Repeat the operations of S413-S414, each time taking the latest node recorded in the data buffer as the basis and reading the direct child nodes under it; S416. When all N-1 positions in the data buffer are filled, spliced ​​data is generated, and a hash method is used to generate a 256-bit data fingerprint for each piece of data as a unique identifier of the data.

7. The method for customizing hierarchical processing of multi-source data according to claim 1, characterized in that: In S4, when data is spliced ​​and trimmed, it also includes: Verify the legitimacy of the target location. The legitimacy verification of the target location includes the following steps: Obtain the target location's level information and compare it with the maximum level. If the target location's level is greater than the maximum level, it is considered illegal. Determine whether the target location is a parent node: If the target location is not a root node, query the database based on the parent node identifier of the target location to see if there is a corresponding parent node record; if there is no parent node record, it is determined to be illegal.

8. The method for customizing hierarchical processing of multi-source data according to claim 1, characterized in that: In S4, it is also included to determine whether the tree structure after splicing or cutting satisfies the integrity constraint, including determining whether the tree structure after splicing or cutting satisfies: a. Node uniqueness constraint: Search the node identification field in the entire multi-branch tree to ensure that the identification of each node in the tree structure is unique within the entire tree structure; b. Parent-child relationship integrity constraint: For each non-root node, the judgment standard database points to its parent node record through a foreign key to ensure that each non-root node has only one parent node; c. Hierarchical continuity constraint: Let the hierarchical set be L = {1, 2, 3, ..., k}. When inserting a new node, the node belongs to the set L at the level i to be inserted. The database query is used to confirm whether its parent node exists. If not, the insertion operation is blocked.

9. A customized hierarchical processing system for multi-source data, characterized in that: The method for implementing the customized hierarchical processing of multi-source data according to any one of claims 1 to 8 comprises: The data storage module is used to establish a front-end database to store the original data obtained from various channels; A database establishment module establishes a standard database, sets up multiple standard data sets in the standard database, creates the node position of the data in each standard data set using a tree structure, and encodes the data according to the node position; The data encoding module associates data according to the node position and the preset encoding rules, and the associated data is encoded and updated according to the node position after association; the preset encoding rules include the following: the data structure encoding is composed of the concatenation of data bits and level bits, and the encoding length M is defined. According to the multi-tree hierarchical structure, the maximum number of data levels N in the standard database is taken, and the level bit is [Lg(N)]+1, and the data bit is M-([Lg(N)]+1); The data bits are divided into layers from left to right according to the number of layers N. The length of each layer of data bits is [(M-([Lg(N)]+1)) / N]; The hierarchical processing module obtains the user's data processing request and performs hierarchical processing on the node positions of related data according to the user's processing request.

Images