An HTC algorithm framework implementation method for batch processing data

Abstract

The application discloses an HTC algorithm framework implementation method for batch processing data and relates to the technical field of data processing. Compared with previous text classification methods, the application improves the problems in the prior art that data processing consumes a long time, and a large number of sub-models are loaded, which leads to excessive memory consumption. The application reduces the occupied memory during classification by reducing the model loading times, solves the problem of large memory consumption when multiple sub-models are used for prediction in the traditional method, solves the problem that many sub-models that are not needed to be called are possibly loaded in the traditional method, and improves the data processing speed from the perspective of batch processing.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to an implementation method of the HTC algorithm framework for batch data processing. Background Technology

[0002] HTC stands for Hierarchical Text Classification. For documents (Doc) participating in HTC, we assign one or more tags.

[0003] HTC represents a very important type of text classification problem in practical work, and there are often multiple approaches to solving it. Generally, we categorize these methods into three main types:

[0004] Method 1: Multiple sub-models form a tree structure, with each model solving only one classification problem;

[0005] Method 2: Treat the tags in the Doc as a series of sub-tags, with a sequence relationship between the sub-tags;

[0006] Method 3: Treat this problem as a multi-label, multi-classification problem.

[0007] Of the three methods above, Method 1 requires multiple sub-models to complete the entire classification task; while Methods 2 and 3 often use a single complete model (e.g., a complex neural network model). Methods 2 and 3 have the advantages of fewer models and simpler input and output. However, in actual production, if the label system is partially adjusted, the models used in Methods 2 and 3 need to be updated, and the updated models cannot guarantee the label classification performance of the un-updated parts. Therefore, in production practice, Method 1, besides using more sub-models, provides better support for updating the label system; only the sub-models involved in updating the labels need to be updated, while the other sub-models can continue to be used.

[0008] When using Method 1, we often need to first load all sub-models into memory, or provide multiple servers, each responsible for serving a portion of the sub-models. Then, for the input Doc, we call all sub-models from the root node to the leaf node according to the top-down structure of the model tree. For a single Doc, the number of models called to complete the processing is equal to the height of the label tree. If there are many labels (e.g., 300 labels, divided into three levels: the first level has 10 nodes; the second level has 5 nodes; the third level has 6 nodes; (6*5*10=300)), 10+5+6=21 models need to be added to memory during the entire processing. If each model consumes an average of 500MB of memory, then all these models will consume approximately 10GB of memory.

[0009] To address the aforementioned issues, this invention proposes an HTC algorithm framework implementation method for batch data processing. Summary of the Invention

[0010] The purpose of this invention is to propose an HTC algorithm framework implementation method for batch data processing to solve the problems mentioned in the background art:

[0011] Existing technical solutions consume a lot of time for data processing, and the excessive number and frequency of loading sub-models leads to excessive memory consumption.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] An HTC algorithm framework implementation method for batch data processing includes the following steps:

[0014] S1: Add a virtual parent node to the first-level tag as the root node of the entire tag tree;

[0015] S2: Divide the second-level nodes according to the second-level tag corresponding to the root node of the tag tree, and so on, to form a tag tree by dividing the nodes corresponding to each level of tags;

[0016] S3: Define a classifier for each node in the label tree that has non-empty child nodes;

[0017] S4: Obtain the document set D from the documents to be submitted by HTC;

[0018] S5: Perform hierarchical label classification on each document in the document set D based on the loading algorithm.

[0019] Preferably, in step S2, the entire tag tree is formally represented as follows:

[0020] Parent(C i ) = root

[0021]

[0022]

[0023] Among them, Parent(C i =root is the root node of the tag tree; The label sequence is k1, k2, ... k n nodes The parent node, i.e. The label sequence is k1, k2, ... k h nodes The child nodes, i.e. k h+1 This represents the (h+1)th level of the tag tree; NumChildrenUnderNode represents the maximum number of child nodes a node can have.

[0024] The nodes in the entire tag tree are divided into two categories: InternalNode and LeafNode. InternalNode indicates that the node has non-empty child nodes; LeafNode indicates that the node has no child nodes.

[0025] Preferably, in S3, for each InternalNode corresponding to S2... Define a corresponding classifier: The classifier assigns a label to the input Doc. in, The hierarchical sequence is represented as k1, k2, ... k m The node, Represents a node The corresponding child nodes.

[0026] Preferably, the number of classifiers is equal to the number of InternalNodes, TotalNumOfInterNodes.

[0027] Preferably, in step S3, all classifiers are aggregated to obtain a classifier set:

[0028]

[0029] Where H is the maximum depth of the tag tree.

[0030] Preferably, the loading algorithm described in S5 is as follows:

[0031] S5.1: Initialize the queue (Queue);

[0032] S5.2: M root Add to the queue (Queue);

[0033] S5.3: Check if the queue is empty. If it is not empty, continue the following loop operation; otherwise, stop the operation:

[0034] S5.3.1: Pop the model from the tail of the queue using pop, load it into memory, and assign it to the variable m;

[0035] S5.3.2: Create an empty set to store the labels predicted by the model;

[0036] S5.3.3: Create an empty list to store the updated collection of documents;

[0037] S5.3.4: Iterate through each element in the document collection D and assign each element to the variable doc;

[0038] S5.3.5: By calling the predict method of the loaded model m, taking doc as input, the loaded model m is inferred and its prediction results are obtained;

[0039] S5.3.6: By calling the addLabel method to add labels to a collection of documents, the prediction result is passed as a parameter and added to the labels of the documents;

[0040] S5.3.7: Add the document "doc" to the end of the document collection by calling the append method;

[0041] S5.3.8: Add an element t to doc;

[0042] S5.3.9: Add element t to set S;

[0043] S5.3.10: Assign the document collection D_n to the variable D;

[0044] S5.3.11: In each iteration, use a TagNode2Model(t) function and add the model output by the TagNode2Model(t) function to the queue Queue. Process each element in the set S in turn through the above loop operation and add the processing result to the queue.

[0045] Preferably, the TagNode2Model(t) function described in S5.3.11 is used to find the next-level classification model object corresponding to the tag node t.

[0046] Compared with existing technologies, this invention provides an HTC algorithm framework implementation method for batch data processing, which has the following beneficial effects:

[0047] This invention reduces the memory usage during classification by reducing the number of model loadings, thus solving the problem of high memory consumption when using multiple sub-models for prediction in traditional methods; it also solves the problem of loading many unnecessary sub-models in traditional methods; and from the perspective of batch processing, it improves the data processing speed. Attached Figure Description

[0048] Figure 1 This is a flowchart of the method mentioned in Embodiment 1 of the present invention;

[0049] Figure 2 This is a schematic diagram of the tag tree node mentioned in Embodiment 1 of the present invention;

[0050] Figure 3 This is a schematic diagram of the tag tree node and its corresponding tag mentioned in Embodiment 2 of the present invention. Detailed Implementation

[0051] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0052] This invention modifies the real-time online inference in Method 1 to an offline batch processing mode. This HTC algorithm structure enables all sub-models to be loaded into memory at most once; only one sub-model is loaded into memory at any given time; and the total time for model loading and data processing is reduced. Specifically, it includes the following:

[0053] Example 1:

[0054] Please see Figure 1-2 The present invention provides an HTC algorithm framework implementation method for batch data processing, comprising:

[0055] S1: Add a virtual parent node to the first-level tag as the root node of the entire tag tree; the details are as follows:

[0056] To ensure that all nodes are placed under the same tree, we add a virtual parent node for the first-level tags, which serves as the root node of the entire tag tree: root.

[0057] S2: Based on the second-level tag corresponding to the root node of the tag tree, divide it into second-level nodes, and so on, dividing the nodes corresponding to each level of tags to form a tag tree; the details are as follows:

[0058] Reference Figure 2We can see that the root node's child nodes include: C1, C2, and C3, representing three categories (e.g., ride-hailing, shopping mall, and hotel); and C1's child nodes include: C 1,1 C 1,2 For child node C 1,1 For example, its parent node is C1; the entire label tree can be formally represented as follows:

[0059] Parent(C i ) = root

[0060]

[0061]

[0062] Among them, Parent(C i =root is the root node of the tag tree; The label sequence is k1, k2, ... k n nodes The parent node, i.e. The label sequence is k1, k2, ... k h The child nodes of the node, i.e. k h+1 The (h+1)th level of the tag tree; NumChildrenUnderNode This indicates the maximum number of child nodes a node can have.

[0063] The nodes in the entire tag tree can be divided into two categories, as defined below:

[0064] InternalNode indicates that the node still has non-empty child nodes;

[0065] LeafNode refers to a node that has no more child nodes.

[0066] S3: Define a classifier for each node in the label tree that has non-empty child nodes; specifically as follows:

[0067] For each InternalNode in the above definition: Define a classifier: The classifier assigns a label to the input Doc. in, The hierarchical sequence is represented as k1, k2, ... k m The node, Represents a node The corresponding child nodes. Therefore, the number of classifiers is the same as the number of InternalNodes, TotalNumOfInterNodes;

[0068] The set of all classifiers can be defined as:

[0069]

[0070] Where H is the maximum depth of the tag tree;

[0071] S4: Obtain the document set D from the documents to be submitted for HTC participation; details are as follows:

[0072] The documents to be processed are D1, D2, ..., D n The documents are merged to obtain document set D.

[0073] S5: Perform hierarchical label classification on each document in the document collection D based on the loading algorithm / model. Details are as follows:

[0074] Based on the above definition and the following process, each of the above classifiers only needs to be loaded into memory once at most to achieve the goal of assigning hierarchical labels to each document in the document set D.

[0075] Initialize a queue (Queue);

[0076] #M root Add to the queue:

[0077] Queue.push(M root )

[0078] # Check if the queue is empty. If it is not empty, continue the following loop operation; otherwise, stop the operation:

[0079] While(Queue.empty()!=True):

[0080] # Pop the model from the tail of the queue (Queue) using `pop`, load it into memory, and assign it to the variable `m`:

[0081] m = loadModel(Queue.pop())

[0082] # Create an empty collection to store the labels predicted by the model:

[0083] S = Set()

[0084] # Create an empty list to store the updated collection of documents:

[0085] D_n=[]

[0086] # Iterate through each element in the document collection D and assign each element to the variable doc:

[0087] for Doc in D:

[0088] #By calling the predict method of the loaded model m, taking the document doc as input, the loaded model m performs inference and obtains its prediction results.

[0089] t = m.predict(Doc)

[0090] #By calling the addLabel method to add labels to a collection of documents, the prediction result is passed as an argument and added to the document's label:

[0091] doc.addLabel(t)

[0092] # Add the document "doc" to the end of the document collection by calling the append method:

[0093] D_n.append(doc)

[0094] # Add an element t to the document doc:

[0095] doc.append(t)

[0096] # Add element t to set S:

[0097] S.add(t)

[0098] #Assign the document collection D_n to the variable D:

[0099] D = D_n

[0100] # In each iteration, a TagNode2Model(t) function is used (meaning that if a document d is labeled with tag t at level h, then when labeling d at level h+1, a classifier: TagNode2Model(t) is needed), and the model output by the TagNode2Model(t) function is added to the queue Queue. The above loop operation is used to process each element in the set S in turn, and the processing result is added to the queue:

[0101] for t in S

[0102] Queue.push(TagNode2Model(t))

[0103] In the above process, each classifier will be loaded into memory at most once; some branches of the tree will not be reached by the top-level classifier, so these classifiers will not be loaded into memory later.

[0104] Example 2:

[0105] Reference Figure 3 , where C i …C n Let C1 represent the classifier, and C0 represent the root node. Documents processed by classifier C0 may be labeled with any one of the sets {L1, L2, L3}. Documents processed by classifier C2 may be labeled with a tag: {L1, L2, L3}. 2,1 ,L 2,2 Any one of them, further, through C 2,2 The document may be marked with {L} 2,2,1 ,L 2,2,2 ,L 2,2,3 Any one of the following. Based on this, the two documents [D1, D2] are classified into three levels, as shown in the table below:

[0106] Table 1. Loading Flowchart for Three-Level Classification

[0107]

[0108]

[0109] Step 0 above is the initialization operation, which will... The model is loaded into the queue. It awaits the classification operation for documents [D1, D2].

[0110] In step 1, The model performs a first-level classification on documents [D1, D2], labeling each document [D1, D2] with a label L2. At this point, set S stores the labels L2 and proceeds to step 2. Finally, the model to be loaded in step 2 is... Load the tags into the queue and store them in the collection of documents [D1, D2].

[0111] In step 2, pop the queue from the queue. The model performs a two-level classification on documents [D1, D2], and labels each document [D1, D2] with a label L. 2,2 and L 2,1 At this time, set S stores tag L 2,2 and L 2,1 Proceed to step 3, and load the model from step 3. and Load the tags into the queue and store them in the collection of documents [D1, D2].

[0112] In step 3, pop the queue from the queue. The model performs a three-level classification of documents [D1, D2], but only labels document D1 with the L tag. 2,1,1 At this time, set S stores tag L2,1,1 Proceed to step 4 and store the tags in the collection of documents [D1, D2];

[0113] In step 4, pop the item from the queue. The model performs a three-level classification of documents [D1, D2], but only labels document D2 with the L tag. 2,2,3 At this time, set S stores tag L 2,1,1 The tags are then stored in the collection of documents [D1,D2] to complete the three-level classification of documents [D1,D2].

[0114] As shown in the table, during the three-level classification process of documents [D1, D2], each classifier is loaded at most once or not loaded at all. For example, classifiers C1 and C3 are not used, so their subsequent classifiers C... 1,1 C 1,2 C 3,1 and C 3,2 It will not be used, thus avoiding being reloaded into memory.

[0115] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for implementing an HTC algorithm framework for batch data processing, characterized in that, Includes the following steps: S1: Add a virtual parent node to the first-level tag as the root node of the entire tag tree. ; S2: Based on the root node of the tag tree The corresponding second-level labels are divided into second-level nodes, and so on, to form a label tree by dividing the nodes corresponding to each level of labels; S3: Define a classifier for each node in the label tree that has non-empty child nodes; S4: Obtain the document collection from the documents of HTC that will participate. ; S5: Document collection based on loading algorithm Each document in the file is categorized using hierarchical tags; the loading algorithm is as follows. S5.1: Initialize the queue ; S5.2: Will Add to queue middle; S5.3: Check Queue If the value is not empty, continue the following loop operation; otherwise, stop the operation: S5.3.1: Through From queue Pop the model from the tail of the queue, load it into memory, and assign it to a variable. ; S5.3.2: Create an empty set to store the labels predicted by the model; S5.3.3: Create an empty list , used to store the updated collection of documents; S5.3.4: Traverse the set of input documents for this iteration Each element in the variable is assigned to a variable. ; S5.3.5: Loading the model via a call of The method will As input, the model will be loaded. Make inferences and obtain its prediction results; S5.3.6: Adding tags to a document collection by calling [function name]. The method involves passing the prediction results as parameters and adding them to the document's tags; S5.3.7: By calling Method, to document Add to The end; S5.3.8: To Add an element ; S5.3.9: To a set Add elements ; S5.3.10: Will Assigning a value to a variable ; S5.3.11: In each iteration, use a function, and The model output by the function is added to the queue. In this process, the sets are processed sequentially through the aforementioned loop operations. Each element in the queue is processed, and the result is added to the queue.

2. The HTC algorithm framework implementation method for batch data processing according to claim 1, characterized in that, In S2, the entire tag tree is formally represented as follows: in, This is the root node of the tag tree; The label sequence is nodes The parent node, i.e. ; The label sequence is nodes The child nodes, i.e. ; For the first tag tree layer; This indicates the maximum number of child nodes a node can have. The nodes in the entire tag tree are divided into and Two categories, among which This indicates that the node has non-empty child nodes; This indicates that the node no longer has any child nodes.

3. The HTC algorithm framework implementation method for batch data processing according to claim 2, characterized in that, In S3, for each of S2 corresponding nodes Define a corresponding classifier: The classifier provides the corresponding label for the input Doc. ,in, The hierarchical sequence is represented as The node, Represents a node The corresponding child nodes.

4. The HTC algorithm framework implementation method for batch data processing according to claim 3, characterized in that, The number of classifiers and quantity equal.

5. The HTC algorithm framework implementation method for batch data processing according to claim 4, characterized in that, In step S3, all classifiers are aggregated to obtain a classifier set: in, This represents the maximum depth of the tag tree.

6. The HTC algorithm framework implementation method for batch data processing according to claim 1, characterized in that, As described in S5.3.11 The function is used to find tag nodes. The corresponding next-level classification model object.

Images