Method, device and computer program product for determining nodes of decision tree

Abstract

The embodiment of the invention relates to a method and device for determining nodes of a decision tree and a computer program product. The method includes determining a plurality of features of a plurality of modalities corresponding to input information. The method further includes generating a multi-modal feature representation by combining the plurality of features of the multiple modalities. The method further includes determining a target path associated with the multi-modal feature representation in a decision tree, the decision tree including a plurality of nodes. The method further includes determining, in a target path, a target node associated with the input information for indicating a question or answer based on the multi-modal feature representation. Through the method, the feature representations corresponding to the input information of different modals can be fused to determine the multi-modal feature representation. In this way, richer and more accurate user intentions can be determined. Besides, the decision tree is traversed on the basis of the multi-modal feature representation, and the target node most related to the input information in the decision tree can be determined, so that accurate and related problems or solutions can be provided for users.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of multimodality, and more particularly, to a method, device, and computer program product for determining nodes of a decision tree. Background Art

[0002] A decision tree is a tree structure consisting of a root node, edges, internal nodes, and leaf nodes. The nodes of a decision tree have questions or solutions for user problems. A decision tree contains multiple paths, where a path refers to a sequence of nodes starting from a root node or an internal node and ending at a leaf node or an internal node. Different paths represent different solutions. Summary of the invention

[0003] Embodiments of the present disclosure relate to methods, devices, and computer program products for determining nodes of a decision tree.

[0004] According to a first aspect of the present disclosure, a method for determining a node of a decision tree is provided, the method comprising determining a plurality of features of a plurality of modalities corresponding to input information. A multimodal feature representation is generated by combining a plurality of features of a plurality of modalities. A target path associated with the multimodal feature representation in a decision tree is determined, the decision tree comprising a plurality of nodes. Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path. By using this method, information of different modalities in the input information input by the user can be processed to determine the multimodal feature representation corresponding to the input information. In this way, by integrating questions of different forms input by the user, a richer and more accurate user intention can be determined. Moreover, by using the multimodal feature representation as a basis for traversing the decision tree to determine the target node, on the one hand, the traversal efficiency can be improved, and on the other hand, the relevance of the target node to the input information can be improved, thereby providing the user with the best problem and solution.

[0005] According to a second aspect of the present disclosure, an electronic device for determining nodes of a decision tree is provided, comprising at least one processor, and a memory, coupled to the at least one processor and having instructions stored thereon, the instructions causing the electronic device to perform actions when executed by the at least one processor, the actions comprising determining multiple features of multiple modalities corresponding to input information. A multimodal feature representation is generated by combining multiple features of multiple modalities. A target path associated with the multimodal feature representation in a decision tree is determined, the decision tree comprising multiple nodes. Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path.

[0006] According to a third aspect of the present disclosure, there is provided a computer program product, which is tangibly stored on a non-volatile computer-readable medium and includes machine-executable instructions, which when executed cause a machine to perform the steps of the method implemented in the first aspect of the present disclosure.

[0007] The purpose of this Summary is to introduce a selection of concepts in a simplified form that are further described in the Detailed Description below. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other objects, features and advantages of the present disclosure will become more apparent through a more detailed description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, wherein like reference numerals generally represent like elements throughout the exemplary embodiments of the present disclosure.

[0009] Figure 1 A schematic diagram showing an example environment in which embodiments according to the present disclosure may be implemented;

[0010] Figure 2 A schematic diagram showing a method of determining nodes of a decision tree using a machine learning model according to an embodiment of the present disclosure is shown;

[0011] Figure 3 A flowchart of a method for determining nodes of a decision tree according to an embodiment of the present disclosure is shown;

[0012] Figure 4 A schematic diagram showing a process of determining a multimodal feature representation according to an embodiment of the present disclosure is shown;

[0013] Figure 5 A schematic diagram illustrating the structure of a decision tree according to an embodiment of the present disclosure;

[0014] Figure 6 A schematic diagram illustrating a process of traversing a decision tree according to an embodiment of the present disclosure; and

[0015] Figure 7 A schematic block diagram of an example device suitable for implementing embodiments of the present disclosure is illustrated. DETAILED DESCRIPTION

[0016] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments described herein, which are instead provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0017] In the description of the embodiments of the present disclosure, the term "including" and similar terms should be understood as open inclusion, that is, "including but not limited to". The term "based on" should be understood as "based at least in part on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc. may refer to different or the same objects. Other explicit and implicit definitions may also be included below.

[0018] A decision tree is a prediction model that represents a mapping relationship between object attributes and object values. Each node in the tree represents an object, and each path represents a possible attribute value. Each leaf node corresponds to the value of the object represented by the path from the root node to the leaf node. Generally, decision trees can be used for user services, such as providing users with the best questions or solutions for questions raised by users.

[0019] When applying decision trees to user services, different users may have different questions and may ask questions or requests in different ways. For example, questions can be asked using text, voice, vision, etc. Technicians need to use different models to encode the questions raised by users, and traverse the decision tree based on the encoded questions to provide users with the most relevant questions or solutions. In this way, on the one hand, building multiple models requires the use of a large amount of processing resources to generate processing costs. On the other hand, a single form of user questions provides less information, making it difficult to find the most relevant questions or solutions when traversing the decision tree, so that it is difficult to provide customers with satisfactory services.

[0020] At least to solve the above and other potential problems, an embodiment of the present disclosure provides a method for determining a node of a decision tree, the method comprising determining multiple features of multiple modalities corresponding to input information. A multimodal feature representation is generated by combining multiple features of multiple modalities. A target path associated with the multimodal feature representation in a decision tree is determined, and the decision tree includes multiple nodes. Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path. By using this method, information of different modals in the input information input by the user can be processed to determine the multimodal feature representation corresponding to the input information. In this way, by integrating different forms of questions input by the user, a richer and more accurate user intention can be determined. In addition, using the multimodal feature representation as a basis for traversing the decision tree to determine the target node can improve the traversal efficiency on the one hand, and on the other hand, it can improve the relevance of the target node to the question input by the user, thereby providing the user with the best problem and solution.

[0021] The basic principles and several exemplary embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Figure 1 1 is a schematic diagram of an example 100 in which the device and / or method according to an embodiment of the present disclosure may be implemented. It should be understood that Figure 1 The numbers and arrangements of components, elements, and systems shown are merely examples, and the architecture diagram may include different numbers and arrangements of components, elements, and systems.

[0022] Figure 1 The example 100 includes input information 102, a decision tree 104, an electronic device 106, and a machine learning model 108. The machine learning model 108 includes a multimodal encoder 110, a graph neural network 112, and a similarity module 114. In the example environment 100, the input information 102 is a request or question input by a user, such as "the computer fan is broken" or "what should I do if my computer is particularly slow". The information in the input information 102 can be in different modalities. For example, Figure 1 As shown, input information 102 includes image information 102-1 and text information 102-2. Among them, modality refers to some ways of expressing or perceiving things, and each source or form of information can be called a modality. For example, the modality of input information can be vision, voice, text, etc. Exemplarily, the input information can be an image, natural language text, or a voice sequence.

[0023] In the example environment 100, the machine learning model 108 may traverse the decision tree 104 based on the input information 102 to determine the target node in the decision tree 104 that is most relevant to the input information 102. Each node in the decision tree may contain a question or answer corresponding to the question input by the user, and after determining the target node, the best question or solution answer may be provided to the user based on the target node.

[0024] It should be understood that Figure 1 The types and numbers of models, data transmission processes, arrangements, etc. shown are only exemplary, and the example 100 may include models of different numbers and arranged in different ways, data transmission processes, and various additional elements, etc. It should be understood that the above examples are only used to illustrate the application of the machine learning model 108. With the development of technology, the machine learning model 108 may include various known or unknown applications in various fields and aspects.

[0025] In the example environment 100, the machine learning model 108 can be installed in any electronic device with processing computing resources or storage resources. For example, the electronic device can have common capabilities such as receiving and sending data requests, real-time data analysis, local data storage, and real-time network connection. Electronic devices can generally include various types of devices. Examples of electronic devices may include, but are not limited to: desktop computers, notebook computers, smart phones, wearable devices, security equipment, smart manufacturing equipment, smart home devices, Internet of Things devices, smart cars, drones, etc., and the present disclosure does not impose any restrictions on this.

[0026] Figure 2 FIG. 1 shows a schematic diagram of determining nodes of a decision tree using a machine learning model according to an embodiment of the present disclosure. Figure 2 As shown, the multimodal encoder 110 can process information of different modes of the input information 102 (e.g. Figure 2 The multimodal encoder 110 may be configured to encode the plurality of features of the plurality of modalities (not shown) to obtain a plurality of features of the plurality of modalities. In addition, the multimodal encoder 110 may fuse the plurality of features of the plurality of modalities to determine the multimodal feature representation 202.

[0027] As an example, the multimodal encoder 110 can be a deep learning model that can process information of multiple modalities, and it can use the correlation between different modalities to improve the representation ability and generalization ability. The multimodal encoder 110 is used to interact and fuse multiple features of different modalities. The multimodal encoder 110 can adapt to different downstream tasks such as visual question answering, image retrieval, machine translation, etc. through pre-training and fine-tuning.

[0028] like Figure 2As shown, the graph neural network 112 can encode the decision tree 104 and determine the decision tree features corresponding to the decision tree. Among them, the decision tree feature representation contains information about all edges and all nodes of the decision tree 104. The multimodal feature representation 202 determined by the multimodal encoder 110 can be input into the graph neural network 112, and the multimodal feature representation 202 can be used as the traversal basis of the graph neural network 112. In some embodiments, the graph neural network 112 can learn complex traversal strategies and search for a target path (not shown) associated with the multimodal feature representation in the decision tree. The target path refers to a node sequence consisting of nodes of different depths and corresponding edges.

[0029] like Figure 2 As shown, in order to further determine the node that may be most relevant to the input information, the similarity module 114 can be used to screen the nodes in the target path based on the multimodal feature representation 202, and determine the target node 204 associated with the multimodal feature representation 202, thereby providing the user with a more accurate and more satisfactory question or answer. For example, the similarity module 114 can be used to determine the similarity between the multimodal feature representation 202 and each node in the target path. Afterwards, one or more nodes with higher similarity can be selected as the target node 204 according to the ranking between the similarities.

[0030] In this way, when this machine learning model is applied to user services, the information of different modes of user input can be fused, and the connection information between different modes can be fully captured. Many metrics are introduced to make the multimodal feature representation obtained after fusion more accurate and richer in user intent. In addition, in the process of determining the multimodal feature representation, the same encoder (for example, Figure 1 The multimodal encoder 110 shown in the figure encodes information of different modes, which can save processing resources and processing costs. In addition, the decision tree is traversed using the multimodal feature representation as the traversal basis to determine the target node. On the one hand, it can improve the traversal efficiency and reduce the time and cost of user services. On the other hand, it can improve the relevance of the target node with the problem input by the user, thereby providing the user with the best solution. This can obviously enhance the performance and efficiency of various user services and improve the comfort of users when using.

[0031] Figure 3 A flow chart of a method for determining nodes of a decision tree according to an embodiment of the present disclosure is shown. Figure 1 The illustrated machine learning model 108 can determine a target node in the decision tree that is associated with the input information and is used to indicate a question or an answer based on the decision tree and the input information.

[0032] In box 302, the input information may include information of multiple modalities, for example, the input information includes visual information, text information and voice information. The input information can be input by the user to characterize the user's needs. Exemplarily, the input information can be an image, a speech sequence and a text consisting of multiple words. Input information of different modalities can characterize the user's needs from multiple dimensions. Among them, multiple modalities refer to the use of multiple modalities to describe the request made by the same user, such as a video, picture, voice, text, etc. describing a request. Alternatively, multiple modalities can also be homogeneous modalities, such as multiple images taken from two types of cameras, such as infrared images and visible light images.

[0033] In some embodiments, the multiple features corresponding to the input information may be image features, text features, language features corresponding to the input information, or infrared image features, visible light image features, etc. Features of different modalities may represent the input information from different dimensions.

[0034] In practical applications, information of different modalities needs to be encoded using different single-modal encoders to obtain features of the corresponding modalities. For example, text information is encoded using a text encoder to determine the text features corresponding to the text information. An image is encoded using an image encoder to determine the image features corresponding to the image. In this way, using multiple single-modal encoders for encoding will not only lead to waste of resources and increased processing costs, but also ignore the connection information between information of different modalities, making it impossible to accurately determine user intent, and thus unable to provide users with accurate solutions.

[0035] Therefore, in some embodiments, the same multimodal encoder (e.g., Figure 1 The multimodal encoding module 110 shown in ) encodes information of different modalities to determine the features corresponding to the information of each modality. For example, text features, image features, speech features, etc. Alternatively, multiple modalities may also refer to different types of wherein the multimodal encoder may be a machine learning model based on an attention mechanism. The multimodal encoder may include a self-attention layer (Self-Attention), a cross-attention layer (Cross-Modal), and a fully connected feedforward network layer (Feed-Forward Networks, FFN). Among them, the fully connected feedforward network layer may include multiple linear layers and a fully connected layer (activation function).

[0036] In block 304, after determining the multiple features of the multiple modalities, the multiple features of the multiple modalities may be fused based on the complementarity between the multiple modalities to generate a multimodal feature representation. In some embodiments, the multimodal feature representation may be a joint representation. The joint representation refers to mapping the features of the multiple modalities together into the same multimodal vector space to obtain a fused feature representation. For example, the inputs of the multiple modalities are x1, x2, ..., x n , the joint representation x m =f(x1, x2, ..., x n ). In this way, the inter-modal relationship between information of different modalities can be better captured, and a multimodal feature representation containing more information can be determined. Multimodal feature representation may include features that can express the same or related semantic information extracted from information of different modalities (e.g., images, text, audio, etc.) of the input information.

[0037] In box 306, the decision tree can be traversed using the multimodal feature representation as a traversal basis to determine a target path associated with the multimodal feature representation. The decision tree includes a root node, internal nodes, and leaf nodes. The leaf node is the node with the highest depth in the decision tree, and each leaf node includes an answer that may be related to the input information. The internal node includes questions that may be related to the input information. Nodes are connected by edges. The target path contains multiple nodes, and a path is formed from the node with the shallowest depth to the node with the deepest depth. It can be understood that each node in the target path is associated with the multimodal feature representation. The number of target paths can be multiple.

[0038] In block 308, since there may be multiple target paths and a target path may contain multiple nodes, in order to find the target node that is closest to and most relevant to the input information in the decision tree, the nodes in the target path may be further screened using multimodal feature representation to determine the target node. In this way, the user may be asked questions or provided with solutions based on the questions or answers corresponding to the nodes at the next depth of the target node in the decision tree.

[0039] In some embodiments, in order to facilitate the similarity comparison between the multimodal feature representation and the nodes in the decision tree, the decision tree can be encoded using a graph neural network to determine the decision tree features of the decision tree. The decision tree features may include the embedding of multiple nodes, such as the embedding of the root node, the embedding of the leaf node, and the embedding of the internal node. Among them, the network architecture of the graph neural network can be an architecture based on the attention mechanism. In some embodiments, the graph neural network may include a self-attention layer, an interactive attention layer, and an FFN layer. The self-attention layer can calculate the attention weights between each pair of nodes in the output sequence and generate output sequences of the same length. The interactive attention layer can calculate the attention weights of each node in the output sequence and the multimodal feature representation.

[0040] In some embodiments, the graph neural network can convert the decision tree into a directed acyclic graph (DAG) form, and the directed acyclic graph form can be encoded as a decision tree feature corresponding to the decision tree. Exemplarily, the graph neural network can be trained based on a loss function to adjust the network parameters of the graph neural network. Among them, the loss function can include a cross entropy loss function (Cross Entropy Loss), a hinge loss function (Hinge Loss Function), etc.

[0041] Through the above method, the information of different modes input by the user can be fused, and the connection information between different modes can be fully captured, so that the multimodal feature representation obtained after fusion may contain more accurate and richer user intentions. In addition, using the multimodal feature representation as the traversal basis to traverse the decision tree to determine the target node can improve the traversal efficiency and reduce the time and cost of customer service on the one hand, and improve the relevance of the target node to the problem input by the user on the other hand, so as to provide the best solution for the user. This can obviously improve the user experience.

[0042] The following will combine Figures 4 to 7 Some embodiments of the present disclosure are described in detail. For ease of understanding, the specific embodiments mentioned in the following description are exemplary and are not intended to limit the scope of protection of the present disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or may omit the actions shown, and the scope of the present disclosure is not limited in this respect.

[0043] Figure 4 FIG. 2 is a schematic diagram showing a process of determining a multimodal feature representation according to an embodiment of the present disclosure. Figure 4As shown, the multimodal encoder 110 includes a coding layer 402, an interactive attention layer 404, and a self-attention layer 406. The core components of the interactive attention layer 404 and the self-attention layer 406 are attention models. The attention model often uses the QKV (Query-Key-Value) mode to determine the output sequence corresponding to the input sequence. For example, given an input sequence The output sequence after processing by the attention model is The specific calculation process is:

[0044] First, the input sequence is mapped to three different spaces to obtain the query matrix Q, key matrix K, and value matrix V:

[0045]

[0046]

[0047]

[0048] Among them, W q , W k , W v is an adjustable parameter of the attention model, D k is the dimension of the key matrix.

[0049] Then the output sequence H can be obtained according to the attention mechanism:

[0050]

[0051] Where softmax(·) is a column-by-column normalization function.

[0052] like Figure 4 As shown, the coding layer 402 can use a cosine function or a sine function to encode the sequence 408 corresponding to the input information. For example, the input sequence X408-1 corresponding to the image in the input information can be encoded, the input sequence Y408-2 corresponding to the voice in the input information can be encoded, and the input sequence Z408-3 corresponding to the text in the input information can be encoded to obtain multiple encoded embeddings (not shown).

[0053] The encoded multiple embeddings are input to the interactive attention layer 404. The interactive attention layer 404 can use the cross attention calculation formula to determine the multimodal vectors corresponding to the two modalities for a pair of modalities formed by any two modalities. For example, the input sequence X408-1 corresponding to the image in the input information can be used as the query matrix, and the input sequence Y408-2 corresponding to the voice in the input information can be used as the key matrix, and the cross attention calculation formula can be used to determine the multimodal feature vector H X-Y :

[0054] Q X =XW qX (5)

[0055] K Y =YW qY (6)

[0056] V Y =YW qY (7)

[0057]

[0058] H X-Y =A X-Y V Y W qX (9)

[0059] Among them, A X-Y is the cross attention matrix. It can be understood that the multimodal feature vectors between the other two modalities can also be determined according to the calculation method of formulas (6)-(9).

[0060] Through the above method, the paired cross-modal mechanism can be used to perform cross-modal fusion on the features of the two modalities. By continuously introducing features of other modalities to strengthen the features of the target modality, a multi-modal feature representation is obtained after the fusion of multiple modalities.

[0061] like Figure 4 As shown, in determining multiple multimodal feature vectors (such as H X-Y , H Z-Y , H Y-X ), the multiple multimodal feature vectors output by the interactive attention layer 404 can be input to the self-attention layer 406, and after being processed by the self-attention layer 406 and the feedforward sublayer (not shown), the final joint representation (multimodal feature representation) is obtained. In this way, the semantic alignment and complementarity between different modalities can be fully captured, so that the user intent contained in the final joint representation is more accurate.

[0062] In some embodiments, a multimodal encoder can be pre-trained in a self-supervised manner. For example, the Masked Language Modeling (MLM) method and the Masked Modal Modeling (MMM) method. Among them, masked language modeling can randomly mask some tags of the natural language modality and predict them based on the contextual information of other modalities. Masked modal modeling can randomly and completely mask a modality and predict them based on the contextual information of other modalities. Taking masked language modeling as an example, masked language modeling is used for pre-training, which can not only learn the intrinsic relationship between natural language text, but also combine visual information with natural language text. According to experience, the masking percentage can be set to 15%. The loss function of MLM can be defined as:

[0063]

[0064] Where X = (x1, x2, ..., x n ) is the input sequence, Y=(y1, y2, …, y n ) is the output sequence. i is the occluded embedding, X \i Remove x from the input sequence i The remaining sequence of .

[0065] In addition, the loss function of MMM can be defined as:

[0066]

[0067] Where X = (x1, x2, ..., x n ) is the input sequence, y i is the occluded embedding.

[0068] In some embodiments, the multimodal encoder can be trained using unlabeled sample image features and sample text features to determine the trained multimodal encoder. The sample image features and sample text features are feature representations for the same target object. For example, a multimodal encoder can be constructed, which includes multiple adjustable network parameters. Afterwards, the sample image features and sample text features can be input into the multimodal encoder, and the sample image features and sample text features can be fused via the multimodal encoder to obtain multimodal sample features.

[0069] Furthermore, a multimodal encoder is used to randomly mask some tag embeddings of sample text features, and the tag embeddings are predicted based on contextual information of other modalities to obtain predicted sample text features. Afterwards, the predicted text features and sample image features can be fused to obtain predicted multimodal sample features (prediction results). Finally, based on the difference between the predicted multimodal sample features and the multimodal sample features, the network parameters can be iteratively adjusted until the difference meets the preset requirements. Among them, the difference between the predicted multimodal sample features and the multimodal sample features can be determined according to the loss function. The difference meeting the preset requirements can be that the number of iterative adjustments is greater than the set number, or the difference is less than the set difference value.

[0070] Through this training method, a large amount of unlabeled data from different modalities can be used to train a joint representation that captures both intra-modal and inter-modal dependencies. Moreover, it can improve the performance and generalization of multi-modal encoders.

[0071] In actual applications, decision trees are deep and complex in structure. Traversing the decision tree requires more processing resources and longer processing time. In order to improve traversal efficiency and save traversal time, in some embodiments, a beam search algorithm can be used to traverse the decision tree based on multimodal feature representation to determine the target path. Figure 5 Describe the structure of a decision tree. Figure 5 A schematic diagram showing the structure of a decision tree according to an embodiment of the present disclosure.

[0072] like Figure 5 As shown, the decision tree 500 includes a root node 502, a plurality of internal nodes (e.g., 504-1, 504-2, 504-3, 504-4, Figure 5 The numbers of all internal nodes are not marked in the figure) and multiple leaf nodes (e.g., 506-1, 506-2, 506-3, Figure 5 Root node 502 is the node with the shallowest depth in decision tree 500, and leaf node 506 is the node with the deepest depth in decision tree 500. The depth of the node increases gradually from the root node to the leaf node. Nodes are connected by edges (e.g., 508-1, 508-2, 508-3, etc.), and each leaf node includes a solution for solving the user's problem. Figure 5 The decision tree 500 shown in FIG. 5 is only an example, and the number and structure of nodes may be modified as needed.

[0073] Figure 6 FIG. 4 is a schematic diagram showing a process of traversing a decision tree according to an embodiment of the present disclosure. Figure 6As shown, the multimodal feature representation can be compared with the embedding corresponding to each first node 602 at the next depth of the root node for similarity, and M first candidate nodes (e.g., first candidate node A, second candidate node B) can be determined. Where M is a predetermined number, which can be set by the user, for example, it can be set to 3, 5, 7, etc. Exemplarily, based on the predetermined number M, M first nodes can be selected from several first nodes with higher similarity as first candidate nodes.

[0074] like Figure 6 As shown, the multimodal feature representation is continued to be compared with the embedding corresponding to each second node 604 at the next depth of the first candidate node for similarity, and M second candidate nodes (e.g., second candidate node Y, second candidate node H) are determined. Finally, the multimodal feature representation is compared with the embedding corresponding to each third node 606 at the next depth of the second candidate node for similarity, and M third candidate nodes (e.g., third candidate node W, third candidate node V) are determined. It can be understood that when the node at the next depth is a leaf node, or the traversal depth reaches the set depth, the traversal is stopped.

[0075] It can be understood that after determining multiple first candidate nodes, multiple second candidate nodes, and multiple third candidate nodes, multiple target paths (for example, target path GAYW, target path GAYV, target path GBH) can be determined based on the connection relationship between the candidate nodes.

[0076] In some embodiments, in order to further determine the nodes that may be most relevant to the multimodal feature, the nodes may be screened based on the similarity between the multimodal feature representation and each node in the target path. In this way, the target node associated with the multimodal feature representation may be determined, thereby providing the user with a more accurate and satisfactory question or answer. For example, a similarity model (e.g., Figure 1 The similarity module 114 shown in FIG. 1 determines the similarity between the multimodal feature representation and each node in the target path. Afterwards, one or more nodes with higher similarity can be selected as the target node based on the ranking between the similarities. Exemplarily, the similarity model can be trained based on the loss function to adjust the network parameters of the similarity model. Among them, the loss function can include a cross entropy loss function, a hinge loss function, etc.

[0077] In some embodiments, decision trees can be applied to a variety of scenarios, and the information included in decision trees applicable to different application scenarios is different. For example, in the application scenario of applying decision trees to medical diagnosis, the object attributes and object values ​​included in the decision tree may be constructed based on the patient's physical signs, medical history and other information. The nodes of the decision tree may include disease inferences and diagnosis results. In the application scenario of using decision trees for text classification, the object attributes and object values ​​included in the decision tree may be determined based on various text information such as news articles, emails, etc., and the nodes of the decision tree may include classification results.

[0078] Therefore, before traversing the decision tree, it is necessary to determine the decision tree associated with the user request based on the user's request. For example, a preset association relationship between input information and application scenario parameters can be pre-constructed. Among them, the application scenario parameters can be determined by the feature information corresponding to the decision tree. Exemplarily, the application scenario parameters can be "education", "medical care", "face recognition", etc. Furthermore, a corresponding decision tree can be selected from multiple decision trees based on the preset association relationship between the input information and the application scenario parameters.

[0079] In this way, before traversing the decision tree, the decision tree that is more relevant to the input information can be determined, thereby saving traversal resources and improving traversal efficiency.

[0080] Figure 7 A schematic block diagram of an example device 700 that may be used to implement embodiments of the present disclosure is shown. Figure 1 The computing device in can be implemented using device 700. Figure 7 As shown, the device 700 includes a central processing unit (CPU) 701, which can perform various appropriate actions and processes according to computer program instructions stored in a read-only memory (ROM) 702 or computer program instructions loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the storage device 700 can also be included. The CPU 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0081] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage page 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0082] The various processes and processing described above, such as method 300, may be performed by processing unit 701. For example, in some embodiments, method 300 may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on device 700 via ROM 702 and / or communication unit 709. When the computer program is loaded into RAM 703 and executed by CPU 701, one or more actions of method 300 described above may be performed.

[0083] The present disclosure may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present disclosure.

[0084] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the above. More specific examples (non-exhaustive list) of computer readable storage medium include: portable computer disk, hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disk read-only memory (CD-ROM), digital versatile disk (DVD), memory stick, floppy disk, mechanical encoding device, such as a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination of the above. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (e.g., a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0085] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0086] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed entirely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0087] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0088] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processing unit of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0089] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0090] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of a module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0091] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The choice of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining a node of a decision tree, comprising: Determine multiple features of multiple modalities corresponding to the input information; Generating a multimodal feature representation by combining multiple features of the multiple modalities; determining a target path associated with the multimodal feature representation in a decision tree, the decision tree comprising a plurality of nodes; and Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path.

2. The method according to claim 1, wherein determining multiple features of multiple modalities corresponding to the input information comprises: The information of different modalities of the input information is encoded respectively via a multimodal encoder to generate multiple features of the multiple modalities.

3. The method according to claim 2, wherein generating a multimodal feature representation by combining multiple features of the multiple modalities comprises: Determine a joint representation corresponding to the multiple features of the multiple modalities by cross-modal fusion of the multiple features of the multiple modalities; as well as The multimodal feature representation is determined based on the joint representation.

4. The method of claim 1, wherein determining a target path in a decision tree associated with the multimodal feature representation comprises: Generate a decision tree feature corresponding to the decision tree by encoding the decision tree through a graph neural network; as well as A target path associated with the multimodal feature is determined by comparing the decision tree feature with the multimodal feature representation.

5. The method of claim 4, wherein generating a decision tree feature corresponding to the decision tree by encoding the decision tree via a graph neural network comprises: A plurality of embeddings corresponding to the plurality of nodes are generated by encoding the plurality of nodes in the decision tree via the graph neural network.

6. The method of claim 5, wherein determining a target path associated with the multimodal feature representation by comparing the decision tree features with the multimodal feature representation comprises: Determining a preset number of first group candidate nodes based on a similarity between the multimodal feature representation and an embedding corresponding to a first node at a next depth from a root node of the decision tree; Determining the preset number of second group candidate nodes based on the similarity between the multimodal feature representation and the embedding corresponding to the second node at the next depth of each candidate node in the first group of candidate nodes; Repeating the step of determining the preset number of second group candidate nodes based on the similarity between the multimodal feature representation and the embedding corresponding to the second node at the next depth of each candidate node in the first group of candidate nodes, and determining the preset number of i-th group candidate nodes, until the node at the next depth is a leaf node or the depth reaches a predetermined depth threshold, and i is greater than 2; The target path is determined based on the first group of candidate nodes, the second group of candidate nodes, and the i-th group of candidate nodes.

7. The method according to claim 1, wherein the input information includes at least two of natural language text, image data, and voice data.

8. The method according to claim 1, wherein the target path includes a plurality of candidate nodes connected in sequence, and based on the multimodal feature representation, determining in the target path a target node for indicating a question or an answer associated with the input information comprises: Determining similarities between the multimodal feature representation and the plurality of candidate nodes; as well as According to the similarity, a target node for indicating a question or an answer associated with the input information is determined from the multiple candidate nodes.

9. The method according to claim 1, further comprising: Determining application scenario parameters associated with the input information; as well as The decision tree that matches the application scenario parameters is selected from multiple decision trees.

10. The method according to claim 2, further comprising: Acquire sample text features and sample image features, wherein the sample text features and the sample image features are content-associated; Constructing a multimodal encoder, wherein the multimodal encoder is provided with training parameters; Inputting the sample text features and the sample image features into the multimodal encoder to generate a prediction result; Based on the difference between the prediction result and the sample text feature and the sample image feature, the training parameter is iteratively adjusted until the difference meets a preset requirement.

11. An electronic device, comprising: at least one processor; as well as A memory coupled to at least one processor and having instructions stored thereon, the instructions causing the electronic device to perform actions when executed by the at least one processor, the actions comprising: Determine multiple feature representations of multiple modalities corresponding to the input information; Generating a multimodal feature representation by combining multiple features of the multiple modalities; determining a target path associated with the multimodal feature representation in a decision tree, the decision tree comprising a plurality of nodes; and Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path.

12. The electronic device according to claim 11, wherein determining the multiple features of the multiple modalities corresponding to the input information comprises: The information of different modalities of the input information is encoded respectively via a multimodal encoder to generate multiple features of the multiple modalities.

13. The electronic device according to claim 12, wherein generating a multimodal feature representation by combining a plurality of features of the plurality of modalities comprises: Determine a joint representation corresponding to the features of the multiple modalities by cross-modal fusion of the multiple features of the multiple modalities; as well as The multimodal feature representation is determined based on the joint representation.

14. The electronic device of claim 11, wherein determining a target path associated with the multimodal feature representation in a decision tree comprises: generating a decision tree feature corresponding to the decision tree by encoding the decision tree via a graph neural network; as well as A target path associated with the multimodal feature representation is determined by comparing the decision tree features to the multimodal feature representation.

15. The electronic device of claim 14, wherein generating a decision tree feature corresponding to the decision tree by encoding the decision tree via a graph neural network comprises: A plurality of embeddings corresponding to the plurality of nodes are generated by encoding the plurality of nodes in the decision tree via the graph neural network.

16. The electronic device of claim 15, wherein determining a target path associated with the multimodal feature representation by comparing the decision tree feature representation with the multimodal feature representation comprises: Determining a preset number of first group candidate nodes based on a similarity between the multimodal feature representation and an embedding corresponding to a first node at a next depth from a root node of the decision tree; Determining the preset number of second group candidate nodes based on the similarity between the multimodal feature representation and the embedding corresponding to the second node at the next depth of each candidate node in the first group of candidate nodes; Repeating the step of determining the preset number of second group candidate nodes based on the similarity between the multimodal feature representation and the embedding corresponding to the second node at the next depth of each candidate node in the first group of candidate nodes, and determining the preset number of i-th group candidate nodes, until the node at the next depth is a leaf node or the depth reaches a predetermined depth threshold, and i is greater than 2; as well as The target path is determined based on the first group of candidate nodes, the second group of candidate nodes, and the i-th group of candidate nodes.

17. The electronic device according to claim 11, wherein the input information comprises at least two of natural language text, image data, and voice data.

18. The electronic device according to claim 11, wherein the target path includes a plurality of candidate nodes connected in sequence, and based on the multimodal feature representation, determining in the target path a target node for indicating a question or an answer associated with the input information comprises: Determining similarities between the multimodal feature representation and the plurality of candidate nodes; as well as According to the similarity, a target node for indicating a question or an answer associated with the input information is determined from the multiple candidate nodes.

19. The electronic device according to claim 11, further comprising: Determining application scenario parameters associated with the input information; as well as The decision tree that matches the application scenario parameters is selected from multiple decision trees.

20. A computer program product tangibly stored on a non-transitory computer readable storage medium and comprising computer executable instructions that, when executed, cause a computer to: Determine multiple features of multiple modalities corresponding to the input information; Generating a multimodal feature representation by combining multiple features of the multiple modalities; determining a target path associated with the multimodal feature representation in a decision tree, the decision tree comprising a plurality of nodes; and Based on the multimodal feature representation, a target node for indicating a question or an answer associated with the input information is determined in the target path.

Images