Information processing method and device, storage medium and electronic device

Abstract

The invention discloses an information processing method and device, a storage medium and an electronic device, and relates to the technical field of smart home / smart home, and the method comprises the steps: obtaining to-be-fused text features obtained by carrying out text coding on to-be-fused text information, dynamic graph features obtained by performing graph coding on a to-be-fused first dynamic graph are obtained, and the first dynamic graph is of a graph-like data structure constructed with text information as nodes; performing feature fusion on the to-be-fused text features and the dynamic graph features according to a group of preset fusion modes to obtain a group of candidate fusion features; selecting a group of target fusion features from the group of candidate fusion features according to the information amount difference between the candidate fusion features in the group of candidate fusion features; and performing dynamic graph updating on the first dynamic graph by using the group of target fusion features to obtain a second dynamic graph.

Description

Technical Field

[0001] The present application relates to the field of smart home / intelligent family, and specifically, to an information processing method and device, a storage medium and an electronic device. Background Art

[0002] At present, Machine Reading Comprehension (MRC) is widely used in customer service robots, intelligent law, intelligent education and other fields. Among them, Interactive Machine Reading Comprehension (IMRC) can cope with large-scale knowledge data sets, integrate and make full use of the information in the knowledge data sets, and learn domain knowledge through reinforcement learning.

[0003] However, in the interactive machine understanding technology based on dynamic knowledge graph, information fusion is only performed through the attention method, and the fusion method is single, which cannot fully utilize the feature information in the knowledge data set. Therefore, it can be seen that the information processing method in the related technology has the problem of low information utilization due to the single information fusion method. Summary of the invention

[0004] The embodiments of the present application provide an information processing method and device, a storage medium, and an electronic device to at least solve the problem of low information utilization rate caused by a single information fusion method in the information processing method of the related art.

[0005] According to one aspect of an embodiment of the present application, a method for processing information is provided, including: obtaining text features to be fused obtained by text encoding text information to be fused, and obtaining dynamic graph features obtained by graph encoding a first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; performing feature fusion on the text features to be fused and the dynamic graph features according to a set of preset fusion methods to obtain a set of candidate fusion features; selecting a set of target fusion features from the set of candidate fusion features according to the information amount difference between each candidate fusion feature in the set of candidate fusion features, wherein the information amount difference between each target fusion feature in the set of target fusion features is greater than or equal to a preset information amount difference threshold; and performing dynamic graph update on the first dynamic graph using the set of target fusion features to obtain a second dynamic graph.

[0006] In an exemplary embodiment, selecting a group of target fusion features from the group of candidate fusion features based on the information amount difference between each candidate fusion feature in the group of candidate fusion features includes: determining the relative entropy corresponding to each candidate fusion feature pair in a group of candidate fusion feature pairs based on the group of candidate fusion features, wherein each candidate fusion feature pair includes two candidate fusion features, and the relative entropy corresponding to each candidate fusion feature pair is the relative entropy between the two candidate fusion features in each candidate fusion feature pair; selecting a group of target fusion features from the group of candidate fusion features based on the relative entropy corresponding to each candidate fusion feature pair.

[0007] In an exemplary embodiment, selecting a group of target fusion features from the group of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair includes: selecting candidate fusion features from the group of candidate fusion features in sequence as current candidate fusion features to perform the following feature selection operations until the group of candidate fusion features is empty: when the relative entropy between the current candidate fusion feature and the selected target fusion feature is greater than or equal to a preset relative entropy threshold, determining the current candidate fusion feature as a target fusion feature, and removing the current candidate fusion feature from the group of candidate fusion features; when the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, removing the current candidate fusion feature from the group of candidate fusion features.

[0008] In an exemplary embodiment, selecting a group of target fusion features from the group of candidate fusion features according to the relative entropy of each candidate fusion feature pair includes: determining a group of fusion feature pairs to be processed according to the relative entropy corresponding to each candidate fusion feature pair, wherein the group of fusion feature pairs to be processed are candidate fusion feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold; sequentially performing the following processing operations on each of the group of fusion feature pairs to be processed as the current fusion feature pair to be processed until the group of fusion feature pairs to be processed is empty: selecting candidate fusion features to be removed from the current fusion feature pair to be processed, removing the candidate fusion features to be removed from the group of candidate fusion features, and removing the fusion feature pairs to be processed that include the candidate fusion features to be removed in the group of fusion feature pairs to be processed; wherein the updated group of candidate fusion features is the group of target fusion features.

[0009] In an exemplary embodiment, before using the set of target fusion features to update the first dynamic image, the method further includes: performing feature update operations less than N times with the set of target fusion features as the first fusion feature layer to obtain (N+1) fusion feature layers, wherein N is a positive integer greater than or equal to 1, and when performing the feature update operation once, the highest fusion feature layer is the current fusion feature layer, and the fusion features in the current fusion feature layer are the current fusion features: performing feature fusion on each of the current fusion features and the reference features corresponding to each of the current fusion features according to at least one fusion method in the set of preset fusion methods to obtain a reference feature corresponding to each of the current fusion features. The next fusion feature corresponding to the current fusion feature, wherein the reference feature corresponding to each of the current fusion features includes at least one of the following: the text feature to be fused, the dynamic image feature, the fusion feature in the fusion feature layer before the current fusion feature layer, and other current fusion features; according to the information difference between the next fusion features corresponding to all the current fusion features, the fusion features in the next fusion feature layer of the current fusion feature layer are selected from the next fusion features corresponding to all the current fusion features to obtain the next fusion feature layer of the current fusion feature layer; wherein the (N+1)th fusion feature layer is the updated set of target fusion features.

[0010] In an exemplary embodiment, the feature fusion of the text features to be fused and the dynamic graph features is performed according to a set of preset fusion methods to obtain a set of candidate fusion features, including at least one of the following: feature fusion processing is performed on the text features to be fused and the dynamic graph features respectively through each feature fusion network in at least one feature fusion network to obtain candidate fusion features corresponding to each feature fusion network; after aligning the feature dimensions of the text features to be fused and the dynamic graph features, feature superposition processing is performed on the text features to be fused and the dynamic graph features whose feature dimensions are aligned according to each feature superposition method in at least one feature superposition method to obtain candidate fusion features corresponding to each feature superposition method.

[0011] In an exemplary embodiment, the method of performing feature fusion processing on the text features to be fused and the dynamic image features respectively through each feature fusion network in at least one feature fusion network to obtain candidate fusion features corresponding to each feature fusion network includes at least one of the following: inputting the text features to be fused as the first weight matrix of the attention network and the dynamic image features as the second weight matrix and the third weight matrix of the attention network into the attention network to obtain the first candidate fusion features output by the attention network; inputting the text features to be fused and the dynamic image features into the target convolutional neural network to obtain the second candidate fusion features output by the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network for feature fusion; performing feature superposition processing on the text features to be fused and the dynamic image features aligned in feature dimensions according to each feature superposition method in at least one feature superposition method to obtain candidate fusion features corresponding to each feature superposition method includes at least one of the following: performing feature addition operation on the text features to be fused and the dynamic image features aligned in feature dimensions to obtain the third candidate fusion features; performing feature splicing operation on the text features to be fused and the dynamic image features aligned in feature dimensions to obtain the fourth candidate fusion features.

[0012] In an exemplary embodiment, before obtaining the text features to be fused obtained by text encoding the text information to be fused, the method also includes: when the text information to be fused contains negative words, performing semantic inversion processing on the text information to be fused to obtain updated text information to be fused.

[0013] According to another aspect of an embodiment of the present application, there is also provided an information processing device, comprising: an acquisition unit, used to acquire text features to be fused obtained by text encoding the text information to be fused, and to acquire dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; a fusion unit, used to perform feature fusion on the text features to be fused and the dynamic graph features according to a set of preset fusion methods to obtain a set of candidate fusion features; a selection unit, used to select a set of target fusion features from the set of candidate fusion features according to the information amount difference between each candidate fusion feature in the set of candidate fusion features, wherein the information amount difference between each target fusion feature in the set of target fusion features is greater than or equal to a preset information amount difference threshold; an updating unit, used to perform dynamic graph update on the first dynamic graph using the set of target fusion features to obtain a second dynamic graph.

[0014] In an exemplary embodiment, the selection unit includes: a determination module, used to determine the relative entropy corresponding to each candidate fusion feature pair in a set of candidate fusion feature pairs based on the set of candidate fusion features, wherein each candidate fusion feature pair includes two candidate fusion features, and the relative entropy corresponding to each candidate fusion feature pair is the relative entropy between the two candidate fusion features in each candidate fusion feature pair; and a selection module, used to select a set of target fusion features from the set of candidate fusion features based on the relative entropy corresponding to each candidate fusion feature pair.

[0015] In an exemplary embodiment, the selection module includes: a first execution submodule, which is used to select candidate fusion features from the group of candidate fusion features in sequence as current candidate fusion features to perform the following feature selection operations until the group of candidate fusion features is empty: when the relative entropy between the current candidate fusion feature and the selected target fusion feature is greater than or equal to a preset relative entropy threshold, the current candidate fusion feature is determined as a target fusion feature, and the current candidate fusion feature is removed from the group of candidate fusion features; when the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, the current candidate fusion feature is removed from the group of candidate fusion features.

[0016] In an exemplary embodiment, the selection module includes: a determination submodule, used to determine a group of fusion feature pairs to be processed according to the relative entropy corresponding to each candidate fusion feature pair, wherein the group of fusion feature pairs to be processed are candidate fusion feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold; a second execution submodule, used to sequentially use each of the fusion feature pairs to be processed in the group of fusion feature pairs to be processed as the current fusion feature pair to be processed to perform the following processing operations until the group of fusion feature pairs to be processed is empty: select the candidate fusion feature to be removed from the current fusion feature pair to be processed, remove the candidate fusion feature to be removed from the group of candidate fusion features, and remove the fusion feature pairs to be processed that contain the candidate fusion feature to be removed in the group of fusion feature pairs to be processed; wherein the updated group of candidate fusion features is the group of target fusion features.

[0017] In an exemplary embodiment, the method further includes: a first execution unit, which is used to perform N or less feature update operations with the set of target fusion features as the first fusion feature layer before updating the first dynamic image using the set of target fusion features, to obtain (N+1) fusion feature layers, wherein N is a positive integer greater than or equal to 1, and when performing the feature update operation once, the highest fusion feature layer is the current fusion feature layer, and the fusion features in the current fusion feature layer are the current fusion features: perform feature fusion on each of the current fusion features and the reference features corresponding to each of the current fusion features according to at least one fusion method in the set of preset fusion methods, A next fusion feature corresponding to each of the current fusion features is obtained, wherein the reference feature corresponding to each of the current fusion features includes at least one of the following: the text feature to be fused, the dynamic image feature, the fusion feature in the fusion feature layer before the current fusion feature layer, and other current fusion features; according to the information difference between the next fusion features corresponding to all the current fusion features, the fusion features in the next fusion feature layer of the current fusion feature layer are selected from the next fusion features corresponding to all the current fusion features to obtain the next fusion feature layer of the current fusion feature layer; wherein the (N+1)th fusion feature layer is the updated set of target fusion features.

[0018] In an exemplary embodiment, the fusion unit includes at least one of the following: a first processing module, used to perform feature fusion processing on the text features to be fused and the dynamic image features respectively through each feature fusion network in at least one feature fusion network, and obtain candidate fusion features corresponding to each feature fusion network; a second processing module, used to align the feature dimensions of the text features to be fused and the dynamic image features, and then perform feature superposition processing on the text features to be fused and the dynamic image features with aligned feature dimensions according to each feature superposition method in at least one feature superposition method, and obtain candidate fusion features corresponding to each feature superposition method.

[0019] In an exemplary embodiment, the first processing module includes at least one of the following: a first input submodule, used to input the text features to be fused as the first weight matrix of the attention network, and the dynamic graph features as the second weight matrix and the third weight matrix of the attention network into the attention network, so as to obtain the first candidate fusion features output by the attention network; a second processing submodule, used to input the text features to be fused and the dynamic graph features into the target convolutional neural network, so as to obtain the second candidate fusion features output by the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network for feature fusion; the second processing module includes at least one of the following: a first execution submodule, used to perform a feature addition operation on the text features to be fused and the dynamic graph features aligned in feature dimensions, so as to obtain the third candidate fusion features; a second execution submodule, used to perform a feature splicing operation on the text features to be fused and the dynamic graph features aligned in feature dimensions, so as to obtain the fourth candidate fusion features.

[0020] In an exemplary embodiment, the method also includes: a second execution unit, which is used to perform semantic inversion processing on the text information to be fused before obtaining the text features to be fused by text encoding the text information to be fused, when the text information to be fused contains negative words, to obtain the updated text information to be fused.

[0021] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the above-mentioned information processing method when running.

[0022] According to another aspect of an embodiment of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the information processing method through the computer program.

[0023] In an embodiment of the present application, a plurality of preset fusion methods are used to fuse text features and dynamic image features to be fused, by obtaining text features to be fused obtained by text encoding of text information to be fused, and obtaining dynamic image features obtained by graph encoding of a first dynamic image to be fused, wherein the first dynamic image is a graph-like data structure constructed with text information as nodes; according to a set of preset fusion methods, feature fusion of text features and dynamic image features to be fused is performed to obtain a set of candidate fusion features; according to the information amount difference between each candidate fusion feature in the set of candidate fusion features, a set of target fusion features is selected from the set of candidate fusion features, wherein the target fusion features in the set of target fusion features are selected from the set of candidate fusion features. The information difference between each target fusion feature is greater than or equal to a preset information difference threshold; a set of target fusion features is used to update the first dynamic image to obtain a second dynamic image. Since a plurality of preset fusion methods are used to fuse the text features to be fused and the dynamic image features, the information in the text features to be fused and the dynamic image features can be fully utilized, and according to the information difference between each fusion feature, the fusion features are screened to obtain fusion features with a large information difference, so as to achieve the purpose of obtaining more information and achieve the technical effect of improving information utilization, thereby solving the problem of low information utilization due to a single information fusion method in the information processing method of the related technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0025] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Figure 1 is a schematic diagram of a hardware environment of an information processing method according to an embodiment of the present application;

[0027] Figure 2 is a flowchart of an optional information processing method according to an embodiment of the present application;

[0028] Figure 3 is a schematic diagram of an optional information processing method according to an embodiment of the present application;

[0029] Figure 4 is a schematic diagram of another optional information processing method according to an embodiment of the present application;

[0030] Figure 5 is a schematic diagram of another optional information processing method according to an embodiment of the present application;

[0031] Figure 6 is a schematic diagram of another optional information processing method according to an embodiment of the present application;

[0032] Figure 7 is a schematic diagram of another optional information processing method according to an embodiment of the present application;

[0033] Figure 8 is a structural block diagram of an optional information processing device according to an embodiment of the present application;

[0034] Fig. 9 It is a structural block diagram of an optional electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0036] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0037] According to one aspect of an embodiment of the present application, a method for processing information is provided. The method for processing information is widely used in smart home, smart home, smart home device ecology, smart house ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the above-mentioned method for processing information can be applied to Figure 1 In the hardware environment composed of the terminal device 102 and the server 104 shown in FIG. Figure 1As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0038] The network may include but is not limited to at least one of the following: wired network, wireless network. The wired network may include but is not limited to at least one of the following: wide area network, metropolitan area network, local area network, and the wireless network may include but is not limited to at least one of the following: WIFI (Wireless Fidelity), Bluetooth. The terminal device 102 may be but is not limited to a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing device, a smart dishwasher, a smart projection device, a smart TV, a smart clothes drying rack, a smart curtain, a smart audio and video, a smart socket, a smart speaker, a smart fresh air device, a smart kitchen and bathroom device, a smart bathroom device, a smart sweeping robot, a smart window cleaning robot, a smart mopping robot, a smart air purification device, a smart steamer, a smart microwave oven, a smart kitchen treasure, a smart purifier, a smart water dispenser, a smart door lock, etc.

[0039] The information processing method of the embodiment of the present application can be executed by the server 104, or by the terminal device 102, or by both the server 104 and the terminal device 102. The information processing method of the embodiment of the present application executed by the terminal device 102 can also be executed by a client installed thereon.

[0040] Taking the information processing method in this embodiment executed by the server 104 as an example, Figure 2 is a flow chart of an optional information processing method according to an embodiment of the present application, such as Figure 2 As shown, the process of the method may include the following steps:

[0041] Step S202, obtaining the text features to be fused obtained by text encoding the text information to be fused, and obtaining the dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph data structure constructed with text information as nodes.

[0042] The information processing method in this embodiment can be applied in the scenario of updating a dynamic graph. In machine reading comprehension technology, it is usually necessary to construct a dynamic graph based on text information and update the dynamic graph based on new text information. A dynamic graph (dynamic knowledge graph) is a graph-like data structure that uses text information as nodes and is constructed based on the relationship information of each node. A dynamic graph can be used to identify entities or concepts and the relationships between them. In a dynamic graph, the "entity-relationship-entity" (or "subject-predicate-object") triple is the basic knowledge unit. Entities or concepts are linked together through the relationships between them to form a network structure, which is similar to a knowledge graph.

[0043] In recent years, machine reading comprehension (MRC) has been widely used in customer service robots, intelligent law, and intelligent education. With the further development of machine reading comprehension technology, machine reading comprehension can also be applied to some sub-fields where natural language understanding technology is relatively mature, such as search engines, advertising, and recommendations. In addition, in the case of dealing with large-scale knowledge data sets, interactive machine reading comprehension (IMRC) technology is also gradually emerging, which can use the idea of ​​reinforcement learning to learn domain knowledge.

[0044] In the application field of machine reading comprehension, although the knowledge content in each field is complex and changeable, and is frequently updated, the instruction documents in each field (that is, documents on relevant knowledge in each field) will have commonalities or similarities in structure or semantics. IMRC technology can store this common structural and semantic information in the model through some storage methods, build a dynamic knowledge graph, use reinforcement learning to simulate human reading patterns, and realize knowledge reasoning.

[0045] However, machine reading comprehension research still faces challenges such as insufficient knowledge and reasoning ability, lack of explainability, and lack of training data. In machine reading comprehension technology, knowledge and reasoning ability require the model to fully understand semantic information, and its knowledge and reasoning ability will affect the effect of the algorithm model. Therefore, it is necessary to integrate the stored structure and semantic information, domain information, and even out-of-domain information. How to make full use of this information is one of the keys to knowledge reasoning. At present, in interactive machine comprehension technology based on dynamic knowledge graphs, information fusion is usually completed only through attention methods. The fusion method is relatively simple and feature information is not fully utilized.

[0046] Interpretability has many dimensions. Generally speaking, it refers to the interpretability in the application stage. In high-credibility fields, such as finance, education, and healthcare, interpretability needs to be fully considered. In machine reading comprehension technology, interpretability is usually the criterion and basis for method design, especially for black box models such as neural networks. The interpretability of the entire model cycle has always been the focus and difficulty. In many neural network-related technologies, the quality of model structure or method design is often verified by improving the effect, and the internal factors that can improve the effect of the structure or method are not paid attention to, that is, the interpretability of the structure design or method selection is missing, which limits the application of the model. In interactive machine comprehension technology based on dynamic knowledge graphs, information fusion is only performed through attention, and how to measure this method is not involved, that is, the design of the model lacks interpretability.

[0047] In order to at least partially solve the above technical problems, in this embodiment, a variety of information fusion methods are combined, and the text features to be fused and the dynamic graph features to be fused are fused through different fusion methods to make the information fusion more complete and maximize the use of the dynamic graph features and the text features to be fused. In addition, based on the information theory method, according to the information difference between each fused feature, the fused features for dynamic graph update are selected, thereby improving the interpretability of the model.

[0048] The first dynamic graph can be a dynamic graph constructed based on existing text information, or a dynamic graph obtained by updating an existing dynamic graph according to the added text information. For the newly added text information, that is, the text information to be fused, the text features to be fused can be obtained by text encoding the text information to be fused, and the first dynamic graph to be fused can be graph encoded to obtain the first dynamic graph features. Here, the timing of graph encoding the first dynamic graph can be performed after the text information to be fused is obtained, or it can be performed after the first dynamic graph is generated or at other times. This is not limited in this embodiment. Here, the dynamic graph is a graph data structure of text information (that is, with text as nodes). In the process of machine reading text, based on the text information, by constructing a dynamic graph, the knowledge in the text information can be used to understand the subsequent text information. At the same time, the subsequent text information can be used as new knowledge to update the dynamic graph to improve the dynamic graph's ability to understand the text information.

[0049] In the interactive machine reading comprehension technology based on dynamic graphs, the update of dynamic graphs is as follows: Figure 3As shown, during the updating process of the dynamic graph, the encoder inputs the current observation (i.e., the text information to be fused) and the current graph state (knowledge) (i.e., the current dynamic graph). The graph encoding (the encoded dynamic graph) and the text encoding (the encoded text information to be fused) are input into the dynamic graph update network (sequential neural network) through information fusion (feature fusion) to update the dynamic graph.

[0050] In the process of updating the dynamic graph, the dynamic graph update network can learn new knowledge in the current observation to realize the update of the dynamic graph. The output of the encoder is the graph knowledge at the next moment (i.e., the dynamic graph at the next moment). After the dynamic graph is updated, it can be decoded by the decoder to obtain the decoded dynamic graph output. The type of the dynamic graph update network can be a neural network, for example, a sequence neural network, etc.

[0051] Optionally, during the dynamic graph update process, other data may be added depending on the dynamic graph update network selected. When a general natural language reconstruction model is used as the dynamic graph update network, the encoder input remains unchanged.

[0052] Step S204: perform feature fusion on the text features to be fused and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features.

[0053] If only one fusion method is used, for example, attention fusion, to fuse the text features to be fused and the dynamic image features, the information of the text features to be fused and the dynamic image features may not be fully utilized. Therefore, the text features to be fused and the dynamic image features to be fused can be fused respectively by multiple fusion methods to fully utilize the information of the text features to be fused and the dynamic image features. In this embodiment, the text features to be fused and the dynamic image features are fused according to each fusion method in a set of preset fusion methods. Here, a set of preset fusion methods may include but are not limited to attention fusion, addition fusion, splicing fusion, neural network fusion and other fusion methods.

[0054] For example, Figure 4 As shown, the graph encoding and text encoding are fused respectively through a variety of fusion methods to obtain information fusion 1, information fusion 2, ..., information fusion m.

[0055] Optionally, after obtaining a group of candidate fusion features, each candidate fusion feature may be fused with the first dynamic image feature and the text feature to be fused, or each candidate fusion feature may be fused with each other to update the group of candidate fusion features.

[0056] Step S206, selecting a group of target fusion features from a group of candidate fusion features according to the information difference between each candidate fusion feature in a group of candidate fusion features, wherein the information difference between each target fusion feature in a group of target fusion features is greater than or equal to a preset information difference threshold.

[0057] The purpose of information fusion is to make full use of the information of each feature and obtain as much information as possible. The differences between the candidate fusion features obtained by different fusion methods are small. Using features with small information difference to update the dynamic graph not only makes the dynamic graph update process cumbersome, but also has little effect on increasing the amount of information of the dynamic graph. Therefore, candidate fusion features with large information difference can be screened out from a group of candidate fusion features to update the dynamic graph, that is, according to the information difference between each candidate fusion feature in a group of candidate fusion features, a group of target fusion features are selected from a group of candidate fusion features, and the information difference between different target fusion features is greater than or equal to the preset information difference threshold.

[0058] Here, information theory can be used to quantify the amount of information. Therefore, the effect of information fusion can be measured by methods in information theory to increase interpretability. The information difference between two fused features can reflect the difference in the amount of information between the two fused features. The greater the difference in the amount of information between the two fused features, the greater the total amount of information after the two fused features are fused. Therefore, the information theory method can be used to determine the information difference between different target fused features.

[0059] Step S208: Use a set of target fusion features to update the first dynamic image to obtain a second dynamic image.

[0060] After obtaining a set of target fusion features, or after at least one round of updating of a set of target fusion features, a set of target fusion features can be used to update the first dynamic graph to obtain a second dynamic graph. Using a set of target fusion features to update the first dynamic graph can be: inputting a set of target fusion features (and the first dynamic graph) into a dynamic graph update model to update the first dynamic graph. The dynamic graph update network in the dynamic graph update model can perform semantic understanding of the target fusion features based on a set of input target fusion features through machine learning and deep learning algorithms, and construct a new dynamic graph to update the first dynamic graph. The dynamic graph update model can be a neural network (for example, a sequence neural network) or other types of natural language reconstruction models, which are not limited in this embodiment.

[0061] Through the above steps S202 to S208, the text features to be fused are obtained by text encoding the text information to be fused, and the dynamic graph features are obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; the text features to be fused and the dynamic graph features are fused according to a set of preset fusion methods to obtain a set of candidate fusion features; according to the information difference between each candidate fusion feature in the set of candidate fusion features, a set of target fusion features is selected from the set of candidate fusion features, wherein the information difference between each target fusion feature in the set of target fusion features is greater than or equal to a preset information difference threshold; the first dynamic graph is updated using a set of target fusion features to obtain a second dynamic graph, which solves the problem of low information utilization rate due to the single information fusion method in the information processing method in the related art, thereby improving the information utilization rate.

[0062] In an exemplary embodiment, a set of target fusion features is selected from a set of candidate fusion features according to the information difference between each candidate fusion feature in a set of candidate fusion features, including:

[0063] S11, determining a relative entropy corresponding to each candidate fusion feature pair in a set of candidate fusion feature pairs according to a set of candidate fusion feature pairs, wherein each candidate fusion feature pair includes two candidate fusion features, and the relative entropy corresponding to each candidate fusion feature pair is the relative entropy between the two candidate fusion features in each candidate fusion feature pair;

[0064] S12, selecting a set of target fusion features from a set of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair.

[0065] Relative entropy is also called KL divergence, which can be used to measure the difference between two distributions. The greater the difference between the two distributions, the greater the difference in the amount of information contained in the two distributions. The calculation formula of KL divergence is shown in formula (1):

[0066]

[0067] Among them, p and q represent information fusion 1 and information fusion 2 respectively (that is, two candidate fusion features in a candidate fusion feature pair), p(x i ),q(x i ) represent the probability distribution of p and q respectively.

[0068] In information fusion, the larger the KL divergence value of the two fused features, the more complementary the two fused features are, that is, the greater the total amount of information that can be provided after the two fused features are fused. Therefore, the KL divergence can be used to measure the difference between the fused features after fusion by different information fusion methods, and the fusion method corresponding to the fused features with larger differences is selected and applied to the model or solution. While improving the utilization rate of information, it increases the interpretability of the fusion method selection, thereby improving the interpretability of the entire model.

[0069] In this embodiment, a group of target fusion features can be selected from a group of candidate fusion features according to the relative entropy between different candidate fusion features in a group of candidate fusion features (i.e., the KL divergence between different candidate fusion features). To this end, the relative entropy between different candidate fusion features can be calculated first. For example, every two candidate fusion features in a group of candidate fusion features can be taken as a candidate fusion feature pair to obtain a group of candidate fusion feature pairs, and the relative entropy corresponding to each candidate fusion feature pair is determined, that is, the relative entropy between the two candidate fusion features in each candidate fusion feature pair is determined, and the information difference between the two candidate fusion features in each candidate fusion feature pair is identified by the relative entropy corresponding to each candidate fusion feature pair.

[0070] According to the relative entropy corresponding to each candidate fusion feature pair, a fusion feature whose information difference with each candidate fusion feature is greater than or equal to a preset information difference threshold can be selected from a group of candidate fusion features as the target fusion feature, and a group of target fusion features is obtained for updating the first dynamic graph.

[0071] For example, Figure 5 As shown, the KL divergence value between each two candidate fusion information is calculated. For two candidate fusion information with a small difference (less than a preset relative entropy threshold), one of them is retained until the differences between the remaining fusion information are large. The preset relative entropy mean can be set according to the actual application, and the number of retained fusion information can also be set.

[0072] Through this embodiment, the information difference between the fused information is measured by KL divergence, and fusion features with a larger information difference are selected, so that the interpretability of information processing can be improved.

[0073] In an exemplary embodiment, a set of target fusion features is selected from a set of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair, including:

[0074] S21, sequentially selecting candidate fusion features from a set of candidate fusion features as current candidate fusion features to perform the following feature selection operations until a set of candidate fusion features is empty:

[0075] When the relative entropy between the current candidate fusion feature and the selected target fusion feature is greater than or equal to a preset relative entropy threshold, the current candidate fusion feature is determined as a target fusion feature, and the current candidate fusion feature is removed from a group of candidate fusion features;

[0076] When the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, the current candidate fusion feature is removed from a set of candidate fusion features.

[0077] In the process of selecting a set of target fusion features from a set of candidate fusion features, the candidate fusion features can be selected in turn as the current candidate fusion features, and it can be determined whether there is a selected target fusion feature. If there is no selected target fusion feature, the current candidate fusion feature can be determined as a target fusion feature, and the current candidate fusion feature can be removed from the set of candidate fusion features. If there is a selected target fusion feature, the following feature selection operations can be performed on the current candidate fusion feature:

[0078] Determine the relative entropy between the current candidate fusion feature and each selected target fusion feature. When the relative entropy between the current candidate fusion feature and each selected target fusion feature is greater than or equal to a preset relative entropy threshold, the current candidate fusion feature can be determined as the target fusion feature, and the current candidate fusion feature is removed from a group of candidate fusion features.

[0079] When the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, that is, when the relative entropy between the current candidate fusion feature and each selected target fusion feature does not all reach the preset threshold, the current candidate fusion feature cannot be selected as the target fusion feature, and the current candidate fusion feature can be removed from a set of candidate fusion features.

[0080] The above feature selection operation is performed on each candidate fusion feature (for example, the above feature selection operation is performed in sequence) until a set of candidate fusion features is empty, and a set of target fusion features can be selected.

[0081] Through this embodiment, by selecting each candidate fusion feature, a candidate fusion feature with a large information difference with other candidate fusion features is selected as a target fusion feature, which can improve the reliability of information processing.

[0082] In an exemplary embodiment, a set of target fusion features is selected from a set of candidate fusion features according to the relative entropy of each candidate fusion feature pair, including:

[0083] S31, determining a group of fused feature pairs to be processed according to the relative entropy corresponding to each candidate fused feature pair, wherein the group of fused feature pairs to be processed are candidate fused feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold;

[0084] S32, taking each to-be-processed fusion feature pair in a group of to-be-processed fusion feature pairs as the current to-be-processed fusion feature pair and performing the following processing operations in turn until the group of to-be-processed fusion feature pairs is empty: selecting a candidate fusion feature to be removed from the current to-be-processed fusion feature pair, removing the candidate fusion feature to be removed from the group of candidate fusion features, and removing the to-be-processed fusion feature pair that contains the candidate fusion feature to be removed from the group of to-be-processed fusion feature pairs; wherein the updated group of candidate fusion features is a group of target fusion features.

[0085] In this embodiment, the candidate fusion features with smaller relative entropy between them and other candidate fusion features can be removed from a group of candidate fusion features, and the remaining candidate fusion features can be used as a group of target fusion features. To this end, according to the relative entropy corresponding to each candidate fusion feature pair, the candidate fusion feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold can be determined as the fusion feature pairs to be processed, that is, a group of fusion feature pairs to be processed, and each fusion feature pair to be processed is used as the current fusion feature pair to be processed to perform the following processing operations:

[0086] A candidate fusion feature to be removed is selected from the current fusion feature pair to be processed, and the candidate fusion feature to be removed is removed from a group of candidate fusion features. Here, the candidate fusion feature to be removed can be any candidate fusion feature in the current fusion feature pair to be processed, that is, only one candidate fusion feature in the candidate fusion feature pair whose corresponding relative entropy is less than a preset relative entropy threshold is retained. Since the candidate fusion feature to be removed has been removed from a group of candidate fusion features, in order to reduce the number of fusion feature pairs to be processed, the fusion feature pairs to be processed that contain the candidate fusion feature to be removed can be removed from a group of fusion feature pairs to be processed.

[0087] When a group of to-be-processed fusion feature pairs is empty, that is, each to-be-processed fusion feature pair in a group of to-be-processed fusion feature pairs is removed, the remaining candidate fusion features in a group of candidate fusion features can be used as a group of target fusion features, that is, the updated group of candidate fusion features can be used as a group of target fusion features, where the relative entropy between each candidate fusion feature in the updated group of candidate fusion features is greater than or equal to a preset relative entropy threshold.

[0088] Through this embodiment, by deleting any one of the two candidate features to be fused whose relative entropy is less than a preset relative entropy threshold, it is possible to ensure that more information is obtained and improve the reliability of information processing.

[0089] In an exemplary embodiment, before updating the first dynamic image using a set of target fusion features, the method further includes:

[0090] S41, performing feature update operations N times or less with a set of target fusion features as the first fusion feature layer, and obtaining (N+1) fusion feature layers, wherein N is a positive integer greater than or equal to 1. When performing a feature update operation, the highest fusion feature layer is the current fusion feature layer, and the fusion features in the current fusion feature layer are the current fusion features:

[0091] Perform feature fusion on each current fusion feature and a reference feature corresponding to each current fusion feature according to at least one fusion method in a set of preset fusion methods to obtain a next fusion feature corresponding to each current fusion feature, wherein the reference feature corresponding to each current fusion feature includes at least one of the following: a text feature to be fused, a dynamic image feature, a fusion feature in a fusion feature layer before the current fusion feature layer, and other current fusion features;

[0092] According to the information difference between the next fusion features corresponding to all current fusion features, the fusion features in the next fusion feature layer of the current fusion feature layer are selected from the next fusion features corresponding to all current fusion features to obtain the next fusion feature layer of the current fusion feature layer.

[0093] In this embodiment, a set of target fusion features can be updated to improve the information difference between the target fusion features. When updating a set of target fusion features, a set of target fusion features obtained by fusing the text features to be fused and the dynamic image features can be used as the first feature fusion layer, and a feature update operation is performed on each feature fusion layer until the (N+1)th fusion feature layer is obtained, where N is a positive integer greater than or equal to 1, and N can be a preset number of feature fusion layers, for example, 4, or other values.

[0094] When performing a feature update operation, the highest fused feature layer can be used as the current fused feature layer, and the fused features in the current fused feature layer can be used as the current fused features. Each current fused feature and the reference feature corresponding to each current fused feature can be fused according to at least one fusion method in a set of preset fusion methods to obtain the next fused feature corresponding to each current fused feature. Here, the reference feature corresponding to each current fused feature can be a text feature to be fused, a dynamic image feature, a fused feature in the fused feature layer before the current fused feature layer, or at least one of other current fused features.

[0095] For example, for each fused feature in the first fused feature layer, feature fusion can be performed with other fused features in the first fused feature layer, or with text features to be fused or dynamic image features; for the second and subsequent fused feature layers, feature fusion can be performed with other fused features in the fused feature layer, or with text features to be fused or dynamic image features, or with fused features in the fused feature layer before the fused feature layer.

[0096] After obtaining the next fusion feature corresponding to all current fusion features, the information difference between the next fusion features corresponding to all current fusion features can be determined, and based on the information difference between the next fusion features corresponding to all current fusion features, the fusion features in the next fusion feature layer of the current fusion feature layer are selected from the next fusion features corresponding to all current fusion features to obtain the next fusion feature layer of the current fusion feature layer. The selection of fusion features based on the information difference between each fusion feature is similar to the above and will not be repeated here.

[0097] In the case that the next fused feature layer of the current fused feature layer is the (N+1)th fused feature layer, the (N+1)th fused feature layer may be used as a set of updated target fused features.

[0098] For example, Figure 6 As shown, feature fusion can be multi-level or cross-level. On the basis of the first-level information fusion, fusion can be performed between the first-level fused information, or the first-level fused information can be fused with the original information (graph encoding or text encoding) to obtain the second-level information fusion, and information fusion can be continued in sequence. Through multi-level or cross-level information fusion, various information fusion methods can be combined to enhance the semantic understanding ability of the dynamic graph update network, thereby achieving the purpose of improving knowledge and reasoning ability.

[0099] Through this embodiment, by performing multi-layer feature fusion and cross-level feature fusion, information can be fully utilized and the reliability of information processing can be improved.

[0100] In an exemplary embodiment, the text features to be fused and the dynamic image features are fused according to a set of preset fusion methods to obtain a set of candidate fusion features, including at least one of the following:

[0101] S51, performing feature fusion processing on the text feature to be fused and the dynamic image feature respectively through each feature fusion network in at least one feature fusion network, and obtaining candidate fusion features corresponding to each feature fusion network;

[0102] S52, after aligning the feature dimensions of the text features to be fused and the dynamic image features, perform feature superposition processing on the text features to be fused and the dynamic image features whose feature dimensions are aligned according to each feature superposition method of at least one feature superposition method to obtain candidate fusion features corresponding to each feature superposition method.

[0103] In this embodiment, for the text features and dynamic image features to be fused, at least one fusion method in a set of preset fusion methods can be used to perform feature fusion to obtain a set of candidate fusion features. The fusion method in the set of preset fusion methods can include at least one of feature fusion through a feature fusion network and feature fusion through feature superposition.

[0104] Feature fusion through the feature fusion network can be performed by selecting at least one feature fusion network from the feature fusion network, inputting the text features and dynamic graph features to be fused into each selected feature fusion network, performing feature fusion processing, and obtaining candidate fusion features corresponding to each feature fusion network.

[0105] Feature fusion through feature superposition can be to first align the feature dimensions of the text features and dynamic image features to be fused, then select at least one feature superposition method from the feature superposition methods, perform feature fusion processing on the text features and dynamic image features to be fused through each selected feature superposition method, and obtain candidate fusion features corresponding to each feature fusion network.

[0106] Through this embodiment, feature fusion is performed through at least one fusion method of a feature fusion network and a feature superposition method, so that the utilization rate of information can be improved.

[0107] In an exemplary embodiment, each feature fusion network in at least one feature fusion network performs feature fusion processing on the text feature to be fused and the dynamic image feature, respectively, to obtain candidate fusion features corresponding to each feature fusion network, including at least one of the following:

[0108] S61, inputting the to-be-fused text features as the first weight matrix of the attention network, and the dynamic graph features as the second weight matrix and the third weight matrix of the attention network into the attention network, to obtain the first candidate fusion feature output by the attention network;

[0109] S62, inputting the text features and dynamic graph features to be fused into the target convolutional neural network to obtain a second candidate fusion feature output by the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network used for feature fusion;

[0110] According to each feature superposition method in at least one feature superposition method, feature superposition processing is performed on the text features to be fused and the dynamic image features whose feature dimensions are aligned, and the candidate fusion features corresponding to each feature superposition method are obtained, including at least one of the following:

[0111] S63, performing a feature addition operation on the text features to be fused and the dynamic image features whose feature dimensions are aligned to obtain a third candidate fusion feature;

[0112] S64, performing a feature concatenation operation on the text features to be fused and the dynamic image features whose feature dimensions are aligned to obtain a fourth candidate fusion feature.

[0113] The feature fusion network may include but is not limited to an attention network and a target convolutional neural network, and at least one of the attention network and the target convolutional neural network may be selected to perform feature fusion processing on the text features to be fused and the dynamic graph features.

[0114] The feature fusion processing of the text features to be fused and the dynamic image features through the attention network can be as follows: the text features to be fused are used as the first weight matrix of the attention network, and the dynamic image features are used as the second weight matrix and the third weight matrix of the attention network. The first candidate fusion feature output by the attention network can be obtained. The attention fusion can be shown as formula (2):

[0115] Attention=softmax((w q ·Q)·(w k ·K) T )·(w v ·V) (2)

[0116] In attention fusion, Q, K, and V represent the query vector, key vector, and value vector, respectively. q 、w k 、w v Represents the weights of Q, K, and V. In this embodiment, Q is the text code (i.e., the text feature to be fused), and K and V are the graph codes (i.e., the dynamic graph features).

[0117] The feature fusion processing of the text features and dynamic image features to be fused by the target convolutional neural network can be as follows: the text features and dynamic image features to be fused are input into the target convolutional neural network, and a second candidate fusion feature output by the target convolutional neural network can be obtained. Here, the type of the target convolutional neural network can include but is not limited to Transformer, convolutional neural network, recurrent neural network, etc.

[0118] The feature superposition method may include but is not limited to feature fusion through feature addition operation or feature splicing and fusion operation. Feature fusion processing can be performed on the fused text features and dynamic image features through at least one of the fusion methods of feature addition operation and feature splicing and fusion operation.

[0119] In the feature fusion by feature addition operation, before the text features to be fused and the dynamic image features to be fused are fused, the feature dimensions of the text features to be fused and the dynamic image features to be fused can be aligned so that the dimensions of the text features to be fused and the dynamic image features to be fused are consistent, and the text features to be fused and the dynamic image features to be fused with the aligned feature dimensions are added to obtain the third candidate fusion feature. The formula of the above addition operation is shown in formula (3):

[0120] sumation=sum(Q,K) (3)

[0121] The above addition operation is to add the corresponding elements in the corresponding dimension, as shown in formula (4):

[0122] q ij +k ij ,i=1,2,…,n,j=1,2,…,m (4)

[0123] Among them, q ij , k ij They are the corresponding elements of the text features and dynamic image features to be fused respectively.

[0124] In the feature fusion through feature splicing operation, after aligning the feature dimensions of the text features to be fused and the dynamic image features, the splicing operation is performed on the text features to be fused and the dynamic image features whose feature dimensions are aligned to obtain the fourth candidate fusion feature. The splicing fusion can be shown as formula (5):

[0125] concatation=concat(Q,K) (5)

[0126] For example, the feature sizes of text encoding and image encoding are aligned, the number of channels of the two can be different, and the text encoding and image encoding are concatenated in the channel dimension.

[0127] Through this embodiment, the text features to be fused and the dynamic image features to be fused are fused in a variety of preset fusion methods, which can improve the utilization rate of information.

[0128] In an exemplary embodiment, before obtaining the text features to be fused obtained by text encoding the text information to be fused, the method further includes:

[0129] S71, when the text information to be fused contains negative words, perform semantic inversion processing on the text information to be fused to obtain updated text information to be fused.

[0130] In the process of updating the dynamic graph, contrastive learning can be used as a strategy for updating the dynamic graph, that is, inputting data from mutually exclusive fields for updating the dynamic graph. Here, the data from mutually exclusive fields is similar to the current observation in content, but contains negative words, and its semantics is opposite to the current observation. If the data from mutually exclusive fields is directly used for updating the dynamic graph, the accuracy of the dynamic graph may be reduced.

[0131] In this embodiment, when the text information to be fused contains negative words, the text information to be fused can be semantically inverted to obtain updated text information to be fused for updating the dynamic graph. It should be noted that when the text information to be fused contains negative words, the dynamic graph update network can be a sequence reconstruction network or a contrastive learning network, and the dynamic graph update network adds the domain knowledge in the text information to be fused to the dynamic graph to obtain an updated dynamic graph.

[0132] Optionally, when the text information to be fused contains negative words, a set of target fusion features obtained after encoding and feature fusion of text information with roughly the same content as the text information to be fused and without negative words and the updated text information to be fused can be input into the dynamic graph update network to update the dynamic graph.

[0133] For example, Figure 7 As shown in the figure, the dynamic graph update process is as follows: the input of the encoder is the observation and the current graph state (knowledge), and the output is the graph knowledge at the next moment. In this process, the implicit knowledge graph (i.e., dynamic graph) learns the new knowledge given by the current observation and updates the knowledge graph. The decoder input is the same as the encoder, but depending on the selected model, other data may be accessed. When a general natural language reconstruction model is used, the input remains consistent, but when contrastive learning is used as a graph update strategy, data from other mutually exclusive fields that are different from the data in this field need to be input. Multiple inputs are continuously enriched through reconstruction or contrastive learning networks to learn domain knowledge in the implicit knowledge graph and stored in the graph network structure.

[0134] Through this embodiment, by performing semantic inversion on text information containing negative words, it is possible to avoid updating the dynamic image with erroneous information, thereby improving the accuracy of information processing.

[0135] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM (Read-Only Memory) / RAM (Random Access Memory), a magnetic disk, or an optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods of each embodiment of the present application.

[0137] According to another aspect of an embodiment of the present application, there is also provided an information processing device for implementing the above-mentioned information processing method, and the information processing device can be applied to a smart device. Figure 8 is a structural block diagram of an optional information processing device according to an embodiment of the present application, such as Figure 8 As shown, the device may include:

[0138] The acquisition unit 802 is used to acquire the text features to be fused obtained by text encoding the text information to be fused, and acquire the dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes;

[0139] A fusion unit 804 is connected to the acquisition unit 802 and is used to perform feature fusion on the fused text features and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features;

[0140] The selection unit 806 is connected to the fusion unit 804 and is used to select a group of target fusion features from a group of candidate fusion features according to the information amount difference between each candidate fusion feature in the group of candidate fusion features, wherein the information amount difference between each target fusion feature in the group of target fusion features is greater than or equal to a preset information amount difference threshold;

[0141] The updating unit 808 is connected to the selecting unit 806, and is used to update the first dynamic image using a set of target fusion features to obtain a second dynamic image.

[0142] It should be noted that the acquisition unit 802 in this embodiment can be used to execute the above step S202, the fusion unit 804 in this embodiment can be used to execute the above step S204, the selection unit 806 in this embodiment can be used to execute the above step S206, and the update unit 808 in this embodiment can be used to execute the above step S208.

[0143] Through the above module, the text features to be fused obtained by text encoding the text information to be fused, and the dynamic graph features obtained by graph encoding the first dynamic graph to be fused are obtained, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; the text features to be fused and the dynamic graph features are fused according to a set of preset fusion methods to obtain a set of candidate fusion features; according to the information difference between each candidate fusion feature in the set of candidate fusion features, a set of target fusion features is selected from the set of candidate fusion features, wherein the information difference between each target fusion feature in the set of target fusion features is greater than or equal to a preset information difference threshold; the first dynamic graph is updated using a set of target fusion features to obtain a second dynamic graph, which solves the problem of low information utilization rate caused by a single information fusion method in the information processing method in the related art, thereby improving the information utilization rate.

[0144] In an exemplary embodiment, the selection unit includes:

[0145] A determination module, used to determine, based on a group of candidate fusion features, a relative entropy corresponding to each candidate fusion feature pair in a group of candidate fusion feature pairs, wherein each candidate fusion feature pair includes two candidate fusion features, and the relative entropy corresponding to each candidate fusion feature pair is a relative entropy between two candidate fusion features in each candidate fusion feature pair;

[0146] The selection module is used to select a set of target fusion features from a set of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair.

[0147] In an exemplary embodiment, the selection module includes:

[0148] The first execution submodule is used to sequentially select candidate fusion features from a set of candidate fusion features as current candidate fusion features to perform the following feature selection operations until a set of candidate fusion features is empty:

[0149] When the relative entropy between the current candidate fusion feature and the selected target fusion feature is greater than or equal to a preset relative entropy threshold, the current candidate fusion feature is determined as a target fusion feature, and the current candidate fusion feature is removed from a group of candidate fusion features;

[0150] When the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, the current candidate fusion feature is removed from a set of candidate fusion features.

[0151] In an exemplary embodiment, the selection module includes:

[0152] A determination submodule is used to determine a group of fusion feature pairs to be processed according to the relative entropy corresponding to each candidate fusion feature pair, wherein a group of fusion feature pairs to be processed are candidate fusion feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold;

[0153] The second execution submodule is used to sequentially use each to-be-processed fusion feature pair in a group of to-be-processed fusion feature pairs as a current to-be-processed fusion feature pair to perform the following processing operations until the group of to-be-processed fusion feature pairs is empty: selecting a candidate fusion feature to be removed from the current to-be-processed fusion feature pair, removing the candidate fusion feature to be removed from the group of candidate fusion features, and removing the to-be-processed fusion feature pair that contains the candidate fusion feature to be removed from the group of to-be-processed fusion feature pairs;

[0154] Among them, the updated set of candidate fusion features is a set of target fusion features.

[0155] In an exemplary embodiment, the above device further comprises:

[0156] The first execution unit is used to perform N or less feature update operations using a set of target fusion features as a first fusion feature layer before updating the first dynamic image using the set of target fusion features to obtain (N+1) fusion feature layers, where N is a positive integer greater than or equal to 1. When performing a feature update operation, the highest fusion feature layer is the current fusion feature layer, and the fusion features in the current fusion feature layer are the current fusion features:

[0157] Perform feature fusion on each current fusion feature and a reference feature corresponding to each current fusion feature according to at least one fusion method in a set of preset fusion methods to obtain a next fusion feature corresponding to each current fusion feature, wherein the reference feature corresponding to each current fusion feature includes at least one of the following: a text feature to be fused, a dynamic image feature, a fusion feature in a fusion feature layer before the current fusion feature layer, and other current fusion features;

[0158] According to the information difference between the next fusion features corresponding to all the current fusion features, the fusion features in the next fusion feature layer of the current fusion feature layer are selected from the next fusion features corresponding to all the current fusion features to obtain the next fusion feature layer of the current fusion feature layer;

[0159] Among them, the (N+1)th fusion feature layer is a set of updated target fusion features.

[0160] In an exemplary embodiment, the fusion unit includes at least one of the following:

[0161] A first processing module is used to perform feature fusion processing on the text feature to be fused and the dynamic image feature respectively through each feature fusion network in at least one feature fusion network, so as to obtain candidate fusion features corresponding to each feature fusion network;

[0162] The second processing module is used to align the feature dimensions of the text features to be fused and the dynamic image features, and then perform feature superposition processing on the text features to be fused and the dynamic image features whose feature dimensions are aligned according to each feature superposition method of at least one feature superposition method to obtain candidate fusion features corresponding to each feature superposition method.

[0163] In an exemplary embodiment, the first processing module includes at least one of the following:

[0164] A first input submodule is used to input the text features to be fused as the first weight matrix of the attention network, and the dynamic graph features as the second weight matrix and the third weight matrix of the attention network into the attention network to obtain the first candidate fusion feature output by the attention network;

[0165] A second processing submodule is used to input the text features and dynamic graph features to be fused into a target convolutional neural network to obtain a second candidate fusion feature output by the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network used for feature fusion;

[0166] The second processing module includes at least one of the following:

[0167] A first execution submodule is used to perform a feature addition operation on the text features to be fused and the dynamic image features that are aligned in feature dimensions to obtain a third candidate fusion feature;

[0168] The second execution submodule is used to perform a feature splicing operation on the text features to be fused and the dynamic image features that are aligned in feature dimensions to obtain a fourth candidate fusion feature.

[0169] In an exemplary embodiment, the above device further comprises:

[0170] The second execution unit is used for performing semantic inversion processing on the text information to be fused before obtaining the text features to be fused obtained by text encoding the text information to be fused, when the text information to be fused contains negative words, to obtain updated text information to be fused.

[0171] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the contents disclosed in the above embodiments. It should be noted that the above modules as part of the device can be run in Figure 1 In the hardware environment shown, it can be implemented by software or by hardware, wherein the hardware environment includes a network environment.

[0172] According to another aspect of the embodiment of the present application, a storage medium is also provided, which can be located on a smart device. Optionally, in this embodiment, the storage medium can be used to execute the program code of any of the above information processing methods in the embodiment of the present application.

[0173] Optionally, in this embodiment, the storage medium may be located on at least one network device among a plurality of network devices in the network shown in the above embodiment.

[0174] Optionally, in this embodiment, the storage medium is configured to store program codes for executing the following steps:

[0175] S1, obtaining text features to be fused obtained by text encoding the text information to be fused, and obtaining dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes;

[0176] S2, performing feature fusion on the fused text features and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features;

[0177] S3, selecting a group of target fusion features from a group of candidate fusion features according to the information amount difference between each candidate fusion feature in the group of candidate fusion features, wherein the information amount difference between each target fusion feature in the group of target fusion features is greater than or equal to a preset information amount difference threshold;

[0178] S4, using a set of target fusion features to update the first dynamic graph to obtain a second dynamic graph.

[0179] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, which will not be described in detail in this embodiment.

[0180] Optionally, in this embodiment, the storage medium may include, but is not limited to, various media that can store program codes, such as a USB flash drive, a ROM, a RAM, a mobile hard disk, a magnetic disk, or an optical disk.

[0181] According to another aspect of an embodiment of the present application, an electronic device for implementing the above-mentioned information processing method is also provided. The electronic device may be a server, a terminal, or a combination thereof.

[0182] Fig. 9 is a structural block diagram of an optional electronic device according to an embodiment of the present application, such as Fig. 9 As shown, it includes a processor 902, a communication interface 904, a memory 906 and a communication bus 908, wherein the processor 902, the communication interface 904 and the memory 906 communicate with each other through the communication bus 908, wherein,

[0183] Memory 906, used to store computer programs;

[0184] The processor 902 is used to implement the following steps when executing the computer program stored in the memory 906:

[0185] S1, obtaining text features to be fused obtained by text encoding the text information to be fused, and obtaining dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes;

[0186] S2, performing feature fusion on the fused text features and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features;

[0187] S3, selecting a group of target fusion features from a group of candidate fusion features according to the information amount difference between each candidate fusion feature in the group of candidate fusion features, wherein the information amount difference between each target fusion feature in the group of target fusion features is greater than or equal to a preset information amount difference threshold;

[0188] S4, using a set of target fusion features to update the first dynamic graph to obtain a second dynamic graph.

[0189] Optionally, the communication bus may be a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig. 9The communication interface is used for communication between the electronic device and other devices.

[0190] The memory may include RAM, or may include non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0191] As an example, the memory 906 may include, but is not limited to, the acquisition unit 802, fusion unit 804, selection unit 806, and update unit 808 in the information processing device. In addition, it may also include, but is not limited to, other module units in the information processing device, which will not be repeated in this example.

[0192] The above-mentioned processor can be a general-purpose processor, which can include but not be limited to: CPU (Central Processing Unit), NP (Network Processor), etc.; it can also be DSP (Digital Signal Processing), ASIC (Application Specific Integrated Circuit), FPGA (Field-Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0193] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0194] It can be understood by those skilled in the art that Fig. 9 The structure shown is for illustration only. The device for implementing the above information processing method may be a terminal device, which may be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, and other terminal devices. Fig. 9 The structure of the electronic device is not limited. Fig. 9 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Fig. 9 Different configurations shown.

[0195] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, which can include: a flash drive, ROM, RAM, a magnetic disk or an optical disk, etc.

[0196] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0197] If the integrated units in the above embodiments are implemented in the form of software functional units and sold or used as independent products, they can be stored in the above computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling one or more computer devices (which may be personal computers, servers, or network devices, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application.

[0198] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0199] In the several embodiments provided in the present application, it should be understood that the disclosed client can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0200] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution provided in this embodiment.

[0201] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically separately, or at least two units may be integrated into one unit. The above integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0202] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for processing information, characterized in that: include: Acquire the text features to be fused obtained by text encoding the text information to be fused, and acquire the dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; Performing feature fusion on the text features to be fused and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features; Selecting a group of target fusion features from the group of candidate fusion features according to the information amount difference between each candidate fusion feature in the group of candidate fusion features, wherein the information amount difference between each target fusion feature in the group of target fusion features is greater than or equal to a preset information amount difference threshold; The first dynamic graph is updated using the set of target fusion features to obtain a second dynamic graph.

2. The method according to claim 1, characterized in that The step of selecting a set of target fusion features from the set of candidate fusion features according to the information amount difference between each candidate fusion feature in the set of candidate fusion features comprises: Determine, according to the set of candidate fusion features, a relative entropy corresponding to each candidate fusion feature pair in a set of candidate fusion feature pairs, wherein each candidate fusion feature pair includes two candidate fusion features, and the relative entropy corresponding to each candidate fusion feature pair is a relative entropy between the two candidate fusion features in each candidate fusion feature pair; A group of target fusion features is selected from the group of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair.

3. The method according to claim 2, characterized in that The selecting a group of target fusion features from the group of candidate fusion features according to the relative entropy corresponding to each candidate fusion feature pair comprises: Select candidate fusion features from the set of candidate fusion features in turn as current candidate fusion features and perform the following feature selection operations until the set of candidate fusion features is empty: When the relative entropy between the current candidate fusion feature and the selected target fusion feature is greater than or equal to a preset relative entropy threshold, the current candidate fusion feature is determined as a target fusion feature, and the current candidate fusion feature is removed from the set of candidate fusion features; In a case where the relative entropy between the current candidate fusion feature and at least one selected target fusion feature is less than a preset relative entropy threshold, the current candidate fusion feature is removed from the set of candidate fusion features.

4. The method according to claim 2, characterized in that: The step of selecting a set of target fusion features from the set of candidate fusion features according to the relative entropy of each candidate fusion feature pair comprises: According to the relative entropy corresponding to each candidate fusion feature pair, a group of fusion feature pairs to be processed is determined, wherein the group of fusion feature pairs to be processed are candidate fusion feature pairs whose corresponding relative entropy is less than a preset relative entropy threshold; Taking each of the to-be-processed fused feature pairs in the set of to-be-processed fused feature pairs as the current to-be-processed fused feature pair, and performing the following processing operations in sequence until the set of to-be-processed fused feature pairs is empty: selecting a candidate fused feature to be removed from the current to-be-processed fused feature pair, removing the candidate fused feature to be removed from the set of candidate fused features, and removing the to-be-processed fused feature pair in the set of to-be-processed fused feature pairs that includes the candidate fused feature to be removed; The updated set of candidate fusion features is the set of target fusion features.

5. The method according to claim 1, characterized in that Before updating the first dynamic graph using the set of target fusion features, the method further includes: The set of target fusion features is used as the first fusion feature layer to perform feature update operations N times or less to obtain (N+1) fusion feature layers, where N is a positive integer greater than or equal to 1. When the feature update operation is performed once, the highest fusion feature layer is the current fusion feature layer, and the fusion features in the current fusion feature layer are the current fusion features: Performing feature fusion on each of the current fusion features and the reference features corresponding to each of the current fusion features according to at least one fusion method in the set of preset fusion methods, to obtain a next fusion feature corresponding to each of the current fusion features, wherein the reference features corresponding to each of the current fusion features include at least one of the following: the text feature to be fused, the dynamic image feature, the fusion feature in the fusion feature layer before the current fusion feature layer, and other current fusion features; According to the information amount difference between the next fusion features corresponding to all the current fusion features, a fusion feature in the next fusion feature layer of the current fusion feature layer is selected from the next fusion features corresponding to all the current fusion features to obtain the next fusion feature layer of the current fusion feature layer; Among them, the (N+1)th fusion feature layer is the updated set of target fusion features.

6. The method according to claim 1, characterized in that The feature fusion of the text features to be fused and the dynamic image features is performed according to a set of preset fusion modes to obtain a set of candidate fusion features, including at least one of the following: By using each feature fusion network in at least one feature fusion network, the text feature to be fused and the dynamic image feature are respectively subjected to feature fusion processing to obtain candidate fusion features corresponding to each feature fusion network; After aligning the feature dimensions of the text features to be fused and the dynamic image features, feature superposition processing is performed on the text features to be fused and the dynamic image features whose feature dimensions are aligned according to each feature superposition method of at least one feature superposition method to obtain candidate fusion features corresponding to each feature superposition method.

7. The method according to claim 6, characterized in that The method of performing feature fusion processing on the text features to be fused and the dynamic image features respectively by each feature fusion network in at least one feature fusion network to obtain candidate fusion features corresponding to each feature fusion network includes at least one of the following: inputting the text features to be fused as the first weight matrix of the attention network, and the dynamic image features as the second weight matrix and the third weight matrix of the attention network into the attention network to obtain a first candidate fusion feature output by the attention network; Inputting the to-be-fused text features and the dynamic graph features into a target convolutional neural network to obtain a second candidate fusion feature output by the target convolutional neural network, wherein the target convolutional neural network is a convolutional neural network for feature fusion; The step of performing feature superposition processing on the text features to be fused and the dynamic image features aligned in feature dimensions according to each feature superposition mode of at least one feature superposition mode to obtain candidate fusion features corresponding to each feature superposition mode includes at least one of the following: performing feature addition operation on the text features to be fused and the dynamic image features aligned in feature dimensions to obtain a third candidate fusion feature; A feature splicing operation is performed on the text feature to be fused and the dynamic image feature whose feature dimensions are aligned to obtain a fourth candidate fusion feature.

8. The method according to any one of claims 1 to 7, characterized in that Before obtaining the text features to be fused obtained by text encoding the text information to be fused, the method further includes: In the case that the text information to be fused contains negative words, a semantic inversion process is performed on the text information to be fused to obtain updated text information to be fused.

9. An information processing device, characterized in that: include: an acquisition unit, used to acquire text features to be fused obtained by text encoding the text information to be fused, and to acquire dynamic graph features obtained by graph encoding the first dynamic graph to be fused, wherein the first dynamic graph is a graph-like data structure constructed with text information as nodes; A fusion unit, used for fusing the text features to be fused and the dynamic image features according to a set of preset fusion methods to obtain a set of candidate fusion features; A selection unit, configured to select a group of target fusion features from the group of candidate fusion features according to the information amount difference between each candidate fusion feature in the group of candidate fusion features, wherein the information amount difference between each target fusion feature in the group of target fusion features is greater than or equal to a preset information amount difference threshold; An updating unit is used to perform a dynamic image update on the first dynamic image using the set of target fusion features to obtain a second dynamic image.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored program, wherein the program executes the method according to any one of claims 1 to 8 when executed.

11. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to execute the method according to any one of claims 1 to 8 through the computer program.

Images