A multi-modal content processing method, apparatus, electronic device, and storage medium

Abstract

An embodiment of the present invention discloses a multi-modal content processing method, apparatus, electronic device, and storage medium. It can obtain the graph structure of the target content in multiple modalities, search for the target graph learning model based on the target model structure description information, aggregate and update the feature information of the nodes through the first fusion unit, perform feature fusion on the nodes of the same target content to obtain the fused graph structure, input the updated graph structure and the fused graph structure into the second fusion unit to make it aggregate and update the feature information of the nodes based on the updated graph structure, perform feature fusion on the nodes of the same target content in the newly updated graph structure and the fused graph structure to obtain the newly fused graph structure, input the newly updated graph structure and the newly fused graph structure into the new second fusion unit for processing to obtain the fused graph structure of the final second fusion unit, and perform content processing after predicting each node in the fused graph structure through the prediction unit. It can improve the accuracy of multi-modal content processing.

Description

Technical Field

[0001] The present invention relates to the field of communication technologies, and in particular, to a multi-modal content processing method, apparatus, electronic device, and storage medium. Background Art

[0002] With the rapid development of the current economy and technology, the scale of complex multi-modal content including multiple modalities such as text, pictures, and videos is gradually expanding. In daily scenarios such as searching and classification, multi-modal information shows an increasingly important role.

[0003] Currently, the main method adopted in processing multi-modal content is to predict the modal information of multiple modalities in a certain multi-modal content to be processed through a model, obtain the prediction results corresponding to the multi-modal content in different modalities, and then process the multi-modal content based on multiple prediction results. However, with this solution, the model can only make judgments based on the information of a certain multi-modal content itself, and the prediction results in different modalities are independent, affecting the accuracy of multi-modal content processing. Summary of the Invention

[0004] Embodiments of the present invention provide a multi-modal content processing method, apparatus, electronic device, and storage medium, which can enrich the available information in processing multi-modal content based on multiple target contents, and improve the accuracy of multi-modal content processing.

[0005] Embodiments of the present invention provide a multi-modal content processing method, including:

[0006] Obtain graph structures of multiple target contents in multiple modalities, where different nodes in one graph structure correspond to feature information of different target contents in the same modality, and the connection relationships of nodes in different graph structures are the same;

[0007] Based on the target model structure description information corresponding to the target content, search for a target graph learning model for processing the graph structure, where the target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit;

[0008] Through the first fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the graph structure, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit;

[0009] Through the current second fusion unit, based on the feature information of each node and its neighboring nodes in the updated graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure. Take the next second fusion unit as the new current second fusion unit, input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, to obtain the fused graph structure output by the last second fusion unit;

[0010] Through the prediction unit, perform prediction on the nodes corresponding to each of the target contents in the fused graph structure, obtain the prediction results of the target contents, and perform content processing based on the prediction results.

[0011] Correspondingly, an embodiment of the present invention further provides a multimodal content processing device, including

[0012] A graph structure acquisition structure, configured to acquire the graph structures of multiple target contents in multiple modalities. Among them, different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same;

[0013] A model search structure, configured to search for a target graph learning model for processing the graph structure based on the target model structure description information corresponding to the target content. The target graph learning model at least includes a first fusion unit, at least one second fusion unit, and a prediction unit;

[0014] A first fusion structure, configured to, through the first fusion unit, based on the feature information of each node and its neighboring nodes in the graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in each updated graph structure, to obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit;

[0015] A second fusion structure, configured to, through the current second fusion unit, based on the feature information of each node and its neighboring nodes in the updated graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure. Take the next second fusion unit as the new current second fusion unit, input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, to obtain the fused graph structure output by the last second fusion unit;

[0016] A prediction structure is used to, through the prediction unit, perform predictions on the nodes corresponding to each of the target contents in the fused graph structure, obtain prediction results of the target contents, and perform content processing based on the prediction results.

[0017] Optionally, the first fusion unit includes a first node aggregation module and a first modality fusion module. The first fusion structure is used to, through the first node aggregation module in the first fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the graph structure, obtain an updated graph structure, and input the updated graph structure into the second fusion unit and the first modality fusion module.

[0018] In the first modality fusion module, feature fusion is performed on the nodes of the same target content in each of the updated graph structures to obtain a fused graph structure, and the fused graph structure is input into the second modality fusion module.

[0019] Optionally, the second fusion unit includes a second node aggregation module and a second modality fusion module. The second fusion structure is used to, through the second node aggregation module in the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, obtain a new updated graph structure, and input the new updated graph structure into the second modality fusion module.

[0020] In the second modality fusion module, feature fusion is performed on the nodes of the same target content in each of the new updated graph structures and the fused graph structure to obtain a new fused graph structure.

[0021] Optionally, the first fusion structure is used to, through a first correlation coefficient calculation function in the first fusion unit, calculate the correlation coefficients between each node and other nodes in the graph structure based on the graph structure.

[0022] Based on the correlation coefficients, determine each node and the neighboring nodes of each node.

[0023] Through a first aggregation function in the first fusion unit, aggregate and update the feature information of each node based on each node and the neighboring nodes of each node.

[0024] Optionally, the second fusion structure is used to, through a second correlation coefficient calculation function in the current second fusion unit, calculate the correlation coefficients between each node and other nodes in the input updated graph structure based on the updated graph structure.

[0025] Based on the correlation coefficients, determine each node and the neighboring nodes of each node.

[0026] Based on the second aggregation function in the second fusion unit, aggregate and update the feature information of each node based on each node and the neighboring nodes of each node.

[0027] Optionally, the multi-modal content processing device provided by the embodiments of the present invention further includes a model training structure for obtaining the model structure information of the graph learning model. The model structure information includes the model structure and at least one optional network module at the positions of each network module in the model structure;

[0028] Perform model structure sampling based on the model structure information to determine the network modules at the network module positions included in each unit of the graph learning model, and obtain at least one graph learning model to be trained. The units include a first fusion unit, a second fusion unit, and a prediction unit;

[0029] Use each graph learning model to be trained to separately predict the sample graph structure, and obtain the first training prediction results corresponding to each graph learning model to be trained. The sample graph structure is labeled with a reference prediction result;

[0030] Determine the target graph learning model to be trained from each graph learning model to be trained according to the reference prediction result and each first training prediction result;

[0031] Use the target graph learning model to be trained to predict the sample graph structure, and obtain a second training prediction result;

[0032] Calculate the training loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result;

[0033] Adjust the model parameters of the target graph learning model to be trained based on the training loss to obtain the trained target graph learning model, and record the target model structure description information of the target graph learning model;

[0034] The model search structure is used to determine the target model structure description information corresponding to the target content based on the target content;

[0035] Search for the target graph learning model for processing the graph structure from the model structure information according to the target model structure description information.

[0036] Optionally, the multi-modal content processing device provided by the embodiments of the present invention further includes a model update structure for updating the sample graph structure according to the graph structures of the target content in multiple modalities and the prediction results of the target content to obtain a new sample graph structure;

[0037] Return to the step of performing model structure sampling based on the model structure information, determining the network modules at the positions of the network modules included in each unit in the graph learning model, and obtaining at least one graph learning model to be trained.

[0038] Optionally, the model parameters of the target graph learning model to be trained include target shared model parameters, and the target shared model parameters are at least part of the global shared model parameters shared by each graph learning model sampled from the model structure information;

[0039] The model training structure is used to calculate the training result loss of the target graph learning model to be trained according to the reference prediction result and each of the second training prediction results;

[0040] Based on a preset loss conversion rule, convert the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained;

[0041] Based on the training result loss and the training parameter loss, obtain the training loss of the target graph learning model to be trained.

[0042] Optionally, the model training structure is used to perform group sparse regularization on the target shared model parameters in the target graph learning model to be trained based on a preset loss conversion rule, and obtain a first group sparse loss value;

[0043] Obtain the historical target shared model parameters of the target graph learning model to be trained;

[0044] Calculate the difference between the target shared model parameters and the historical target shared model parameters, and perform group sparse regularization on the product of the difference and the transposed matrix of the historical target shared model parameters to obtain a second group sparse loss value;

[0045] Based on preset first group sparse weights and second group sparse weights, perform weighted calculation on the first group sparse loss value and the second group sparse loss value to obtain the training parameter loss.

[0046] Optionally, the multi-modal content processing device provided in the embodiments of the present invention further includes a global parameter update structure, which is used to obtain the target model parameters of the trained target graph learning model;

[0047] Based on the target model parameters, update the global shared model parameters.

[0048] Optionally, the first fusion structure is used to perform feature mapping on the feature information of each node in each updated graph structure to obtain a mapped graph structure including the mapped nodes;

[0049] Determine the fusion weights of each of the mapped nodes based on the feature information of the mapped nodes in each of the mapped graph structures;

[0050] Perform feature fusion on the mapped nodes of the same target content in each of the mapped graph structures according to the feature information of each of the mapped nodes and each of the fusion weights to obtain a fused graph structure.

[0051] Optionally, the prediction structure is used to perform operations on the feature information of the nodes corresponding to each of the target contents in the fused graph structure respectively through the activation function in the prediction unit to obtain the operation results corresponding to each of the target contents;

[0052] Perform probability prediction processing on each of the operation results through the probability prediction function in the prediction unit to obtain the prediction scores corresponding to each of the target contents;

[0053] Determine the prediction categories corresponding to each of the prediction scores as the prediction results of each of the target contents based on a preset prediction result mapping relationship and each of the prediction scores.

[0054] Optionally, the graph structure acquisition structure is used to acquire multiple target contents, perform modal feature extraction on each of the target contents respectively to obtain the feature information of each of the target contents in multiple modalities;

[0055] Determine the association relationships between each of the target contents;

[0056] Based on the association relationships, use the feature information of different target contents in the same modality as graph nodes respectively to construct the graph structure of the target contents in multiple modalities.

[0057] Correspondingly, an embodiment of the present invention further provides an electronic device, including a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in any one of the multi-modal content processing methods provided by the embodiments of the present invention.

[0058] Correspondingly, an embodiment of the present invention further provides a computer-readable storage medium, the computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in any one of the multi-modal content processing methods provided by the embodiments of the present invention.

[0059] In addition, an embodiment of the present invention further provides a computer program product, including a computer program or instruction, and when the computer program or instruction is executed by a processor, the steps in any one of the multi-modal content processing methods provided by the embodiments of the present invention are implemented.

[0060] By adopting the solution of the embodiment of the present invention, graph structures of multiple target contents in multiple modalities can be obtained. Among them, different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same. Based on the target model structure description information corresponding to the target content, a target graph learning model for processing the graph structure is searched. The target graph learning model at least includes a first fusion unit, at least one second fusion unit, and a prediction unit. Through the first fusion unit, based on each node and its neighboring nodes in the graph structure, the feature information of each node is aggregated and updated, and the feature information of the nodes of the same target content in each updated graph structure is fused to obtain a fused graph structure. The updated graph structures and the fused graph structure are input into the second fusion unit. Through the current second fusion unit, based on each node and its neighboring nodes in the updated graph structure, the feature information of each node is aggregated and updated, and the feature information of the nodes of the same target content in the newly updated graph structure and the fused graph structure is fused to obtain a new fused graph structure. The next second fusion unit is used as the new current second fusion unit, and the newly updated graph structure and the new fused graph structure are input into the new current second fusion unit for processing to obtain the fused graph structure output by the last second fusion unit. Through the prediction unit, predictions are made for the nodes corresponding to each target content in the fused graph structure to obtain the prediction results of the target content, and content processing is performed based on the prediction results. Since in the embodiment of the present invention, graph structures of multiple target contents in multiple modalities are obtained, in addition to the information of the target content itself, the target graph learning model can also use the correlation relationships between the target contents for prediction, enriching the available information when processing multi-modal content. In addition, in the embodiment of the present invention, the feature information of the same target content in different modalities is updated, fused, and then predicted, so that the overall prediction result can be obtained by integrating the information of different modalities instead of independently predicting the information in different modalities. Therefore, the target graph learning model can obtain the feature information of the nodes containing the multi-modal information of the target content itself and the correlation relationship information between the target contents for prediction during prediction, which can improve the accuracy of multi-modal content processing. Description of the Drawings

[0061] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the following drawings are only some embodiments of the present invention. For those skilled in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0062] Figure 1 It is a schematic diagram of the scenario of the multi-modal content processing method provided by the embodiment of the present invention;

[0063] Figure 2 is a flowchart of the multi-modal content processing method provided by an embodiment of the present invention;

[0064] Figure 3 is a schematic topological diagram of the model structure information provided by an embodiment of the present invention;

[0065] Figure 4 is a schematic flowchart of multi-modal continuous graph learning provided by an embodiment of the present invention;

[0066] Figure 5 is a schematic diagram of performing different tasks on multi-modal content provided by an embodiment of the present invention;

[0067] Figure 6 is another flowchart of the multi-modal content processing method provided by an embodiment of the present invention;

[0068] Figure 7 is a schematic structural diagram of the multi-modal content processing device provided by an embodiment of the present invention;

[0069] Figure 8 is another schematic structural diagram of the multi-modal content processing device provided by an embodiment of the present invention;

[0070] Figure 9 is a schematic structural diagram of the electronic device provided by an embodiment of the present invention. Detailed implementation manners

[0071] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts belong to the scope of protection of the present invention.

[0072] An embodiment of the present invention provides a multi-modal content processing method, device, electronic device and computer-readable storage medium. Specifically, an embodiment of the present invention provides a multi-modal content processing method applicable to a multi-modal content processing device, and the multi-modal content processing device can be integrated in an electronic device.

[0073] The electronic device can be a device such as a terminal, including but not limited to a mobile terminal and a fixed terminal. For example, the mobile terminal includes but not limited to a smart phone, a smart watch, a tablet computer, a notebook computer, a smart vehicle, etc. Among them, the fixed terminal includes but not limited to a desktop computer, a smart TV, etc.

[0074] The electronic device may also be a device such as a server. The server may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It may also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms, but is not limited thereto.

[0075] The multi-modal content processing method according to the embodiments of the present invention may be implemented by a server, or jointly implemented by a terminal and a server.

[0076] Taking the joint implementation of the multi-modal content processing method by a terminal and a server as an example, the method will be described below.

[0077] As Figure 1 shown, the multi-modal content processing system provided by the embodiments of the present invention includes a terminal 10, a server 20, etc.; the terminal 10 is connected to the server 20 through a network, for example, through a wired or wireless network connection, etc. Among them, the terminal 10 may exist as a terminal for a user to send or view target content that can be processed to the server 20.

[0078] Among them, the terminal 10 may be a terminal for a user to upload target content to be processed, and is used to send the target content to be processed to the server 20.

[0079] The server 20 may be used to obtain the graph structures of multiple target contents in multiple modalities. Among them, different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same. Based on the target model structure description information corresponding to the target content, a target graph learning model for processing the graph structure is searched. The target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit.

[0080] The server 20 can, through the first fusion unit in the target graph learning model, aggregate and update the feature information of each node in the graph structure based on each node and its neighboring nodes in the graph structure, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, input each updated graph structure and the fused graph structure into the second fusion unit, and through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, obtain a new fused graph structure, use the next second fusion unit as the new current second fusion unit, input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, obtain the fused graph structure output by the last second fusion unit, and through the prediction unit, perform prediction on the nodes corresponding to each target content in the fused graph structure to obtain the prediction result of the target content.

[0081] The server 20 can perform content processing on the target content based on the prediction result.

[0082] It can be understood that in some embodiments, the steps of the multimodal content processing executed by the server 20 can also be executed by the terminal 10, and the embodiments of the present invention do not limit this.

[0083] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the preferred order of the embodiments.

[0084] The embodiments of the present invention will be described from the perspective of a multimodal content processing device, and the multimodal content processing device can be specifically integrated in a server or a terminal.

[0085] As Figure 2 shown, the specific process of the multimodal content processing method in this embodiment can be as follows:

[0086] 201. Obtain the graph structures of multiple target contents in multiple modalities, where different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same.

[0087] Among them, the target content can be content such as an article, an image, or a video, or content composed of information in a web page.

[0088] Specifically, different modalities correspond to different information carriers when the user obtains information. For example, the target content can include at least one of modalities such as text, pictures, audio, or video.

[0089] It should be noted that the forms of different target contents can be the same or different; the corresponding modalities in different target contents can be the same or different.

[0090] For example, among the multiple target contents obtained, there are target content 1, target content 2, and target content 3. Among them, target content 1 can be an article, and the article includes contents in two modalities: text and image; target content 2 can be an image, and the image includes contents in two modalities: text and image; target content 3 can be an article, and the article includes contents in three modalities: text, image, and video, and so on.

[0091] In the embodiments of the present invention, each graph structure corresponds to one modality, and each node in the graph structure corresponds to a target content. Among them, the connection relationship between the nodes actually indicates the relevance between the target contents. Therefore, even for graph structures in different modalities, the node connection relationships therein are the same.

[0092] Specifically, when generating the graph structure, the nodes in the graph structure can be determined according to the feature information of the target content in different modalities, and the connection relationship of the nodes in the graph structure can be determined according to the association relationship between the target contents. That is to say, the step of "obtaining the graph structures of multiple target contents in multiple modalities" can include:

[0093] Obtain multiple target contents, respectively perform modality feature extraction on each target content, and obtain the feature information of each target content in multiple modalities;

[0094] Determine the association relationship between each target content;

[0095] Based on the association relationship, respectively use the feature information of different target contents in the same modality as graph nodes, and construct the graph structures of the target contents in multiple modalities.

[0096] Among them, performing modality feature extraction on each target content can be to parse each target content to obtain the sub - contents corresponding to each target content in different modalities, and perform feature extraction on each sub - content to obtain the feature information of each target content in multiple modalities.

[0097] For example, taking the target content as a multimedia article, the text, pictures, videos, or audios in the multimedia article can be respectively extracted, and then feature extraction is respectively performed on the text, pictures, videos, or audios to obtain the feature information corresponding to the multimedia article in the modalities of text, picture, video, or audio.

[0098] For another example, taking the target content as a video, feature extraction can be respectively performed on the subtitles, audio, and video frame sequence in the video to obtain the feature information corresponding to the video in the modalities of text, audio, and video.

[0099] It can be understood that if a certain target content lacks the content corresponding to a certain modality, in the graph structure of that modality, the node corresponding to the target content can be represented by an empty feature or a feature with a specific value.

[0100] Specifically, the association relationship between target contents can be obtained based on the interaction between the user and the target contents and the social relationship of the user. For example, the target contents can establish an association relationship according to interactions such as being clicked, read, or commented on by the same user. Or, different target contents can be interacted with by different users, and there is a friendship relationship between the different users. Or, different target contents can be created by the same creator, and so on.

[0101] Specifically, when constructing the graph structure, the connection relationship between different graph nodes can be determined according to the type of the association relationship between target contents and the strength of the association relationship between target contents.

[0102] For example, if there are 3 target contents that have been interacted with by the same user, a connection relationship can be established between the graph nodes corresponding to the 3 target contents. Further, the edge weight corresponding to the connection relationship can be determined according to the time interval between the interaction moments of the user with the 3 target contents, and so on.

[0103] It can be understood that the graph structures of multiple target contents in multiple modalities can also be pre-processed for the target contents, and can be directly obtained during multi-modal content processing to improve the efficiency of multi-modal content processing. The specific pre-processing process is similar to the foregoing process, and will not be elaborated herein in the embodiments of the present invention.

[0104] 202. Based on the target model structure description information corresponding to the target content, a target graph learning model for processing the graph structure is searched, and the target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit.

[0105] In the embodiments of the present invention, there can be at least one model structure description information, and each model structure description information describes a graph learning model, and specifically can describe the model structure of the graph learning model and the corresponding network modules at the positions of the respective network modules in the model structure.

[0106] In some examples, each model structure description information can correspond to a multi-modal content processing task. At this time, the target model structure description information corresponding to the target content can be determined from the model structure description information according to the processing task corresponding to the target content.

[0107] In some other examples, each model structure description information may correspond to a reference graph structure. At this time, the target model structure description information corresponding to the target content can be determined from the model structure description information according to the similarity degree between the graph structure of the target content and each reference graph structure.

[0108] Alternatively, based on the multi-modal information and the graph structure corresponding to the target content, the target model structure description information suitable for processing the target content can be searched from the model structure description information.

[0109] In the actual application process, the graph learning model can be obtained through pre-training. Through the pre-training process, the parameters in the graph learning model can be adjusted so that the graph learning model can achieve better multi-modal content prediction performance. Therefore, before step 202, the multi-modal content processing method provided by the embodiments of the present invention may further include:

[0110] Obtain the model structure information of the graph learning model, where the model structure information includes the model structure and at least one optional network module at the positions of each network module in the model structure;

[0111] Based on the model structure information, perform model structure sampling to determine the network modules at the positions of the network modules included in each unit in the graph learning model, and obtain at least one graph learning model to be trained. The units include a first fusion unit, a second fusion unit, and a prediction unit;

[0112] Through each graph learning model to be trained, respectively predict the sample graph structure to obtain the first training prediction results corresponding to each graph learning model to be trained. The sample graph structure is marked with a reference prediction result;

[0113] According to the reference prediction result and each first training prediction result, determine the target graph learning model to be trained from each graph learning model to be trained;

[0114] Use the target graph learning model to be trained to predict the sample graph structure to obtain the second training prediction result;

[0115] According to the reference prediction result and each second training prediction result, calculate the training loss of the target graph learning model to be trained;

[0116] Based on the training loss, adjust the model parameters of the target graph learning model to be trained to obtain the trained target graph learning model.

[0117] Among them, the model structure information is information that provides a model structure, optional network modules, etc. that can be shared by each graph learning model. In the embodiments of the present invention, the model structure of the model structure information can be composed of several network modules with connection relationships such as a first fusion unit, at least one second fusion unit, and a prediction unit.

[0118] Specifically, the optional network modules are modules that can be used to form a graph learning model at each network module position. For example, as Figure 3 shown, the first fusion unit 1, the first fusion unit 2, and the first fusion unit 3 are the optional network modules corresponding to the network module position of the first fusion unit. The second fusion unit 1 and the second fusion unit 2 are the optional network modules corresponding to the network module position of the second fusion unit.

[0119] For ease of understanding, as Figure 3 shown, the model structure information can be represented as a graph with a fixed topology. Each node in the graph represents an optional network module at a network module position, and each edge represents the information flow between the optional network modules. The nodes and edges can represent the model structure. Each optional network module represented by a node has its own module parameters, and these parameters are only used when a specific calculation is active.

[0120] Through model structure sampling, at least one sub-model corresponding to the model result information can be formed as a graph learning model according to the model structure and optional network modules in the model structure information. For example, taking the model structure information shown in Figure 3 as an example, graph learning models such as "the first fusion unit 1 → the second fusion unit 1 → the prediction unit", "the first fusion unit 2 → the second fusion unit 1 → the prediction unit", and "the first fusion unit 2 → the second fusion unit 2 → the second fusion unit 1 → the prediction unit" can be obtained.

[0121] It should be noted that Figure 3 is only a schematic diagram shown for ease of describing the model structure information. The topology graph of the model structure information can actually be a topology graph with richer Figure 3 content.

[0122] In the actual application process, a large number of graph learning models to be trained can be obtained through model structure sampling. If each graph learning model to be trained is trained, it will consume a large amount of computing resources. Therefore, one or N graph learning models to be trained with the best initial performance can be selected first according to the prediction performance of each graph learning model to be trained on the sample graph structure, that is, the target graph learning models to be trained.

[0123] Furthermore, the target graph learning models to be trained are trained so that the target graph learning models can achieve the best multi-modal content processing effect.

[0124] Generally, in related technologies, a model with a fixed structure is used to process all multimodal content. However, a large amount of multimodal content stems from real-world events that actually occur or the random inspiration of creators, rather than existing fixed content. Therefore, multimodal content is ever-changing, and its statistical characteristics are also constantly changing and difficult to predict. Using a fixed model to process constantly changing multimedia content often results in less than ideal processing effects.

[0125] In an embodiment of the present invention, an architecture-evolving multimodal graph continuous learning method is proposed. An automatic architecture search method is used to dynamically adjust the model architecture during the continuous learning process, so that the sub-structures of the model can continuously adapt to new data distributions. That is, the multimodal content processing method provided by the embodiment of the present invention may further include:

[0126] Updating the sample graph structure according to the graph structure of the target content in multiple modalities and the prediction result of the target content to obtain a new sample graph structure;

[0127] Returning to the step of performing model structure sampling based on the model structure information to determine the network modules at the positions of the network modules included in each unit of the graph learning model, and obtaining at least one graph learning model to be trained.

[0128] Among them, for updating the sample graph structure, it may be to reconstruct a new sample graph structure according to the target content and the content corresponding to the original sample graph structure.

[0129] Alternatively, each sample graph structure may correspond to the processing time. Updating the sample graph structure may be to directly add the graph structure of the target content and the processing time of the target content to the sample graph structure, so that the graph learning model can perform continuous learning according to the processing times of different sample graph structures.

[0130] In some embodiments, as the multimodal content continues to change, the model structure and model parameters of the target graph learning model may also change continuously. If each target graph learning model is saved, it will occupy a large storage space. Therefore, the model information of each target graph learning model can be recorded, and when needed, the corresponding part can be activated from the model structure information for use to save the storage space of the model. Therefore, the multimedia content processing method provided by the embodiment of the present invention may further include:

[0131] Recording the target model structure description information of the target graph learning model;

[0132] Correspondingly, the step of "searching for a target graph learning model for processing the graph structure based on the target model structure description information corresponding to the target content" may specifically include:

[0133] Based on the target content, determine the target model structure description information corresponding to the target content;

[0134] According to the target model structure description information, search in the model structure information to obtain a target graph learning model for processing the graph structure.

[0135] Take Figure 3 as an example. If the target model structure description information describes the model structure of a graph learning model of "First Fusion Unit 1 → Second Fusion Unit 1 → Prediction Unit", then the First Fusion Unit 1, Second Fusion Unit 1, and Prediction Unit can be determined from the model structure information and connected to obtain the target graph learning model.

[0136] In some alternative embodiments, the training loss of the target graph learning model to be trained can be calculated according to mean square error, cross entropy, relative entropy, or the like.

[0137] In some other alternative embodiments, in order to relieve the storage pressure of storing the parameters of different optional network modules in the model structure information, the embodiments of the present invention design a sharing strategy to share parameters among the same optional network modules in different graph learning models.

[0138] Therefore, when training the graph learning model, in addition to the sample graph structure and its reference prediction result, the training can also be combined with the shared parameters. That is to say, the model parameters of the target graph learning model to be trained include target shared model parameters. The step of "calculating the training loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result" can specifically include:

[0139] Calculate the training result loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result;

[0140] Based on a preset loss conversion rule, convert the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained;

[0141] Based on the training result loss and the training parameter loss, obtain the training loss of the target graph learning model to be trained.

[0142] Among them, the target shared model parameters are at least part of the global shared model parameters shared by each graph learning model sampled from the model structure information. The global shared model parameters are the parameters that can be shared by each graph learning model in the model structure information.

[0143] In the actual application process, the target shared model parameters have a certain impact on the prediction accuracy of the target graph learning model. Therefore, by setting a loss transformation rule, the impact of the target shared model parameters on the prediction accuracy can be transformed into a specific form of training parameter loss for representation.

[0144] For example, the training loss can be expressed by the following formula:

[0145]

[0146] where is the training loss, is the reference prediction result (i.e., the actual label) corresponding to each node in the sample graph structure, is the second training prediction result corresponding to each node, and N is the batch size (i.e., the number of nodes in the sample graph structure).

[0147] where is the training parameter loss. Through the training parameter loss, the adjustment of the parameters of the target graph learning model to be trained can be constrained during the training process.

[0148] Specifically, the step of "transforming the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained based on a preset loss transformation rule" may include:

[0149] Performing group sparse regularization on the target shared model parameters in the target graph learning model to be trained based on a preset loss transformation rule to obtain a first group sparse loss value;

[0150] Obtaining the historical target shared model parameters of the target graph learning model to be trained;

[0151] Calculating the difference between the target shared model parameters and the historical target shared model parameters, and performing group sparse regularization on the product of the difference and the transposed matrix of the historical target shared model parameters to obtain a second group sparse loss value;

[0152] Performing weighted calculation on the first group sparse loss value and the second group sparse loss value based on preset first group sparse weights and second group sparse weights to obtain the training parameter loss.

[0153] where the training parameter loss can be expressed by the following formula:

[0154]

[0155] where λ b is the first group sparse weight, λ o is the second group sparse weight, is the first group sparse loss value, is the second group of sparse loss values.

[0156] Specifically, the first group of sparse loss values can be calculated by the following formula:

[0157]

[0158] The second group of sparse loss values can be calculated by the following formula:

[0159]

[0160] Where W is the target shared model parameter of the target graph learning model to be trained during the current training, and W ref is the historical target shared model parameter obtained during the historical training process of the target graph learning model to be trained, is the global shared model parameter.

[0161] Where ψ ref = W - W ref , and ψ ref is a learnable parameter, which is group sparse and orthogonal to Based on ψ ref , the adjustment of the parameters of the target graph learning model to be trained during the training process can be restricted, aiming to search for target shared model parameters that are both group sparse and orthogonal to the historical target shared model parameters.

[0162] For example, when training the target graph learning model for a new task T t , it can be checked whether there are shared parameters corresponding to the network modules in the global shared model parameters of the model structure information of each network module in the target graph learning model. If so, calculate the first group of sparse loss values and the second group of sparse loss values to use group sparse regularization and orthogonal constraints to learn the parameters in the target graph learning model; if not, the training parameter loss can be calculated only according to the first group of sparse loss values, that is, only use group sparse regularization to restrict the learning of the model parameters.

[0163] In the embodiments of the present invention, by limiting that the target shared model parameter needs to be orthogonal to the historical model shared model parameter, the learning of the target shared model parameter can be associated with the knowledge learned in the past, and the ability of the model to learn historical tasks can be maintained. The finally obtained target graph learning model performs well on the new task and at the same time retains the memory ability for historical tasks.

[0164] Optionally, in order to enable the modal structure information to continuously adapt to the changing target content in actual applications, the global shared model parameters in the modal structure information can be continuously updated, so that the graph learning model sampled from the model structure information has model parameters adapted to the new data. Therefore, the multi-modal content processing method provided by the embodiments of the present invention may further include:

[0165] Obtain the target model parameters of the trained target graph learning model;

[0166] Update the global shared model parameters based on the target model parameters.

[0167] The multi-modal content processing solution involved in this embodiment focuses on saving information such as the model structure and model parameters of different graph learning models with modal structure information that can be shared among different models. While maintaining the realizability of the graph learning model, it also makes the overall model parameters to be saved less and more convenient for deployment.

[0168] Such as Figure 4 As shown, the multi-modal continuous graph learning framework based on neural architecture search includes three stages: data processing, neural architecture search, and shared model training and maintenance. In the data processing stage, the original data is processed to construct a multi-modal graph neural network. In the model architecture search stage, the neural structure and parameters found can not only remember past knowledge but also perform well in new tasks. In the shared model training stage, the best structure found in the neural architecture search stage is retrained. In the maintenance stage, some parameters in the outdated blocks of the old architecture are explicitly deleted, and new blocks are used to construct the model, and these new blocks save the knowledge from new tasks and historical tasks.

[0169] 203. Through the first fusion unit, based on each node and its neighboring nodes in the graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in each updated graph structure to obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit.

[0170] Among them, the first fusion unit is a neural network unit in the target graph learning model that only inputs the graph structure corresponding to the target content.

[0171] In some alternative embodiments, the first fusion unit can be implemented based on a Transformer model.

[0172] In other embodiments, such as Figure 4 As shown, the first fusion unit may include a first node aggregation module and a first modal fusion module, and step 203 may specifically include:

[0173] Through the first node aggregation module in the first fusion unit, based on each node in the graph structure and its neighboring nodes, aggregate and update the feature information of each node to obtain an updated graph structure, and input the updated graph structure into the second fusion unit and the first modality fusion module;

[0174] In the first modality fusion module, perform feature fusion on the nodes of the same target content in each updated graph structure to obtain a fused graph structure, and input the fused graph structure into the second modality fusion module.

[0175] Among them, the first node aggregation module is used to aggregate and update the feature information of each node in the graph structure. Specifically, the step "through the first fusion unit, based on each node in the graph structure and its neighboring nodes, aggregate and update the feature information of each node" includes:

[0176] Through the first correlation coefficient calculation function in the first fusion unit, based on the graph structure, calculate the correlation coefficient between each node and other nodes in the graph structure;

[0177] According to the correlation coefficient, determine each node and its neighboring nodes;

[0178] Through the first aggregation function in the first fusion unit, based on each node and its neighboring nodes, aggregate and update the feature information of each node.

[0179] For example, taking the graph structures corresponding to the target content in the text modality and the image modality as an example, the first node aggregation module can be represented by the following formula:

[0180]

[0181] Among them, Aggregate(·) is the first aggregation function, is the first correlation coefficient calculation function for calculating the correlation coefficient between a node and its neighboring nodes, is the network parameter of the first node aggregation module, is the input feature, N(u) is the receptive field of node u (i.e., the set composed of the neighbor nodes of node U). v represents the image modality, and m represents the text modality.

[0182] Alternatively, for the convenience of calculation, the first node aggregation module can be represented by the following formula:

[0183]

[0184] Among them, σ is the activation function.

[0185] For example, under the network module at the network module position of the first fusion unit, there may be a first associated calculation sub-module, a first aggregation operation sub-module, and a first activation sub-module. At least one optional network sub-module may correspond to different sub-module positions.

[0186] Specifically, the operation functions in different optional network sub-modules can be as shown in Table 1:

[0187]

[0188] Table 1

[0189] That is, the first aggregation operation sub-module can have four optional network sub-modules, and the aggregation functions applied in these four optional network sub-modules can be sum, mean, max, and mlp respectively. The first activation sub-module can have 7 optional network sub-modules, and the first associated calculation sub-module can have 7 optional network sub-modules.

[0190] It should be noted that the functions in Table 1 are only examples. In the actual application process, the functions corresponding to each sub-module can include but are not limited to the operation methods shown in Table 1. The embodiments of the present invention do not limit this.

[0191] It can be understood that Figure 4 although it is shown in that there are two first node aggregation modules in the first fusion unit to process the graph structures of two modalities respectively, however, there can also be only one first node aggregation module in the first fusion unit to process the graph structures of multiple modalities in sequence.

[0192] That is to say, the number of first node aggregation modules can be the same as the number of modalities to be processed, or can be different from the number of modalities to be processed. In some examples, the number of first node aggregation modules can be greater than the number of modalities to be processed. For example, according to the processing speed of the first node aggregation module for the graph structures of different modalities, different numbers of node aggregation modules can be set to process the graph structures of different modalities. For example, if the node aggregation update speed of the first node aggregation module for the text modality is greater than that for the image modality, at this time, two interconnected first node aggregation modules can be used to process the graph structure of the text modality, and one first node aggregation module can be used to process the graph structure of the image modality.

[0193] It should be noted that the aggregation functions, activation functions, and related functions used in each first node aggregation module in the first aggregation unit can be completely the same or at least one of them can be different. The embodiments of the present invention do not limit this.

[0194] Optionally, the first modality fusion module for fusing the nodes of the same target content can be represented by the following formula:

[0195]

[0196] Among them, is a node in the fused graph structure, and are nodes corresponding to the same target content in the updated graph structures of different modalities.

[0197] The FUSION operation can be to directly add or multiply the feature information corresponding to the nodes of the same target content for fusion processing. Or, the step of "performing feature fusion on the nodes of the same target content in each updated graph structure to obtain a fused graph structure" can specifically include:

[0198] Performing feature mapping on the feature information of each node in each updated graph structure to obtain a mapped graph structure containing the mapped nodes;

[0199] Based on the feature information of the mapped nodes in each mapped graph structure, determining the fusion weights of each mapped node;

[0200] According to the feature information of each mapped node and each fusion weight, performing feature fusion on the mapped nodes of the same target content in each mapped graph structure to obtain a fused graph structure.

[0201] For example, the feature mapping process of obtaining the mapped nodes can be expressed by the following formula:

[0202]

[0203] Among them, is the mapped node, is a node in the updated graph structure, is the node corresponding deviation vector, is a trainable weight matrix. FC represents performing feature mapping on the feature information of each node, such as mapping through a fully connected layer, etc.

[0204] Again for example, the fusion weights of each mapped node can be calculated based on the cross-attention mechanism:

[0205] Among them, is the fusion weight of each mapped node, and are the mapped nodes corresponding to the same target content in each mapped graph structure.

[0206] A modality may contain non-informative or even misleading information. In this case, directly performing modality fusion may result in negative message passing. The cross-attention mechanism can be used to filter out the non-informative and misleading components from weak modalities.

[0207] Alternatively, the fusion weights of the mapped nodes can be calculated based on the co-attention mechanism:

[0208]

[0209] Different from the cross-attention mechanism, the co-attention mechanism jointly considers text attention and image attention, and thus can model visual information and text information simultaneously.

[0210] Furthermore, based on the feature information of each mapped node and each fusion weight, the weighted feature information corresponding to each mapped node can be calculated:

[0211]

[0212] where is the weighted feature information.

[0213] Fuse the weighted feature information corresponding to the same target content in different mapped graph structures:

[0214] [[ID=2,]]

[0215] That is, the feature information corresponding to each node in the fused graph structure is obtained

[0216] Specifically, the operation functions in different optional network modules corresponding to the first modality fusion module can be as shown in Table 2:

[0217]

[0218] Table 2

[0219] It should be noted that the functions in Table 2 are only examples. In actual application, the functions corresponding to the first modality fusion module may include but are not limited to the operation methods shown in Table 2, and the embodiments of the present invention do not make any limitations in this regard.

[0220] 204. Through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure. Use the next second fusion unit as the new current second fusion unit, and input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, to obtain the fused graph structure output by the last second fusion unit.

[0221] It can be understood that in the target graph learning model, the number of second fusion units can be one or several, and those skilled in the art can connect several second fusion units according to actual application scenarios.

[0222] In some optional embodiments, the second fusion unit can be implemented based on the Transformer model.

[0223] In other embodiments, the second fusion unit includes a second node aggregation module and a second modality fusion module. The step "Through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure" can specifically include:

[0224] Through the second node aggregation module in the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, to obtain a new updated graph structure, and input the new updated graph structure into the second modality fusion module;

[0225] In the second modality fusion module, perform feature fusion on the nodes of the same target content in each new updated graph structure and the fused graph structure, to obtain a new fused graph structure.

[0226] Among them, the second node aggregation module is used to aggregate and update the feature information of each node in the updated graph structure. Specifically, the step "Through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure" can include:

[0227] Through the second correlation coefficient calculation function in the current second fusion unit, calculate the correlation coefficient between each node and other nodes in the input updated graph structure;

[0228] According to the correlation coefficient, determine each node and its neighboring nodes;

[0229] Based on each node and its neighboring nodes, the feature information of each node is aggregated and updated through the second aggregation function in the second fusion unit.

[0230] For example, taking the graph structures corresponding to the target content in the text modality and the image modality as an example, the second node aggregation module can be represented by the following formula:

[0231]

[0232] Among them, Aggregate(·) is the second aggregation function, is the second correlation coefficient calculation function for calculating the correlation coefficient between node m i and its neighboring node u, is the network parameter of the second node aggregation module, is the output of the previous layer (input features when l = 1), N(u) is the receptive field of node m i (that is, the set composed of the neighbor nodes of node m i ).

[0233] Alternatively, for the convenience of calculation, the second node aggregation module can be represented by the following formula:

[0234]

[0235] Among them, σ is the activation function.

[0236] For example, under the network module at the network module position of the second fusion unit, there may be a second correlation calculation sub-module, a second aggregation operation sub-module, and a second activation sub-module. Different sub-module positions may correspond to at least one optional network sub-module.

[0237] Specifically, the operation functions in different optional network sub-modules can also be as shown in Table 1, which will not be elaborated in this embodiment of the present invention.

[0238] It can be understood that Figure 4 although it shows that there are two second node aggregation modules in the second fusion unit to process the graph structures of two modalities respectively, however, there may also be only one second node aggregation module in the second fusion unit to process the graph structures of multiple modalities in sequence.

[0239] That is to say, the number of second node aggregation modules can be the same as the number of modalities to be processed, or different from the number of modalities to be processed. It should be noted that the aggregation functions, activation functions, and correlation functions used in each second node aggregation module in the second aggregation unit can be completely the same or at least one of them is different, and this embodiment of the present invention does not make any limitations in this regard.

[0240] Optionally, the second modality fusion module that fuses nodes of the same target content can be represented by the following formula:

[0241]

[0242] Wherein, is a node in the new fused graph structure, and are nodes corresponding to the same target content in the new updated graph structures of different modalities, is a node in the fused graph structure.

[0243] The FusionCell operation can be to directly perform fusion processing such as adding or multiplying the feature information corresponding to the nodes of the same target content. Or, the step "performing feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure to obtain a new fused graph structure" can specifically include:

[0244] Performing feature mapping on the feature information of each node in each new updated graph structure and the fused graph structure to obtain a new mapped graph structure including new mapped nodes;

[0245] Based on the feature information of the new mapped nodes in each new mapped graph structure, determining new fusion weights for each new mapped node;

[0246] According to the feature information of each new mapped node and each new fusion weight, performing feature fusion on the new mapped nodes of the same target content in each new mapped graph structure to obtain a new fused graph structure.

[0247] Among them, the calculation processes such as the feature mapping process of obtaining new mapped nodes and the fusion weights of new mapped nodes are similar to the calculation processes in the first fusion unit, and are not described in detail in this embodiment of the present invention.

[0248] It can be understood that the specific aggregation functions, activation functions, correlation functions, and fusion operations listed in Tables 1 and 2 used in the first fusion unit and the second fusion unit can be the same or different.

[0249] When there are several second fusion units in the target learning model, the specific aggregation functions, activation functions, correlation functions, and fusion operations listed in Tables 1 and 2 used by each second fusion unit can be the same or different. This embodiment of the present invention does not make any limitations in this regard.

[0250] 205. Through the prediction unit, perform prediction on the nodes corresponding to each target content in the fused graph structure to obtain a prediction result of the target content, and perform content processing based on the prediction result.

[0251] Among them, the prediction unit is a neural network unit that implements the prediction function in the target graph learning network. Specifically, step 205 may specifically include:

[0252] Through the activation function in the prediction unit, the feature information of the nodes corresponding to each target content in the fused graph structure is respectively operated to obtain the operation results corresponding to each target content;

[0253] Through the probability prediction function in the prediction unit, probability prediction processing is performed on each operation result to obtain the prediction scores corresponding to each target content;

[0254] Based on the preset prediction result mapping relationship and each prediction score, the prediction category corresponding to each prediction score is determined as the prediction result of each target content.

[0255] For example, the prediction process of the prediction unit can be represented by the following formula:

[0256]

[0257] Among them, tanh is the activation function and softmax is the probability prediction function.

[0258] Specifically, a linear classification method can be used to predict the prediction scores of the nodes in the fused graph structure output by the last second fusion unit.

[0259] In some embodiments, after obtaining the new fused graph structure, the last second fusion unit may further process the new fused graph structure based on the latest aggregation update result obtained by the second node aggregation module in the last second fusion unit.

[0260]

[0261] Among them, h′ u is the node in the fused graph structure obtained after processing the new fused graph structure again, and is the latest aggregation update result obtained by the second node aggregation module in the last second fusion unit, is the node in the new fused graph structure.

[0262] At this time, the processing process of the prediction unit can be represented by the following formula:

[0263]

[0264] Among them, W is a trainable weight matrix and b is a bias vector. The softmax function is used to obtain the final prediction score

[0265] As Figure 5 shown, low-quality recognition or high-quality mining and other predictions can be made on the multi-modal target content. According to the prediction results, different processing methods can be adopted for the target content. For example, if the prediction score indicates that the target content belongs to high-quality content of the humanities and arts type, the target content can be recommended to users interested in the field of humanities and arts, and so on.

[0266] In Figure 5 it, the input can be the target content including pictures and text, as well as the relational graph structure that can represent the associated relationship corresponding to the target content. According to the target content and the relational graph structure, the graph structure corresponding to the target content can be generated. According to different model tasks, different processing can be performed on the target content. It can be understood that the target graph learning models corresponding to different tasks may be different, and the models that can be trained under different tasks may also vary. For example, after the execution of Task1, Task2, and Task3, the trained model structure becomes increasingly rich.

[0267] That is to say, with the continuous learning of new tasks, the expressive ability of the graph learning model that can be sampled from the model structure information will be continuously improved, and the model structure information can maintain the memory ability.

[0268] As can be seen from the above, the embodiments of the present invention can obtain the graph structures of multiple target contents in multiple modalities. The target graph learning model can use not only the information of the target content itself, but also the associated relationship between the target contents for prediction, enriching the information available when processing multi-modal content. In addition, in the embodiments of the present invention, the feature information of the same target content in different modalities is updated and fused before prediction, and the overall prediction result can be obtained by integrating the information of different modalities, rather than independently predicting the information in different modalities. Therefore, when predicting, the target graph learning model can obtain the feature information of the nodes containing the multi-modal information of the target content itself and the associated relationship information between the target contents for prediction, which can improve the accuracy of multi-modal content processing.

[0269] According to the method described in the previous embodiments, the following will give further detailed examples.

[0270] In this embodiment, it will be described in combination with Figure 1 the system of.

[0271] As Figure 6 shown, the specific process of the multi-modal content processing method in this embodiment can be as follows:

[0272] 601. Sample the model structure information based on the graph learning model to determine the network modules at the positions of the network modules included in each unit of the graph learning model, and obtain at least one graph learning model to be trained.

[0273] Among them, the model structure information includes the model structure and at least one optional network module at the positions of each network module in the model structure.

[0274] Each unit in the graph learning model includes a node aggregation module, a modality fusion module, and a prediction unit.

[0275] In the actual application process, model structure sampling can be performed according to different multi-modal content processing tasks. For different tasks, different graph learning models to be trained will be activated from the module structure information and learned.

[0276] 602. Respectively predict the sample graph structure through each graph learning model to be trained to obtain the first training prediction results corresponding to each graph learning model to be trained. Determine the target graph learning model to be trained from each graph learning model to be trained according to the reference prediction result and each first training prediction result.

[0277] Due to the large domain differences between different task contents, usually using a single model for multi-modal content processing has poor effects, and the effects of a single model on contents from different sources vary greatly and are unstable.

[0278] The automatic architecture search strategy used enables the model to have automatic scalability. When facing data with real-world distribution changes, the model can adaptively adjust the corresponding sub-structures according to the data distribution.

[0279] 603. Predict the sample graph structure through the target graph learning model to be trained to obtain the second training prediction result. Calculate the training loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result.

[0280] In the embodiments of the present invention, the expected accuracy is maximized by minimizing the training loss Maximize:

[0281] <##

[0282]

[0283] Among them, P(a; θ) is the distribution of the model structure a parameterized by the controller parameter θ, and w * is the shared parameter.

[0284] Specifically The calculation process is as detailed in the previous embodiments, and the embodiments of the present invention will not be elaborated here.

[0285] Among them, the model structure information includes global shared parameters that can be used by each target graph learning model to be trained corresponding to different tasks. When it comes to a new task T t When training the target graph learning model, it will be checked whether there are parameters in the operation function of each module or unit in the model structure information.

[0286] If there are parameters, group sparse regularization and orthogonal constraints will be used to learn the new parameters in the target graph learning model. If not, only group sparse regularization constraints will be imposed on the learning of new parameters. Restricting the learning of new parameters based on the group sparse regularization sharing strategy can make the model structure information possess knowledge inductiveness.

[0287] 604. Adjust the model parameters of the target graph learning model to be trained based on the training loss, obtain the trained target graph learning model, and record the target model structure description information of the target graph learning model.

[0288] The pseudo-code for multi-modal architecture evolution continuous graph learning is shown in the following table:

[0289]

[0290] In order to enable the modal structure information to continuously adapt to the continuously changing target content in actual applications, the global shared model parameters in the modal structure information can be continuously updated, so that the graph learning model sampled from the model structure information has model parameters adapted to the new data. After step 304, therefore, the multi-modal content processing method provided by the embodiments of the present invention may further include:

[0291] Obtain the target model parameters of the trained target graph learning model;

[0292] Update the global shared model parameters based on the target model parameters.

[0293] The multi-modal content processing solution involved in this embodiment focuses on saving the model structure, model parameters and other information of different graph learning models with modal structure information that can be shared among different models, maintaining the realizability of the graph learning model while making the overall model parameters to be saved less and more convenient for deployment.

[0294] 605. Obtain the graph structures of multiple target contents in multiple modalities, determine the target model structure description information corresponding to the target content based on the target content, and search for the target graph learning model for processing the graph structure from the model structure information according to the target model structure description information.

[0295] Among them, each graph structure of the target content corresponds to a modality, and each node in the graph structure corresponds to a target content. Among them, the connection relationship between the nodes actually indicates the association relationship between the target contents.

[0296] In the embodiments of the present invention, representation learning of the target content is simultaneously performed based on the multi-modal information of the target content itself and the complex graph structure information generated by social networks, etc., and the multi-modal information and content association information are comprehensively utilized.

[0297] The data from each modality can be represented as a graph G m =(V m , E), E = {(i, j)}, where I m represents the features from a specific modality m, corresponding to visual features and text features respectively.

[0298] 606. Through the node aggregation module, based on each node and its neighboring nodes in the graph structure, the feature information of each node is aggregated and updated to obtain an updated graph structure, and the updated graph structure is input into the next node aggregation module and the modality fusion module connected to the node aggregation module.

[0299] Among them, the node aggregation module can be the first node aggregation module and the second node aggregation module shown in Figure 3 . There is a connection relationship between the node aggregation modules.

[0300] Among them, each node aggregation module can be represented as:

[0301]

[0302] Among them, Aggregate(·) is the aggregation function, is the correlation coefficient in the table, is the network parameter, is the output of the previous layer (the input feature when l = 1), and N(u) is the receptive field of node u.

[0303] 607. Through the modality fusion module, the features of the nodes of the same target content in each updated graph structure are fused to obtain a fused graph structure, and the fused graph structure is input into the next modality fusion module connected to the modality fusion module.

[0304] Among them, the modality fusion module can be the first modality fusion module and the second modality fusion module shown in Figure 3 . There is a connection relationship between the modality fusion modules.

[0305] After updating the representation of the nodes in a specific modality, the representations of different modalities can be modality-fused through the modality fusion module to obtain a new representation, which can be expressed as

[0306]

[0307] 608. Take the next node aggregation module as the new node aggregation module, aggregate and update the feature information of each node based on the nodes and neighboring nodes in the updated graph structure to obtain a new updated graph structure, and input the new updated graph structure into the next node aggregation module and the next modality fusion module connected to the new node aggregation module.

[0308] Utilize the high-order connectivity between nodes, stack more information propagation layers and fusion layers to further enrich the representation. Through the GNN unit, obtain the unimodal information of each node

[0309]

[0310] 609. Take the next modality fusion module as the new modality fusion module, perform feature fusion on the nodes of the same target content in the fused graph structure and each new updated graph structure to obtain a new fused graph structure, and input the new fused graph structure into the next modality fusion module connected to the new modality fusion module.

[0311] Through the fusion unit, obtain the multimodal fusion information of each node

[0312]

[0313] 610. Return to execute steps 608 and 609 until the fused graph structure output by the last modality fusion module is obtained.

[0314] The output h′ of the last layer's modality fusion module u can be used as the input of the prediction unit (classification layer), and a linear layer is used to predict the results of each node:

[0315]

[0316]

[0317] where W is a trainable weight matrix and b is a bias vector. Use the softmax function to obtain the final prediction scores

[0318] 611. Through the prediction unit, perform predictions on the nodes corresponding to each target content in the fused graph structure to obtain the prediction results of the target content, and perform content processing based on the prediction results.

[0319] The predicted score that can predict the unit output is used as the prediction result of the target content.

[0320] According to the prediction result, the content quality classification of the target content, the probability of being interacted with by a certain user or a certain type of user, etc. can be determined.

[0321] For example, content processing such as filtering and backward recommendation can be performed on the target content with low-quality content quality.

[0322] 612. Update the sample graph structure according to the graph structure of the target content in multiple modalities and the prediction result of the target content to obtain a new sample graph structure, and return to execute step 601.

[0323] In the embodiment of the present invention, an architecture-evolvable multi-modal graph neural network model structure information (AdaMGNN) is designed, which is used as the shared information of the entire stage. Since the model structure information needs to be shared and utilized between different tasks and different architectures, its design is different from that of general multi-modal GNNs. In order to better perform continuous learning, the model structure information needs to 1) make full use of the associations between different contents; 2) fuse different modalities for further node classification; 3) the model structure obtained by sampling the model structure information can be adaptively adjusted according to the needs of new tasks.

[0324] As can be seen from the above, the embodiment of the present invention can enrich the available information when processing multi-modal content based on multiple target contents, and improve the accuracy of multi-modal content processing.

[0325] In order to better implement the above method, correspondingly, the embodiment of the present invention also provides a multi-modal content processing device.

[0326] Reference Figure 7 , the device may include:

[0327] A graph structure acquisition structure 701, which can be used to acquire the graph structures of multiple target contents in multiple modalities. Among them, different nodes in a graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same;

[0328] A model search structure 702, which can be used to search for a target graph learning model that can be used to process the graph structure based on the target model structure description information corresponding to the target content. The target graph learning model may at least include a first fusion unit, at least one second fusion unit, and a prediction unit;

[0329] The first fusion structure 703 can be used to aggregate and update the feature information of each node in the graph structure based on each node and its neighboring nodes through the first fusion unit, fuse the features of the nodes with the same target content in each updated graph structure to obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit;

[0330] The second fusion structure 704 can be used to aggregate and update the feature information of each node in the updated graph structure based on each node and its neighboring nodes through the current second fusion unit, fuse the features of the nodes with the same target content in the newly updated graph structure and the fused graph structure to obtain a new fused graph structure, use the next second fusion unit as the new current second fusion unit, input the newly updated graph structure and the new fused graph structure into the new current second fusion unit for processing, and obtain the fused graph structure output by the last second fusion unit;

[0331] The prediction structure 705 can be used to predict the nodes corresponding to each target content in the fused graph structure through the prediction unit to obtain the prediction results of the target content, and perform content processing based on the prediction results.

[0332] In some optional embodiments, the first fusion unit may include a first node aggregation module and a first modality fusion module. The first fusion structure 703 can be used to aggregate and update the feature information of each node in the graph structure based on each node and its neighboring nodes through the first node aggregation module in the first fusion unit to obtain an updated graph structure, and input the updated graph structure into the second fusion unit and the first modality fusion module;

[0333] In the first modality fusion module, fuse the features of the nodes with the same target content in each updated graph structure to obtain a fused graph structure, and input the fused graph structure into the second modality fusion module.

[0334] In some optional embodiments, the second fusion unit may include a second node aggregation module and a second modality fusion module. The second fusion structure 704 can be used to aggregate and update the feature information of each node in the updated graph structure based on each node and its neighboring nodes through the second node aggregation module in the current second fusion unit to obtain a newly updated graph structure, and input the newly updated graph structure into the second modality fusion module;

[0335] In the second modality fusion module, fuse the features of the nodes with the same target content in each newly updated graph structure and the fused graph structure to obtain a new fused graph structure.

[0336] In some alternative embodiments, the first fusion structure 703 can be used to calculate the correlation coefficients between each node and other nodes in the graph structure based on the graph structure through the first correlation coefficient calculation function in the first fusion unit;

[0337] Determine each node and the neighboring nodes of each node according to the correlation coefficients;

[0338] Aggregate and update the feature information of each node based on each node and the neighboring nodes of each node through the first aggregation function in the first fusion unit;

[0339] In some alternative embodiments, the second fusion structure 704 can be used to calculate the correlation coefficients between each node and other nodes in the updated graph structure based on the input updated graph structure through the second correlation coefficient calculation function in the current second fusion unit;

[0340] Determine each node and the neighboring nodes of each node according to the correlation coefficients;

[0341] Aggregate and update the feature information of each node based on each node and the neighboring nodes of each node through the second aggregation function in the second fusion unit.

[0342] In some alternative embodiments, as Figure 8 shown, the multimodal content processing device provided by the embodiments of the present invention may further include a model training structure 706, which can be used to obtain the model structure information of the graph learning model. The model structure information may include the model structure and at least one optional network module at the positions of each network module in the model structure;

[0343] Perform model structure sampling based on the model structure information to determine the network modules at the positions of the network modules included in each unit in the graph learning model, and obtain at least one graph learning model to be trained. The unit may include a first fusion unit, a second fusion unit, and a prediction unit;

[0344] Predict the sample graph structure through each graph learning model to be trained to obtain the first training prediction results corresponding to each graph learning model to be trained. The sample graph structure is labeled with a reference prediction result;

[0345] Determine the target graph learning model to be trained from each graph learning model to be trained according to the reference prediction result and each first training prediction result;

[0346] Predict the sample graph structure through the target graph learning model to be trained to obtain the second training prediction result;

[0347] Calculate the training loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result;

[0348] Adjust the model parameters of the target graph learning model to be trained based on the training loss, obtain the trained target graph learning model, and record the target model structure description information of the target graph learning model;

[0349] The model search structure can be used to determine the target model structure description information corresponding to the target content based on the target content;

[0350] According to the target model structure description information, search for the target graph learning model that can be used to process the graph structure from the model structure information.

[0351] In some alternative embodiments, the multimodal content processing device provided by the embodiments of the present invention may further include a model update structure, which can be used to update the sample graph structure according to the graph structures of the target content in multiple modalities and the prediction results of the target content to obtain a new sample graph structure;

[0352] Return to the step of performing model structure sampling based on the model structure information to determine the network modules at the network module positions included in each unit in the graph learning model, and obtain at least one graph learning model to be trained.

[0353] In some alternative embodiments, the model parameters of the target graph learning model to be trained may include target shared model parameters, and the target shared model parameters are at least part of the global shared model parameters shared by each graph learning model sampled from the model structure information;

[0354] The model training structure 706 can be used to calculate the training result loss of the target graph learning model to be trained according to the reference prediction result and each second training prediction result;

[0355] Based on a preset loss conversion rule, convert the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained;

[0356] Based on the training result loss and the training parameter loss, obtain the training loss of the target graph learning model to be trained.

[0357] In some alternative embodiments, the model training structure 706 can be used to perform group sparse regularization on the target shared model parameters in the target graph learning model to be trained based on a preset loss conversion rule to obtain a first group sparse loss value;

[0358] Obtain the historical target shared model parameters of the target graph learning model to be trained;

[0359] Calculate the difference between the target shared model parameters and the historical target shared model parameters, and perform group sparse regularization on the product of the difference and the transpose matrix of the historical target shared model parameters to obtain the second group sparse loss value;

[0360] Based on the preset first group sparse weight and the second group sparse weight, perform weighted calculation on the first group sparse loss value and the second group sparse loss value to obtain the training parameter loss.

[0361] In some optional embodiments, the multimodal content processing device provided by the embodiments of the present invention may further include a global parameter update structure, which can be used to obtain the target model parameters of the trained target graph learning model;

[0362] Update the global shared model parameters based on the target model parameters.

[0363] In some optional embodiments, the first fusion structure can be used to perform feature mapping on the feature information of each node in each updated graph structure to obtain a mapped graph structure including the mapped nodes;

[0364] Determine the fusion weights of each mapped node based on the feature information of the mapped nodes in each mapped graph structure;

[0365] According to the feature information of each mapped node and each fusion weight, perform feature fusion on the mapped nodes of the same target content in each mapped graph structure to obtain a fused graph structure.

[0366] In some optional embodiments, the prediction structure 705 can be used to perform operations on the feature information of the nodes corresponding to each target content in the fused graph structure respectively through the activation function in the prediction unit to obtain the operation results corresponding to each target content;

[0367] Perform probability prediction processing on each operation result through the probability prediction function in the prediction unit to obtain the prediction scores corresponding to each target content;

[0368] Based on the preset prediction result mapping relationship and each prediction score, determine the prediction category corresponding to each prediction score as the prediction result of each target content.

[0369] In some optional embodiments, the graph structure acquisition structure 701 can be used to obtain multiple target contents, perform modal feature extraction on each target content respectively to obtain the feature information of each target content in multiple modalities;

[0370] Determine the association relationship between each target content;

[0371] Based on the association relationship, respectively use the feature information of different target contents in the same modality as graph nodes to construct a graph structure of the target content in multiple modalities.

[0372] In specific implementation, each of the above structures can be implemented as an independent entity, or can be arbitrarily combined and implemented as the same or several entities. For the specific implementation of each of the above structures, reference can be made to the method embodiments described above, which will not be elaborated here. As can be seen from the above, through the multimodal content processing device, the information available for processing multimodal content can be enriched based on multiple target contents, improving the accuracy of multimodal content processing.

[0373] In addition, an embodiment of the present invention further provides a computer device, which can be a terminal or a server. As Figure 9 shown, it shows a schematic structural diagram of the computer device involved in the embodiment of the present invention. Specifically:

[0374] The computer device may include components such as a processor 901 with one or more processing cores, a memory 902 with one or more computer-readable storage media, a power supply 903, and an input unit 904. Those skilled in the art can understand that Figure 9 the computer device structure shown in

[0375] does not constitute a limitation on the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or have different component arrangements. Among them:

[0376] The memory 902 can be used to store software programs and modules. The processor 901 executes various functional applications and data processing by running the software programs and modules stored in the memory 902. The memory 902 mainly includes a program storage area and a data storage area. Among them, the program storage area can store an operating system, application programs required for at least one function (such as a sound playback function, an image playback function, etc.); the data storage area can store data created according to the use of the computer device. In addition, the memory 902 can include a high-speed random access memory, and can also include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 902 can also include a memory controller to provide the processor 901 with access to the memory 902.

[0377] The computer device further includes a power supply 903 for powering each component. Preferably, the power supply 903 can be logically connected to the processor 901 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 903 can also include any components such as one or more DC or AC power supplies, a recharge system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0378] The computer device may further include an input unit 904, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0379] Although not shown, the computer device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 901 in the computer device will load the executable files corresponding to the processes of one or more application programs into the memory 902 according to the following instructions, and the processor 901 will run the application programs stored in the memory 902 to implement various functions as follows:

[0380] Obtain the graph structures of multiple target contents in multiple modalities, where different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same;

[0381] Based on the target model structure description information corresponding to the target content, search for a target graph learning model for processing the graph structure. The target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit;

[0382] Through the first fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the graph structure, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit;

[0383] Through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, obtain a new fused graph structure, take the next second fusion unit as the new current second fusion unit, input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, and obtain the fused graph structure output by the last second fusion unit;

[0384] Through the prediction unit, perform prediction on the nodes corresponding to each target content in the fused graph structure, obtain the prediction result of the target content, and perform content processing based on the prediction result.

[0385] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructions, or by instructions controlling related hardware. The instructions can be stored in a computer-readable storage medium and loaded and executed by a processor.

[0386] Therefore, an embodiment of the present invention provides a computer-readable storage medium, in which multiple instructions are stored. The instructions can be loaded by a processor to execute the steps in any of the multimodal content processing methods provided by the embodiments of the present invention. For example, the instructions can execute the following steps:

[0387] Obtain the graph structures of multiple target contents in multiple modalities. Among them, different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same;

[0388] Based on the target model structure description information corresponding to the target content, search for a target graph learning model for processing the graph structure. The target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit;

[0389] Through the first fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the graph structure, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit;

[0390] Through the current second fusion unit, based on the nodes in the updated graph structure and the neighboring nodes' feature information for each node to perform aggregation and update, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure. Use the next second fusion unit as the new current second fusion unit, and input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, to obtain the fused graph structure output by the last second fusion unit;

[0391] Through the prediction unit, perform prediction on the nodes corresponding to each target content in the fused graph structure, to obtain the prediction result of the target content, and perform content processing based on the prediction result.

[0392] For the specific implementation of each of the above operations, reference can be made to the previous embodiments, which will not be elaborated here.

[0393] Among them, the computer-readable storage medium may include: read-only memory (ROM, Read Only Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disc, etc.

[0394] Since the instructions stored in the computer-readable storage medium can execute the steps in any of the multimodal content processing methods provided in the embodiments of the present invention, therefore, the beneficial effects achievable by any of the multimodal content processing methods provided in the embodiments of the present invention can be realized. For details, reference can be made to the previous embodiments, which will not be elaborated here.

[0395] According to one aspect of the present application, there is also provided a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. The processor of the electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device executes the methods provided in the various alternative implementations in the above embodiments.

[0396] The above has introduced in detail a multimodal content processing method, apparatus, electronic device and storage medium provided by the embodiments of the present invention. Specific examples are used in this article to elaborate on the principle and implementation manner of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manner and application scope. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A multimodal content processing method, characterized in that, Including: Obtain the graph structures of multiple target contents in multiple modalities. Among them, different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same. The target content is at least one of text, picture, audio, or video; Based on the target model structure description information corresponding to the target content, search for a target graph learning model for processing the graph structure. The target graph learning model at least includes a first fusion unit, at least one second fusion unit, and a prediction unit; Through the first fusion unit, based on each node and adjacent nodes in the graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit; Through the current second fusion unit, based on each node and adjacent nodes in the updated graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in the newly updated graph structure and the fused graph structure, obtain a new fused graph structure, use the next second fusion unit as the new current second fusion unit, input the newly updated graph structure and the new fused graph structure into the new current second fusion unit for processing, and obtain the fused graph structure output by the last second fusion unit; Through the prediction unit, perform prediction on the nodes corresponding to each target content in the fused graph structure, obtain the prediction result of the target content, and perform content processing based on the prediction result.

2. The multimodal content processing method according to claim 1, wherein The first fusion unit includes a first node aggregation module and a first modality fusion module; The step of, through the first fusion unit, based on each node and adjacent nodes in the graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input each updated graph structure and the fused graph structure into the second fusion unit, includes: Through the first node aggregation module in the first fusion unit, based on each node and adjacent nodes in the graph structure, aggregate and update the feature information of each node, obtain an updated graph structure, and input the updated graph structure into the second fusion unit and the first modality fusion module; In the first modality fusion module, perform feature fusion on the nodes of the same target content in each updated graph structure, obtain a fused graph structure, and input the fused graph structure into the second fusion unit.

3. The multimodal content processing method according to claim 1, wherein The second fusion unit includes a second node aggregation module and a second modality fusion module; The step of, through the current second fusion unit, based on each node and adjacent nodes in the updated graph structure, aggregate and update the feature information of each node, perform feature fusion on the nodes of the same target content in the newly updated graph structure and the fused graph structure, obtain a new fused graph structure, includes: Through the second node aggregation module in the current second fusion unit, based on each node and its neighboring nodes in the updated graph structure, aggregate and update the feature information of each node to obtain a new updated graph structure, and input the new updated graph structure into the second modality fusion module; In the second modality fusion module, perform feature fusion on the nodes of the same target content in each new updated graph structure and the fused graph structure to obtain a new fused graph structure.

4. The multimodal content processing method according to claim 1, wherein The aggregating and updating of the feature information of each node based on each node and its neighboring nodes in the graph structure through the first fusion unit includes: Through the first correlation coefficient calculation function in the first fusion unit, calculate the correlation coefficient between each node and other nodes in the graph structure based on the graph structure; According to the correlation coefficient, determine each node and the neighboring nodes of each node; Through the first aggregation function in the first fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes; The aggregating and updating of the feature information of each node based on each node and its neighboring nodes in the updated graph structure through the current second fusion unit includes: Through the second correlation coefficient calculation function in the current second fusion unit, calculate the correlation coefficient between each node and other nodes in the input updated graph structure based on the updated graph structure; According to the correlation coefficient, determine each node and the neighboring nodes of each node; Through the second aggregation function in the second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes.

5. The multimodal content processing method according to claim 1, wherein Before searching for the target graph learning model for processing the graph structure based on the target model structure description information corresponding to the target content, the method further includes: Obtain the model structure information of the graph learning model, where the model structure information includes the model structure and at least one optional network module at the positions of each network module in the model structure; Based on the model structure information, perform model structure sampling to determine the network modules at the positions of the network modules included in each unit in the graph learning model, and obtain at least one graph learning model to be trained, where the units include the first fusion unit, the second fusion unit, and the prediction unit; Through each graph learning model to be trained, respectively predict the sample graph structure to obtain the first training prediction results corresponding to each graph learning model to be trained, where the sample graph structure is labeled with the reference prediction result; According to the reference prediction result and each first training prediction result, determine the target graph learning model to be trained from each graph learning model to be trained; Through the target graph learning model to be trained, predict the sample graph structure to obtain the second training prediction result; According to the reference prediction result and each second training prediction result, calculate the training loss of the target graph learning model to be trained; Adjust the model parameters of the target graph learning model to be trained based on the training loss, obtain the trained target graph learning model, and record the target model structure description information of the target graph learning model; Searching for a target graph learning model for processing the graph structure based on the target model structure description information corresponding to the target content includes: Determine the target model structure description information corresponding to the target content based on the target content; Search for a target graph learning model for processing the graph structure from the model structure information according to the target model structure description information.

6. The multimodal content processing method according to claim 5, wherein The method further includes: Update the sample graph structure according to the graph structure of the target content in multiple modalities and the prediction result of the target content to obtain a new sample graph structure; Return to execute the step of performing model structure sampling based on the model structure information, determining the network modules at the network module positions included in each unit in the graph learning model, and obtaining at least one graph learning model to be trained.

7. The multimodal content processing method according to claim 5, wherein The model parameters of the target graph learning model to be trained include target shared model parameters, and the target shared model parameters are at least part of the global shared model parameters shared by each graph learning model sampled from the model structure information; Calculating the training loss of the target graph learning model to be trained according to the reference prediction result and each of the second training prediction results includes: Calculate the training result loss of the target graph learning model to be trained according to the reference prediction result and each of the second training prediction results; Based on a preset loss conversion rule, convert the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained; Based on the training result loss and the training parameter loss, obtain the training loss of the target graph learning model to be trained.

8. The multimodal content processing method according to claim 7, wherein The converting the target shared model parameters in the target graph learning model to be trained into the training parameter loss of the target graph learning model to be trained based on a preset loss conversion rule includes: Based on a preset loss conversion rule, perform group sparse regularization on the target shared model parameters in the target graph learning model to be trained to obtain a first group sparse loss value; Obtain the historical target shared model parameters of the target graph learning model to be trained; Calculate the difference between the target shared model parameters and the historical target shared model parameters, and perform group sparse regularization on the product of the difference and the transpose matrix of the historical target shared model parameters to obtain a second group sparse loss value; Based on a preset first group sparse weight and second group sparse weight, perform weighted calculation on the first group sparse loss value and the second group sparse loss value to obtain the training parameter loss.

9. The multimodal content processing method according to claim 7, wherein The method further includes: Obtain the target model parameters of the trained target graph learning model; Update the global shared model parameters based on the target model parameters.

10. The multimodal content processing method according to claim 1, wherein The fusing the features of the nodes of the same target content in each updated graph structure to obtain a fused graph structure includes: Perform feature mapping on the feature information of each node in each updated graph structure to obtain a mapped graph structure containing the mapped nodes; Based on the feature information of the mapped nodes in each of the mapped graph structures, determine the fusion weights of each of the mapped nodes; According to the feature information of each of the mapped nodes and each of the fusion weights, perform feature fusion on the mapped nodes of the same target content in each of the mapped graph structures to obtain a fused graph structure.

11. The multimodal content processing method according to claim 1, wherein The prediction of the target content corresponding to each node in the fused graph structure by the prediction unit to obtain the prediction result of the target content includes: Through the activation function in the prediction unit, perform operations on the feature information of the nodes corresponding to each target content in the fused graph structure respectively to obtain the operation results corresponding to each target content; Through the probability prediction function in the prediction unit, perform probability prediction processing on each of the operation results to obtain the prediction scores corresponding to each target content; Based on the preset prediction result mapping relationship and each of the prediction scores, determine the prediction category corresponding to each of the prediction scores as the prediction result of each target content.

12. The multimodal content processing method according to any one of claims 1-11, characterized in that, The obtaining of the graph structures of multiple target contents in multiple modalities includes: Obtain multiple target contents, and respectively perform modal feature extraction on each of the target contents to obtain the feature information of each of the target contents in multiple modalities; Determine the association relationship between each of the target contents; Based on the association relationship, respectively use the feature information of different target contents in the same modality as graph nodes to construct the graph structures of the target contents in multiple modalities.

13. A multimodal content processing device, characterized in that, Including: A graph structure acquisition structure for obtaining the graph structures of multiple target contents in multiple modalities, where different nodes in one graph structure correspond to the feature information of different target contents in the same modality, and the connection relationships of the nodes in different graph structures are the same, and the target content is at least one of text, picture, audio, or video; A model search structure for searching, based on the target model structure description information corresponding to the target content, for a target graph learning model for processing the graph structure, where the target graph learning model includes at least a first fusion unit, at least one second fusion unit, and a prediction unit; A first fusion structure for, through the first fusion unit, aggregating and updating the feature information of each node based on each node and its neighboring nodes in the graph structure, performing feature fusion on the nodes of the same target content in each updated graph structure to obtain a fused graph structure, and inputting each of the updated graph structures and the fused graph structure into the second fusion unit; A second fusion structure, which is used to, through the current second fusion unit, aggregate and update the feature information of each node based on each node and its neighboring nodes in the updated graph structure, perform feature fusion on the nodes of the same target content in the new updated graph structure and the fused graph structure, to obtain a new fused graph structure, use the next second fusion unit as the new current second fusion unit, input the new updated graph structure and the new fused graph structure into the new current second fusion unit for processing, and obtain the fused graph structure output by the last second fusion unit; A prediction structure, which is used to, through the prediction unit, perform prediction on the nodes corresponding to each of the target contents in the fused graph structure, obtain the prediction results of the target contents, and perform content processing based on the prediction results.

14. An electronic device, characterized in that, It includes a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the steps in the multimodal content processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores multiple instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the multimodal content processing method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program or instructions, characterized in that, When the computer program or instruction is executed by a processor, it realizes the steps of the multimodal content processing method according to any one of claims 1 to 12.

Images