A method and system for audio-visual video question answering based on multimodal heterogeneous graph

Abstract

The present invention discloses an audio-visual video question-answering method and system based on a multimodal heterogeneous graph, which relates to technical fields such as multimodal interaction, video content analysis, and intelligent question-answering. The method extracts visual features and audio features from audio and video; constructs motion-guided and audio-guided heterogeneous graphs respectively according to the visual features and audio features; aggregates the motion-guided and audio-guided heterogeneous graphs to obtain visual features and audio features after audio-visual interaction; extracts question features from query questions, selects key visual areas according to the question features, and obtains sparse visual features; constructs a question-guided heterogeneous graph according to the question features and the sparse visual features, and obtains local visual features and global visual features; fuses the question features with the audio features, local visual features, and global visual features after audio-visual interaction, respectively, to obtain fused question features; obtains a prediction result of the question answer according to the multimodal fused question features, and improves the accuracy of question-answering.

Description

Technical Field

[0001] The present invention relates to technical fields such as multimodal interaction, video content analysis, and intelligent question and answer, and in particular to an audio-visual video question and answer method and system based on a multimodal heterogeneous graph. Background Art

[0002] The intelligent question-answering technology of visual-audio-text multimodality (audio-text multimodality) aims to expand the machine's understanding of the real human environment and build a system architecture for perceiving and reasoning audio-visual multimodality. Benefiting from the widespread dissemination of audio and video on the Internet, the understanding and dialogue based on audio and video content have attracted attention in many fields. For example: the audio-visual multimodal question-answering system can play the role of an intelligent teacher in early childhood cognitive education; in cross-modal video recommendation, the system can retrieve similar videos based on any modal content, breaking through the cross-modal bottleneck.

[0003] The rapid development of deep learning in recent years has also promoted the development of audio-visual multimodal intelligent question-answering technology towards more robust interaction and reasoning capabilities, but it is still limited by some challenges:

[0004] 1. Existing methods do not discuss moving or stationary objects in the video enough, which limits the network model's capture of key objects. To a certain extent, it is easy to lose attention to the sounding objects in the video, resulting in the phenomenon of incorrect matching of question and answer objects. For example: In 2022, Guangyao Li et al. published an article titled "Learning to Answer Question in Dynamic Audio-Viusal Scenarios" at the top international conference IEEE Conference on Computer Vision and Pattern Recognition. This article first proposed to focus on the association between visual images and audio clips based on the Attention mechanism, and capture cross-modal associations from the representation attention level of different modalities. However, this method of relying on modal representation will fail in some cases. For example, in the video, there are two people holding violins on the left and right, the left one is playing continuously, and the right one is just holding the violin. When the question-answering system is asked, "How many violins are making sounds from beginning to end?", the system will answer incorrectly without considering the movement of the object.

[0005] 2. Existing methods are not compatible enough with various complex video environments. Faced with a large amount of complex and changeable video content, the system tends to over-focus on the background information of the video and ignore the sound objects related to the question in the video. For example: In 2023, Zailong Chen and other authors published "Question-Aware Global-Local Video Understanding Network for Audio-Visual Question Answering" in the internationally renowned journal IEEE Transactions on Circuits and Systems for Video Technology, and Guangyao Li published "Progressive Spatio-temporal Perception for Audio-Visual Question Answering" at the internationally renowned conference ACM International conference on Multimedia. These articles all suggest that paying attention to finer-grained visual features is conducive to the correct answer to the question, but these fine-grained features also include background information into the network model to participate in computational reasoning. This method allows the model to receive a large amount of background information, which is not conducive to improving the generalization ability of the model. Summary of the invention

[0006] In order to overcome the defects in the above-mentioned prior art, the present invention provides an audio-visual video question and answer method based on a multimodal heterogeneous graph, aiming to improve the multimodal audio-visual video question and answer system, enhance the system's processing and analysis capabilities of user input videos, and improve the accuracy of question and answer.

[0007] To achieve the above object, the present invention adopts the following technical solutions, including:

[0008] A method for audio-visual video question answering based on a multimodal heterogeneous graph comprises the following steps:

[0009] S1, extracts visual features and audio features from audio and video;

[0010] S2, constructs motion-guided heterogeneous graphs based on visual and audio features;

[0011] S3, constructs an audio-guided heterogeneous graph based on visual features and audio features;

[0012] S4, aggregate the motion-guided heterogeneous graph and the audio-guided heterogeneous graph to obtain the visual features and audio features after audio-visual interaction;

[0013] S5, extracting the problem features in the query question, selecting the key visual area according to the problem features, and obtaining sparse visual features;

[0014] S6, constructing a question-guided heterogeneous graph based on the question features and sparse visual features to obtain local visual features and global visual features;

[0015] S7, multimodal fusion: the question features are fused with the audio features, local visual features and global visual features after audio-visual interaction to obtain fused question features;

[0016] S8, based on the question features after multimodal fusion, obtain the prediction result of the answer to the question.

[0017] Preferably, in step S1, the audio and video are divided to obtain T video frames and T audio clips; feature extraction is performed on each video frame and audio clip to obtain visual features and audio features of the audio and video;

[0018] Among them, the visual feature is recorded as V={v t |t=1,...,T}, , v t is the visual feature of the tth video frame, , N 2 represents the number of visual areas in the video frame image, and d is the feature dimension;

[0019] The audio feature is recorded as I={a t |t=1,...,T},I∈R T×d , where a t is the audio feature of the t-th audio clip, a t ∈R 1×d .

[0020] Preferably, the specific process of step S2 is as follows:

[0021] S21, calculate the motion density m of each video frame t , calculated as:

[0022] First, the motion density ρ of each visual area is calculated i t :

[0023] ;

[0024] Among them, ||·|| represents the feature binary norm normalization processing; v i t With v i t+1 represents the regional visual features corresponding to the i-th visual area in the adjacent video frames, i.e., the t-th video frame and the t+1-th video frame; ρi t represents the motion density of the i-th visual area in the t-th video frame;

[0025] Then the motion information ρ of the tth video frame is calculated t :

[0026] ;

[0027] Get the average value μ of the motion information of all video frames:

[0028] ;

[0029] Finally, the motion density m of each video frame is calculated. t , m t = (1-λ)ρ t +λμ; where λ is the balance coefficient;

[0030] S22, obtain the connection matrix A of the motion-guided heterogeneous graph m t :

[0031] ;

[0032] in, Indicates the dot product between two matrix elements; the superscript T indicates matrix transposition, and the symbol ";" indicates matrix concatenation;

[0033] S23, determining the graph node N of the motion-guided heterogeneous graph t =[v t ; a t ];

[0034] S24, construct motion-guided heterogeneous graph G m t , G m t =ReLU(A m t N t W m );

[0035] Among them, W m is a motion-guided heterogeneous graph G m t The learnable parameters of , ReLU is the activation function;

[0036] G m t (N t ) = [v m t ; a m t ],vm t Heterogeneous graph G guided by motion m t The graph node N t The visual features in a m t Heterogeneous graph G guided by motion m t The graph node N t The audio features in .

[0037] Preferably, the specific process of step S3 is as follows:

[0038] S31, calculate the sound activation matrix s for each video frame t :

[0039] ;

[0040] in, Represents matrix multiplication;

[0041] S32, obtaining the connection matrix A of the audio-guided heterogeneous graph s t :

[0042] ;

[0043] S33, determining the graph node N of the audio-guided heterogeneous graph t =[v t ; a t ];

[0044] S34, constructing audio-guided heterogeneous graph G s t , G s t =ReLU(A s t N t W s );

[0045] Among them, W s is the audio-guided heterogeneous graph G s t The learnable parameters of

[0046] G s t (N t ) = [v s t ; a s t ],v s t The heterogeneous graph G guided by audio st The graph node N t The visual features in a s t The heterogeneous graph G guided by audio s t The graph node N t The audio features in .

[0047] Preferably, in step S4, the motion-guided heterogeneous graph G of step S2 is m t and the audio-guided heterogeneous graph G of step S3 s t Aggregation, specifically, merging the graph nodes of the two heterogeneous graphs through a linear layer, and finally obtaining the graph node N1 after audio-visual interaction t =[v1 t ; a1 t ];in, is the visual feature after audio-visual interaction, a1 t ∈R 1×d It is the audio feature after audio-visual interaction.

[0048] Preferably, in step S5, the question feature extracted from the query question is q∈R K×d , K is the number of words, d is the feature dimension; select the key visual area according to the problem feature q and obtain the sparse visual features, as shown below:

[0049] ;

[0050] Among them, α i t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The regional visual feature v1 of the i-th visual area in i t The similarity between t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The similarity between the regional visual features in β i t is the visual region mask obtained according to the problem features, which is used to retain the top r% of the regional visual features with the highest similarity, and the remaining regional visual features are reassigned to 0; using the visual region mask β i t Visual features after audio-visual interaction v1 t After mask operation, sparse visual feature v2 is obtained t .

[0051] Preferably, the specific process of step S6 is as follows:

[0052] S61, determine the graph node M of the problem-guided heterogeneous graph t =[v2 t ; q];

[0053] S62, calculate the connection matrix A of the problem-guided heterogeneous graph q t :

[0054] A q t = M t M t T = [v2 t ; q][v2 t ; q] T ;

[0055] S63, constructing a problem-guided heterogeneous graph G q t , G q t =ReLU(A q t M t W q );

[0056] Among them, W q is a motion-guided heterogeneous graph G q t The learnable parameters of

[0057] G q t (M t )=[v q t ; q q t ],v q t The heterogeneous graph G guided by the problem q t The graph node M t The local visual features in ,q q t The heterogeneous graph G guided by the problem q t The graph node M t Characteristics of the problem.

[0058] Preferably, in step S7,

[0059] The question features are fused with the audio features after audio-visual interaction to construct the first heterogeneous graph of the question modality and the audio modality. The specific method is as follows:

[0060] The audio feature I1={a1 t|t=1,...,T} are concatenated as the graph node γ1=[q ; I1] of the first heterogeneous graph, and the connection matrix A1=[q ; I1][q ; I1] of the first heterogeneous graph is calculated. T , the first heterogeneous graph G qa =ReLU(A1γ1W1); where W1 is the first heterogeneous graph G qa The learnable parameters of

[0061] G qa (γ1)=[q qa ; a qa ], q qa is the first heterogeneous graph G qa The question feature in the graph node γ1 is the question feature after fusion with the audio feature after audio-visual interaction, a qa is the first heterogeneous graph G qa The audio features in the graph node γ1 of ;

[0062] The problem features are fused with the local visual features to construct a second heterogeneous graph of the problem modality and the local visual features. The specific method is as follows:

[0063] The problem feature q is combined with the local visual feature V1={v q t |t=1,...,T} are concatenated as the graph node γ2=[q ; V1] of the second heterogeneous graph, and the connection matrix A2=[q ; V1][q ; V1] of the second heterogeneous graph is calculated. T , the second heterogeneous graph G qV1 =ReLU(A2γ2W2); where W2 is the second heterogeneous graph G qV1 The learnable parameters of

[0064] G qV1 (γ2)=[q qV1 ; v qV1 ], q qV1 The second heterogeneous graph G qV1 The problem feature in the graph node γ2, that is, the problem feature after fusion with the local visual feature, v qV1 The second heterogeneous graph G qV1 The local visual features in the graph node γ2;

[0065] The problem features are fused with the global visual features to construct a third heterogeneous graph of the problem modality and the global visual modality. The specific method is as follows:

[0066] The local visual feature V1={v q t|t=1,...,T} and take the average to get the global visual feature V2. The problem feature q and the global visual feature V2 are concatenated as the graph node γ3=[q ; V2] of the third heterogeneous graph. The connection matrix A3=[q ; V2][q ; V2] of the third heterogeneous graph is calculated. T , the third heterogeneous graph G qV2 =ReLU(A3γ3W3); where W3 is the third isomorphic graph G qV2 The learnable parameters of

[0067] G qV2 (γ3)=[q qV2 ; v qV2 ], q qV2 is the third heterogeneous graph G qV2 The problem feature in the graph node γ3, that is, the problem feature after fusion with the global visual feature, v qV2 is the third heterogeneous graph G qV2 The global visual features in the graph node γ3 of .

[0068] Preferably, in step S8, the fused problem feature q obtained in step S7 is qa ,q qV1 and q qV2 After superposition, the prediction result of the answer to the question is obtained through the fully connected network layer.

[0069] The present invention also provides an audio-visual video question-answering system based on a multimodal heterogeneous graph, which is applicable to the above-mentioned audio-visual video question-answering method based on a multimodal heterogeneous graph, and the system includes: a motion-guided heterogeneous graph module, an audio-guided heterogeneous graph module, a question-guided heterogeneous graph module and a multimodal fusion reasoning module;

[0070] The motion-guided heterogeneous graph module is used to construct a motion-guided heterogeneous graph;

[0071] The audio-guided heterogeneous graph module is used to construct an audio-guided heterogeneous graph;

[0072] The problem-guided heterogeneous graph module is used to construct a problem-guided heterogeneous graph;

[0073] The multimodal fusion reasoning module is used to perform multimodal fusion and obtain a prediction result of the answer to the question based on the question features after multimodal fusion.

[0074] The advantages of the present invention are:

[0075] (1) Traditional methods often ignore the motion information of objects in the video. When there are multiple objects in the video that are semantically related to the sound, traditional methods cannot distinguish which one corresponds to the sound, and therefore it is difficult to give a correct answer. The present invention proposes a motion-guided heterogeneous graph network to capture the motion change information of adjacent video frames and discover significant objects with large motion changes in the video. In order to further enhance the model's understanding of visual-audio matching, the present invention also proposes an audio-guided heterogeneous graph network. By modeling the correspondence between vision and audio, the graph structure is used to learn the relationship between intra-modality and inter-modality, improve the quality of audio-visual representation learning, and improve the accuracy of capturing the sound object.

[0076] (2) In order to reduce the visual information that is irrelevant to the input question, the present invention designs a graph sparsification method based on visual regions. By using the question information as a guide, the visual regions that are weakly associated with the question text are thinned out, so that the model can focus on the visual regions of interest to the question, thereby achieving higher model robustness.

[0077] (3) The present invention introduces visual change information and visual-audio feature association as guidance to improve the model's capture of moving objects and sound-emitting objects. The two can achieve visual and auditory synergy and information complementarity, and can better capture significant visual information related to sound.

[0078] (4) The present invention introduces a visual area sparse mechanism, which helps the model focus on the visual area required by the problem and improves the robustness of the model to different scenarios.

[0079] (5) The present invention adopts a visual-auditory collaboration approach, which is further guided by text representation, which is consistent with the process that people first have a specific understanding of the video content and then use question representation to guide. The method has strong generalization ability and performs well in the existing Music-AVQA and AVQA datasets, as well as in different video question-answering scenarios.

[0080] (6) The present invention is based on the architecture of a graph network, and the overall number of model parameters is small. Compared with some existing methods based on the Transformer architecture and based on large-scale data pre-training, the audio-visual video question-answering method based on a multimodal heterogeneous graph proposed in the present invention has the same performance with a smaller number of parameters, and is more advantageous at the system deployment level. BRIEF DESCRIPTION OF THE DRAWINGS

[0081] Figure 1 The present invention is a flowchart of an audio-visual video question-answering method based on a multimodal heterogeneous graph. DETAILED DESCRIPTION

[0082] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0083] The multimodal heterogeneous graph audio-visual video question-answering system of the present invention can accept audio videos with variable lengths, and after deployment, can respond to users' question requests about the context content of the video in real time, and each inquiry will be answered and a specific answer will be generated.

[0084] Depend on Figure 1 As shown, a method for audio-visual video question answering based on a multimodal heterogeneous graph specifically includes the following steps:

[0085] S1, extracting visual features and audio features of the audio and video. Dividing the audio and video to obtain T video frames and T audio clips. In this embodiment, a video with audio is divided at a frequency of taking one frame every 6 seconds, and the video is divided to obtain T video frames, and correspondingly obtain T audio clips.

[0086] The pre-trained CLIP-VIT-L / 14 model is used to extract features from each video frame to obtain regional visual features; V = {v t |t=1,...,T}, , where N 2 Represents the number of visual areas in a video frame (video frame image); v t is the visual feature of the tth video frame, , d is the feature dimension; the t-th video frame is the video frame image at the t-th moment.

[0087] Accordingly, the pre-trained CLAP model is used to extract features from each audio clip to obtain audio features; the audio features are recorded as I = {a t |t=1,...,T},I∈R T×d , where a t is the audio feature of the t-th audio clip, a t ∈R 1×d ; The t-th audio segment is the audio segment from the t-1-th moment to the t-th moment.

[0088] S2, constructs a motion-guided heterogeneous graph.

[0089] Step S2 is specifically as follows:

[0090] S21, obtain the information required for constructing the motion-guided heterogeneous graph, i.e., the motion density m of each video framet .

[0091] The motion density ρ of each visual area in the video frame is measured by the following formula i t :

[0092] ;

[0093] Among them, ||·|| represents the feature binary norm normalization processing; v i t With v i t+1 represents the regional visual features corresponding to the i-th visual area in the adjacent video frames, i.e., the t-th video frame and the t+1-th video frame; ρ i t represents the motion density of the ith visual area in the tth video frame.

[0094] Thus, the motion information ρ of the tth video frame can be obtained t for: .

[0095] The average value μ of the motion information of all video frames can be expressed as:

[0096] .

[0097] Integrate local and global motion information to calculate the motion density m of each video frame t for:

[0098] m t = (1-λ)ρ t +λμ;

[0099] Wherein, λ represents the balance coefficient, and in this embodiment, λ is set to 0.2; m t represents the motion density of the tth video frame, .

[0100] S22, obtain the connection matrix A of the motion-guided heterogeneous graph m t .

[0101] Movement density m t It can reflect the strength of the object movement in the video. The model should focus on those significant objects with large movement amplitudes. By t Multiply by the visual feature v t To achieve the enhancement of visual features. Furthermore, by calculating the cosine similarity of the enhanced visual features and audio features, the connection matrix of the audio-guided heterogeneous graph can be obtained: , as shown below:

[0102] ;

[0103] in, It represents the dot product between two matrix elements; the superscript T represents the matrix transpose operation, and the symbol “;” represents the matrix concatenation.

[0104] S23, determining the graph node N of the motion-guided heterogeneous graph t .

[0105] In addition to the connection matrix A m t In addition, another basic element for constructing heterogeneous graphs is graph nodes. For each time t, the visual feature v t and audio feature a t As visual nodes and audio nodes, and spliced ​​together as a graph node, denoted by N t =[v t ; a t ], .

[0106] S24, construct motion-guided heterogeneous graph G m t .

[0107] Get graph node N t and the connection matrix A m t After that, construct the motion-guided heterogeneous graph G m t , as shown below:

[0108] G m t =ReLU(A m t N t W m );

[0109] Among them, W m ∈R d×d is a motion-guided heterogeneous graph G m t is a learnable parameter, and ReLU is the activation function. t The role of the connection matrix A m t contains information about the strength of the object motion in the video, which in turn affects the heterogeneous graph G m t Different graph nodes N t The weight of is set so that the visual nodes with large motion changes are enhanced, and the audio nodes are also enhanced due to the interaction with the visual nodes.

[0110] G m t(N t ) = [v m t ; a m t ],v m t Heterogeneous graph G guided by motion m t The graph node N t The visual features in a m t Heterogeneous graph G guided by motion m t The graph node N t The audio features in .

[0111] S3, constructs an audio-guided heterogeneous graph.

[0112] S31, obtain the information required for constructing the audio-guided heterogeneous graph, namely the sound activation moment s for each video frame t , the specific calculation method is shown in the following formula:

[0113] ;

[0114] in, represents matrix multiplication; ||·|| represents the normalization of the characteristic norm; the superscript T represents the transpose operation of the matrix; s t represents the sound activation matrix of the t-th video frame, .

[0115] S32, obtaining the connection matrix A of the audio-guided heterogeneous graph s t .

[0116] In addition to using motion information to guide the enhancement of the visual object area that changes significantly in the video, the sound-generating object can also be discovered through audio. Similar to the execution process of step S22, by t Multiply by the visual feature v t To enhance the visual features of the vocalization, the connection matrix of the audio-guided heterogeneous graph , can be calculated by the following formula:

[0117] ;

[0118] in, It represents the dot product between two matrix elements; the superscript T represents the matrix transpose operation, and the symbol “;” represents the matrix concatenation.

[0119] S33, determining the graph node N of the audio-guided heterogeneous graph t .

[0120] Similar to the execution process of step S23, the audio-guided heterogeneous graph G s t The graph nodes and motion-guided heterogeneous graph G m t The graph nodes are consistent. For each time t, the visual feature v t and audio feature a t As visual nodes and audio nodes, and spliced ​​together as a graph node, denoted by N t =[v t ; a t ].

[0121] S34, constructing audio-guided heterogeneous graph G s t .

[0122] Similar to the execution process of step S24, the graph node N is obtained. t and the connection matrix A s t After that, construct the audio-guided heterogeneous graph G s t , as shown below:

[0123] G s t =ReLU(A s t N t W s );

[0124] Among them, W s ∈R d×d is the audio-guided heterogeneous graph G s t The learnable parameters of .

[0125] G s t (N t ) = [v s t ; a s t ],v s t The heterogeneous graph G guided by audio s t The graph node N t The visual features in a s t The heterogeneous graph G guided by audio s t The graph node N t The audio features in .

[0126] S4, the heterogeneous graph G guided by the motion in step S2 mt and the audio-guided heterogeneous graph G of step S3 s t Aggregation, specifically, merging the graph nodes of the two heterogeneous graphs through a linear layer, and finally obtaining the graph node N1 after audio-visual interaction t =[v1 t ;a1 t ];in, is the visual feature after audio-visual interaction, a1 t ∈R 1×d It is the audio feature after audio-visual interaction.

[0127] S5, according to the query question, use the text encoder of CLIP-VIT-L / 14 to extract the question feature q∈R K×d , K is the number of words, d is the feature dimension; search the visual area of ​​each video frame to select the key visual area related to the question answer. The algorithm executed is as follows:

[0128] ;

[0129] Among them, α i t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The regional visual feature v1 of the i-th visual area in i t The similarity between (cosine similarity); α t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The similarity between the regional visual features in . The top r% of regional visual features with the highest similarity are retained, and the remaining regional visual features are reassigned to 0 and will not be updated in the subsequent process. The visual area mask β is obtained according to the problem characteristics. i t Visual features v1 t After mask operation, sparse visual feature v2 is obtained t .

[0130] S6, constructing a problem-guided heterogeneous graph G q t .

[0131] S61, for each time t, the sparse visual feature v2 t It is concatenated with the problem features as a graph node, denoted as M t =[v2 t ; q].

[0132] S62, Computing the connectivity matrix of problem-guided heterogeneous graphs via cosine similarity , as shown below:

[0133] A q t = M t M t T = [v2 t ; q][v2 t ; q] T ;

[0134] S63, graph node M of heterogeneous graph guided by question t and the connection matrix A q t , construct a problem-guided heterogeneous graph G q t , as shown below:

[0135] G q t =ReLU(A q t M t W q );

[0136] Among them, W q ∈R d×d is a motion-guided heterogeneous graph G q t The learnable parameters of .

[0137] G q t (M t )=[v q t ; q q t ],v q t The heterogeneous graph G guided by the problem q t The graph node M t The local visual features in ,q q t The heterogeneous graph G guided by the problem q t The graph node M t Characteristics of the problem.

[0138] S7, multimodal fusion, because the final answer to the question needs to be realized, the present invention designs a multi-scale visual fusion method, which is composed of heterogeneous graphs of the question modality and other modalities, as follows:

[0139] Construct the first heterogeneous graph of the question modality and the audio modality as follows:

[0140] The problem feature q∈R K×d The audio feature I1 after audio-visual interaction = {a1 t |t=1,...,T}∈R T×d The nodes γ1=[q; I1]∈R of the first heterogeneous graph are concatenated together. (K+T)×d , calculate the connection matrix A1=[q ;I1][q ;I1] of the first heterogeneous graph T , the first heterogeneous graph G qa =ReLU(A1γ1W1); where W1∈R d×d is the first heterogeneous graph G qa The learnable parameters of G qa (γ1)=[q qa ; a qa ], q qa is the first heterogeneous graph G qa The question feature in the graph node γ1 is the question feature after fusion with the audio feature after audio-visual interaction, a qa is the first heterogeneous graph G qa The audio features in the graph node γ1 of .

[0141] Construct the second heterogeneous graph of question modality and local visual features as follows:

[0142] The problem feature q∈R K×d With the local visual feature V1={v q t |t=1,...,T} are concatenated together as the graph node γ2=[q ; V1] of the second heterogeneous graph, , calculate the connection matrix A2=[q ; V1][q ; V1] of the second heterogeneous graph T , the second heterogeneous graph G qV1 =ReLU(A2γ2W2); where W2∈R d×d It is the second heterogeneous graph G qV1 The learnable parameters, G qV1 (γ2)=[q qV1 ; v qV1 ], q qV1 The second heterogeneous graph G qV1 The problem feature in the graph node γ2, that is, the problem feature after fusion with the local visual feature, v qV1 The second heterogeneous graph G qV1 The local visual features in the graph node γ2 of .

[0143] Construct the third heterogeneous graph of the question modality and the global visual modality as follows:

[0144] The local visual feature V1={v qt |t=1,...,T} and take the average to get the global visual feature V2∈R T×d , the problem feature q∈R K×d With the global visual feature V2∈R T×d The nodes γ3=[q; V2]∈R are concatenated as the third heterogeneous graph. (K+T)×d , calculate the connection matrix A2=[q ; V2][q ; V2] of the third heterogeneous graph T , the third heterogeneous graph G qV2 =ReLU(A3γ3W3); where W3∈R d×d It is the third heterogeneous graph G qV2 The learnable parameters, G qV2 (γ3)=[q qV2 ; v qV2 ], q qV2 is the third heterogeneous graph G qV2 The problem feature in the graph node γ3, that is, the problem feature after fusion with the global visual feature, v qV2 is the third heterogeneous graph G qV2 The global visual features in the graph node γ3 of .

[0145] S8, the q obtained in step S7 is qa ,q qV1 and q qV2 After superposition, the prediction result of the answer to the question is obtained through the fully connected network layer.

[0146] In this embodiment, the cross entropy loss function is used as the optimization target to train the network model. When training the network, the AdamW optimizer is used and the learning rate is set to 1×10 -4 , a total of 35 cycles of training.

[0147] An audio-visual video question-answering system based on a multimodal heterogeneous graph, wherein the network model of the system includes: a motion-guided heterogeneous graph module, an audio-guided heterogeneous graph module, a question-guided heterogeneous graph module, and a multimodal fusion reasoning module. The motion-guided heterogeneous graph module is used to construct a motion-guided heterogeneous graph. The audio-guided heterogeneous graph module is used to construct an audio-guided heterogeneous graph. The question-guided heterogeneous graph module is used to construct a question-guided heterogeneous graph. The multimodal fusion reasoning module is used to perform multimodal fusion, and obtain the prediction result of the answer to the question according to the question features after multimodal fusion.

[0148] In this embodiment, experiments are conducted on the MUSIC-AVQA benchmark dataset, and the experimental results of the method of the present invention are compared with the experimental results of various methods in the prior art. The comparison is shown in Table 1 below:

[0149] Table 1 Comparative analysis of question-answering accuracy

[0150]

[0151] According to Table 1, the performance of the method of the present invention greatly exceeds other audio-visual video question-answering methods in the prior art. The sources of the above-mentioned other audio-visual video question-answering methods in the prior art are as follows:

[0152] [1] Li G, Wei Y, Tian Y, et al. Learning to answer questions in dynamic audio-visual scenarios[C] / / Proceedings of the IEEE / CVF Conference on ComputerVision and Pattern Recognition. 2022: 19108-19118.

[0153] [2]Lao M, Pu N, Liu Y, et al. Coca: Collaborative causal regularization for audio-visual question answering[C] / / Proceedings of the AAAI Conference on Artificial Intelligence. 2023, 37(11): 12995-13003.

[0154] [3]Li G, Hou W, Hu D. Progressive spatio-temporal perception for audio-visual question answering[C] / / Proceedings of the 31st ACM International Conference on Multimedia. 2023: 7808-7816.

[0155] [4]Lin YB, Sung YL, Lei J, et al. Vision transformers are parameter-efficient audio-visual learners[C] / / Proceedings of the IEEE / CVFConference on Computer Vision and Pattern Recognition. 2023: 2299-2309.

[0156] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An audio-visual video question answering method based on multimodal heterogeneous graph, characterized in that: The following steps are involved: S1, extracts visual features and audio features from audio and video; S2, constructs motion-guided heterogeneous graphs based on visual and audio features; S3, constructs an audio-guided heterogeneous graph based on visual features and audio features; S4, aggregate the motion-guided heterogeneous graph and the audio-guided heterogeneous graph to obtain the visual features and audio features after audio-visual interaction; S5, extracting the problem features in the query question, selecting the key visual area according to the problem features, and obtaining sparse visual features; S6, constructing a question-guided heterogeneous graph based on the question features and sparse visual features to obtain local visual features and global visual features; S7, multimodal fusion: the question features are fused with the audio features, local visual features and global visual features after audio-visual interaction to obtain fused question features; S8, based on the question features after multimodal fusion, obtain the prediction result of the answer to the question; In step S1, the audio and video are divided to obtain T video frames and T audio clips; feature extraction is performed on each video frame and audio clip to obtain visual features and audio features of the audio and video; Among them, the visual features are recorded as , , v t is the visual feature of the tth video frame, , N 2 represents the number of visual areas in the video frame image, and d is the feature dimension; The audio feature is recorded as I={a t |t=1,...,T},I∈R T×d , where a t is the audio feature of the t-th audio clip, a t ∈R 1×d ; The specific process of step S2 is as follows: S21, calculate the motion density m of each video frame t , calculated as: First, the motion density ρ of each visual area is calculated i t : ； Among them, ||·|| represents the feature binary norm normalization processing; v i t With v i t+1 represents the regional visual features corresponding to the i-th visual area in the adjacent video frames, i.e., the t-th video frame and the t+1-th video frame; ρ i t represents the motion density of the i-th visual area in the t-th video frame; Then the motion information ρ of the tth video frame is calculated t : ； Get the average value μ of the motion information of all video frames: ； Finally, the motion density m of each video frame is calculated. t , m t = (1-λ)ρ t +λμ; where λ is the balance coefficient; S22, obtain the connection matrix A of the motion-guided heterogeneous graph m t : ； in, Indicates the dot product between two matrix elements; the superscript T indicates matrix transposition, and the symbol ";" indicates matrix concatenation; S23, determining the graph node N of the motion-guided heterogeneous graph t =[v t ; a t ]; S24, construct motion-guided heterogeneous graph G m t , G m t =ReLU(A m t N t W m ); Among them, W m is a motion-guided heterogeneous graph G m t The learnable parameters of , ReLU is the activation function; G m t (N t ) = [v m t ; a m t ],v m t Heterogeneous graph G guided by motion m t The graph node N t The visual features in a m t Heterogeneous graph G guided by motion m t The graph node N t Audio features in ; The specific process of step S3 is as follows: S31, calculate the sound activation matrix s for each video frame t : ； in, Represents matrix multiplication; S32, obtaining the connection matrix A of the audio-guided heterogeneous graph s t : ； S33, determining the graph node N of the audio-guided heterogeneous graph t =[v t ; a t ]; S34, constructing audio-guided heterogeneous graph G s t , G s t =ReLU(A s t N t W s ); Among them, W s is the audio-guided heterogeneous graph G s t The learnable parameters of G s t (N t ) = [v s t ; a s t ],v s t The heterogeneous graph G guided by audio s t The graph node N t The visual features in a s t The heterogeneous graph G guided by audio s t The graph node N t The audio features in .

2. The method for audio-visual video question answering based on multimodal heterogeneous graph according to claim 1, characterized in that: In step S4, the motion-guided heterogeneous graph G of step S2 is m t and the audio-guided heterogeneous graph G of step S3 s t Aggregation, specifically, merging the graph nodes of the two heterogeneous graphs through a linear layer, and finally obtaining the graph node N1 after audio-visual interaction t =[v1 t ;a1 t ];in, is the visual feature after audio-visual interaction, a1 t ∈R 1×d It is the audio feature after audio-visual interaction.

3. The method for audio-visual video question answering based on multimodal heterogeneous graph according to claim 2, characterized in that: In step S5, the question feature in the query question is extracted as q∈R K×d , K is the number of words, d is the feature dimension; select the key visual area according to the problem feature q and obtain the sparse visual features, as shown below: ； Among them, α i t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The regional visual feature v1 of the i-th visual area in i t The similarity between t Represents the visual feature v1 after the question feature q and the audio-visual interaction t The similarity between the regional visual features in β i t is the visual region mask obtained according to the problem features, which is used to retain the top r% of the regional visual features with the highest similarity, and the remaining regional visual features are reassigned to 0; using the visual region mask β i t Visual features after audio-visual interaction v1 t After mask operation, sparse visual features v2 are obtained t .

4. The method for audio-visual video question answering based on multimodal heterogeneous graph according to claim 3, characterized in that: The specific process of step S6 is as follows: S61, determine the graph node M of the problem-guided heterogeneous graph t =[v2 t ; q]; S62, calculate the connection matrix A of the problem-guided heterogeneous graph q t : A q t = M t M t T = [v2 t ; q][v2 t ; q] T ; S63, constructing a problem-guided heterogeneous graph G q t , G q t =ReLU(A q t M t W q ); Among them, W q is a motion-guided heterogeneous graph G q t The learnable parameters of G q t (M t )=[v q t ; q q t ],v q t A heterogeneous graph G guided by the problem q t The graph node M t The local visual features in ,q q t The heterogeneous graph G guided by the problem q t The graph node M t Characteristics of the problem.

5. The method for audio-visual video question answering based on multimodal heterogeneous graph according to claim 4, characterized in that: In step S7, The question features are fused with the audio features after audio-visual interaction to construct the first heterogeneous graph of the question modality and the audio modality. The specific method is as follows: The audio feature I1 after the question feature q and the audio-visual interaction is t |t=1,...,T} are concatenated as the graph node γ1=[q ; I1] of the first heterogeneous graph, and the connection matrix A1=[q ; I1][q ; I1] of the first heterogeneous graph is calculated. T , the first heterogeneous graph G qa =ReLU(A1γ1W1); where W1 is the first heterogeneous graph G qa The learnable parameters of G qa (γ1)=[q qa ; a qa ], q qa is the first heterogeneous graph G qa The question feature in the graph node γ1 is the question feature after fusion with the audio feature after audio-visual interaction, a qa is the first heterogeneous graph G qa The audio features in the graph node γ1 of ; The problem features are fused with the local visual features to construct a second heterogeneous graph of the problem modality and the local visual features. The specific method is as follows: The problem feature q is combined with the local visual feature V1={v q t |t=1,...,T} are concatenated as the graph node γ2=[q ; V1] of the second heterogeneous graph, and the connection matrix A2=[q ; V1][q ; V1] of the second heterogeneous graph is calculated. T , the second heterogeneous graph G qV1 =ReLU(A2γ2W2); where W2 is the second heterogeneous graph G qV1 The learnable parameters of G qV1 (γ2)=[q qV1 ; v qV1 ], q qV1 The second heterogeneous graph G qV1 The problem feature in the graph node γ2, that is, the problem feature after fusion with the local visual feature, v qV1 The second heterogeneous graph G qV1 The local visual features in the graph node γ2; The problem features are fused with the global visual features to construct a third heterogeneous graph of the problem modality and the global visual modality. The specific method is as follows: The local visual feature V1={v q t |t=1,...,T} and take the average to get the global visual feature V2. The problem feature q and the global visual feature V2 are concatenated as the graph node γ3=[q ; V2] of the third heterogeneous graph. The connection matrix A3=[q ; V2][q ; V2] of the third heterogeneous graph is calculated. T , the third heterogeneous graph G qV2 =ReLU(A3γ3W3); where W3 is the third isomorphic graph G qV2 The learnable parameters of G qV2 (γ3)=[q qV2 ; v qV2 ], q qV2 is the third heterogeneous graph G qV2 The problem feature in the graph node γ3, that is, the problem feature after fusion with the global visual feature, v qV2 is the third heterogeneous graph G qV2 The global visual features in the graph node γ3 of .

6. The method for audio-visual video question answering based on multimodal heterogeneous graph according to claim 5, characterized in that: In step S8, the fused problem feature q obtained in step S7 is qa ,q qV1 and q qV2 After superposition, the prediction result of the answer to the question is obtained through the fully connected network layer.

7. An audio-visual video question-answering system based on a multimodal heterogeneous graph, characterized in that: An audio-visual video question-answering method based on a multimodal heterogeneous graph applicable to any one of claims 1 to 6 above, the system comprising: a motion-guided heterogeneous graph module, an audio-guided heterogeneous graph module, a question-guided heterogeneous graph module and a multimodal fusion reasoning module; The motion-guided heterogeneous graph module is used to construct a motion-guided heterogeneous graph; The audio-guided heterogeneous graph module is used to construct an audio-guided heterogeneous graph; The problem-guided heterogeneous graph module is used to construct a problem-guided heterogeneous graph; The multimodal fusion reasoning module is used to perform multimodal fusion and obtain a prediction result of the answer to the question based on the question features after multimodal fusion.

Images