Improved audiovisual questions and answers

Abstract

This disclosure describes techniques for improving audiovisual questions and answers. The machine learning model is configured for audiovisual questions and answers (AVQA). The machine learning model comprises a first sub-model configured to capture semantic audio information and output an audio spatial feature map # imgabs0 # machine learning model comprises a second sub-model configured to extract visual features xvs and audio features xas and further configured to obtain a problem vector xq. The machine learning model includes a third sub-model configured to capture audiovisual correspondence at a granularity level. A balanced AVQA data set is created. The balanced AVQA dataset includes a balanced answer distribution in each question category. A machine learning model is trained to use at least a subset of the balanced AVAQ dataset to answer questions related to visual objects, sounds, and associations thereof in the video.

Description

Background Art

[0001] Machine learning models are increasingly being used in a variety of industries to perform a variety of different tasks. Such tasks may include audio-visual tasks. Improved techniques for utilizing and training machine learning models are desirable. BRIEF DESCRIPTION OF THE DRAWINGS

[0002] The following detailed description may be better understood when read in conjunction with the accompanying drawings. For the purpose of illustration, example embodiments of various aspects of the disclosure are shown in the drawings; however, the invention is not limited to the specific methods and instrumentalities disclosed.

[0003] Figure 1 An example system for improving audio-visual responses according to the present disclosure is shown.

[0004] Figure 2 Example training data imbalance according to the present disclosure is shown.

[0005] Figure 3 Another example training data imbalance according to the present disclosure is shown.

[0006] Figure 4 Example distributions of training data before and after balancing according to the present disclosure are shown.

[0007] Figure 5 An example training data sample according to the present disclosure is shown.

[0008] Figure 6 An example process for improving audio-visual responses that may be performed in accordance with the present disclosure is shown.

[0009] Figure 7 Another example process for improving audio-visual responses that may be performed in accordance with the present disclosure is shown.

[0010] Figure 8 Another example process for improving audio-visual responses that may be performed in accordance with the present disclosure is shown.

[0011] Fig. 9 Another example process for improving audio-visual responses that may be performed in accordance with the present disclosure is shown.

[0012] Fig.10 Example qualitative results according to the present disclosure are shown.

[0013] Fig.11 Example quantitative results according to the present disclosure are shown.

[0014] Fig.12 Example quantitative results according to the present disclosure are shown.

[0015] Fig.13Example quantitative results according to the present disclosure are shown.

[0016] Fig.14 Example quantitative results according to the present disclosure are shown.

[0017] Fig.15 Example quantitative results according to the present disclosure are shown.

[0018] Fig.16 An example computing device is shown that can be used to perform any of the techniques disclosed herein. DETAILED DESCRIPTION

[0019] In audio-visual learning, the interaction between audio and visual information provides a rich way to understand dynamic scenes. A special task that embodies this synergy is audio-visual question answering (AVQA). Given a video of an instrument performance and the associated music audio, the AVQA machine learning model can be used to answer questions associated with the video, audio, and / or the relationship between the video and audio. Unlike the visual question answering (VQA) task that only handles two modalities (e.g., vision and language), AVQA is designed to bridge and reason across three modalities (e.g., vision, language, and audio).

[0020] In order to train the AVQA machine learning model, a training dataset (hereinafter referred to as the "MUSICAVQA dataset") can be used. The MUSICAVQA dataset includes three main question categories by modality: audiovisual questions, visual questions, and audio questions. Across all three question categories, five question aspects are considered, including existence question aspects (e.g., "yes / no" questions asking whether something exists), time question aspects (e.g., questions asking the order in which musical instruments are performed), counting question aspects (e.g., questions asking the number of objects or sounds in a video), location question aspects (e.g., questions asking the location of the performance and / or where a specific performer is located), and comparison question aspects (e.g., questions comparing different instruments in a video). By combining the modality type with various question aspects, the MUSICAVQA dataset ultimately obtained 33 question templates in which the name of an instrument can be changed from one instrument name to another based on the video. Each question template contains a fixed set of answers, ranging from binary answers (e.g., "yes" and "no") to counting answers (e.g., "1", "2", "3", etc.).

[0021] However, the MUSICAVQA dataset has strong biases that undermine the reliability of the MUSICAVQA dataset as a trusted benchmark. For example, in a specific question category that asks whether the audio track of a video comes from the instrument being played in the video, over 90% of the answers in the dataset are "yes". For audio-visual timing questions, when asked which instrument sounded first in the video, nearly 80% of the answers in the dataset are "at the same time". For counting questions, more than half of the answers in the dataset are small numbers (e.g., 1 and 2). These imbalances exist in all five question aspects. Such biases may have a negative impact on the training of AVQA models, as the model will be trained to favor the most common answers in the training set, while ignoring the importance of video and audio and the reasoning between the three. Therefore, improving audio-visual answering techniques is desirable.

[0022] Improved techniques for audio-visual answering are described herein. The system described herein includes a machine learning model that is capable of learning connections across three different modalities. The machine learning model described herein extends existing methods by adding an additional audio spectrogram transformer (AST) branch for audio-visual localization and designing cross-modal pixel attention between audio and visual space graphs. The machine learning model described herein is trained on an improved balanced dataset (hereinafter referred to as MUSIC-AVQA v2.0). Because the dataset is more balanced than the MUSICAVQA dataset, the machine learning model described herein performs better than the existing AVQA machine learning model.

[0023] Figure 1 An example system 100 for improving audiovisual answering is illustrated. System 100 may include a machine learning model configured for audiovisual question answering (AVQA). The machine learning model may include a first sub-model 102, a second sub-model 104, and a third sub-model 106. The machine learning model may be configured to answer an audiovisual question associated with a music performance video (e.g., "How many violins are there that sound in the video?"). The machine learning model may receive the audiovisual question. For example, the audiovisual question may be received by a question encoder of the second sub-model 104. The question encoder may generate a question vector x representing the audiovisual question. q .

[0024] The first sub-model 102 may include an audio-visual fusion branch. The first sub-model 102 may include an audio spectrogram transformer (AST). The first sub-model 102 may receive as input an audio stream (e.g., a spectrogram) associated with a music performance video. The first sub-model 102 may be configured to extract audio features from the audio stream using a pre-trained AST. The first sub-model 102 may merge the extracted features with the visual feature branch of the second sub-model 104 using audio-visual spatial and temporal localization. The first sub-model 102 may enable the system 100 to capture richer semantic audio information than existing systems. Specifically, the hidden output of the first sub-model 102 may be extracted. The hidden output of the first sub-model 102 may include a spatial feature map The first sub-model 102 can be in the spatial feature map Average pooling is applied to generate the audio vector

[0025] The second sub-model 104 may receive as input an audio stream (e.g., a spectrogram) and a visual stream associated with the music performance video. The second sub-model 104 may be configured to extract visual features (e.g., a visual graph) from the audio stream and the visual stream. vs and audio feature x as For example, the second sub-model 104 may include a 2-tower pre-trained Swin-Transformer V2 with an audio-visual adapter for audio-visual fusion (e.g., “LAVISH”). The LAVISH component can extract visual features (e.g., visual graphs) from audio streams and videos. vs and audio feature x as The second sub-model 104 can be used to analyze the audio feature x as Apply average pooling to produce the audio vector x a The second sub-model 102 may be relative to the visual feature (eg, visual map) x vs To calculate the audio vector x a The resulting spatial localization module output is a vector in each frame Thus capturing the visual features that are focused on by the audio. Combined with audio features, temporal localization can be achieved in is applied, and then the question vector x generated by the long short-term memory (LSTM) encoder along the time axis q focus on.

[0026] The first sub-model 102 may be configured with respect to the visual features (eg, visual map) x output from the visual branch of the second sub-model 104. vs , calculate the audio vector The resulting spatial localization module output is the vector Thereby capturing the visual features that the audio focuses on. After that, the first sub-model 102 can be constructed along the time axis by the question vector x q The first audio-visual fusion vector of interest Execute time positioning on. It can be concatenated with the audio features and then generated by the long short-term memory (LSTM) encoder along the time axis to produce the question vector x q The output of the time positioning module can be the first problem perceptual audio and video fusion vector

[0027] The third sub-model 106 may include a cross-modal pixel attention sub-model. The third sub-model 106 may be configured to capture and refine audio-visual correspondences at a granular level. Existing spatial localization methods use mean-pooled audio vectors to compute attention with visual spatial graphs. However, using mean-pooled audio vectors to compute attention with visual spatial graphs results in the loss of spatial details of spectrogram features and is unidirectional (e.g., only the audio vector is used as a query to the visual graph without any reciprocal interaction). To address these limitations, the third sub-model 106 is based on the visual features x vs and audio feature x as , aiming to more effectively capture the correspondence between visual and audio modalities. Specifically, given x vs ∈R H×W×C represents the frame-level visual map output from the second sub-model 104 and x as ∈R H×W×C Representing the frame-level features of the spectrogram output from the second sub-model 104, the third sub-model 106 can calculate the two pixel-wise audiovisual attention between the two maps. The spatial dimensions of the two feature maps can first be flattened to (HW)×C, resulting in x′ vs and x′ as Then, the cross attention between the two flattened graphs can be calculated, where each graph pays attention to the other graph. The cross attention between the two flattened graphs can be expressed as follows:

[0028]

[0029] The obtained x vas and x avs denote the pixel fusion images for the audio and visual branches respectively. The third sub-model 106 may normalize the two images. The third sub-model 106 may average pool the spatial dimensions of the normalized images to generate two vectors x av and xva The third sub-model 106 can convert the two vectors x av and x va concatenate to generate the second audio-visual fusion vector Second audio-visual fusion vector Can be used with (i.e., the outputs of the spatial localization modules from the first sub-model 102 and the second sub-model 104) have the same dimension. Therefore, subsequent operations (such as temporal localization) are consistent with the other two branches. The third sub-model 106 can be located along the time axis by the problem vector x q The second audio-visual fusion vector of interest The third sub-model 106 can generate a second problem perceptual audio-visual fusion vector

[0030] The system 100 can concatenate the outputs from various branches of the machine learning model to generate the final problem-aware audio-visual fusion vector x qav The system 100 can be used to generate a problem-aware audio-visual fusion vector (e.g., the output of the time localization module of the second sub-model 104), the first problem perceptual audio-visual fusion vector (e.g., the output of the temporal localization module of the first sub-model 102) and the second problem perceptual audio-visual fusion vector (e.g., the output of the third sub-model 106) to generate the final problem perceptual audio-visual fusion vector x qav . Problem-aware audio-visual fusion vector First question: Perceptual audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector The connections can be passed to a multilayer perceptron (MLP) for classification. The classification can indicate an answer to an audio-visual question (e.g., “How many violins are there in the video?”). For example, the answer can be “three” or any other suitable answer to the audio-visual question.

[0031] Machine learning models can be trained on balanced datasets. Existing AVQA machine learning models can be trained on imbalanced datasets. Figure 2-Figure 3 Showing the imbalance of examples in the existing AVQA training dataset. Figure 2Example training data imbalances for audiovisual counting questions are shown. For counting question 201, "How many instruments sound in the video?" there may be 1036 training data samples. Of these 1036 training data samples, 24% are associated with the answer "1", 54% are associated with the answer "2", 14% are associated with the answer "3", and only 8% are associated with answers greater than "3". Similar imbalances occur for other audiovisual counting questions. For counting question 202, "How many types of instruments sound in the video?" there may be 943 training data samples. Of these 943 training data samples, 29% are associated with the answer "1", 56% are associated with the answer "2", 12% are associated with the answer "3" and only 3% are associated with answers greater than "3". For counting question 203, "How many instruments in the video have no sound from the beginning to the end?" there may be 1081 training data samples. Of these 1081 training data samples, 66% are associated with the answer "0", 23% are associated with the answer "1", 6% are associated with the answer "2", and only 5% are associated with answers greater than "2". There may be 139 training data samples for count question 204 "How many sounding objects are there in the video?" Of these 139 training data samples, 9% are associated with the answer "0", 53% are associated with the answer "1", 23% are associated with the answer "2", 7% are associated with the answer "3" and only 8% are associated with answers greater than "3". Therefore, when given an audio-visual count question, an AVQA machine learning model trained on this data may be strongly biased towards answering smaller numbers.

[0032] Figure 3 An example training data imbalance 300 for a visual-audio time question is shown. For the visual-audio time question "Which object made a sound first or last?", four different answers are possible: "left", "center", "right", and "simultaneously". There may be 619 training samples associated with this question. Of the 619 training samples, 448 (e.g., 72.5%) are associated with the answer "simultaneously". Therefore, when given a visual-audio time question, an AVQA machine learning model trained on this data may be strongly biased towards answering "simultaneously".

[0033] A balanced training dataset (e.g., MUSIC-AVQA v2.0) can be generated for training a machine learning model. The balanced training dataset enriches the unbalanced training dataset with additional (e.g., additional 1204) real videos, thereby additionally including 7.9k question-answer pairs. A large portion of the additional videos may be videos of instrumental ensembles involving three or more instruments, thereby capturing more complex audio-visual relationships.

[0034] In an embodiment, generating a balanced training data set may include pinpointing biased questions in an unbalanced training data set. In order to pinpoint biased questions, the distribution of answers for each question template under each question type may be determined or analyzed. If a single answer represents more than a threshold percentage (e.g., 60%) of responses to binary questions or more than a threshold percentage (e.g., 50%) of multi-class questions (e.g., questions with more than two possible answers), it may be determined that the question is biased. For example, it may be determined that one or more of audiovisual presence questions, audiovisual count questions, audiovisual time questions, visual location questions, visual count questions, audio count questions, and audio comparison questions may exhibit a skewed answer distribution for at least one of their templates. Within these types, multiple templates may exhibit deviations.

[0035] In the imbalanced training dataset, binary questions dominate for audio-visual problems. For binary questions, the answer is "yes" or "no". For example, for the question "Is this sound from the instrument in the video?", the answer to 90% of the data samples in the imbalanced training dataset is "yes". In order to create a question-answer pair with the answer "no", the sound track in the video can be replaced with audio of other instrument types. In order to make the question-answer pair non-trivial, the instrument set can be classified as "strings", "woodwinds", "brass", and "percussion". There is a 50% chance that the sound track will be replaced by a different instrument of the same classification, and a 50% chance that the sound track will be replaced by instrumental music belonging to other classifications. Using this method, 794 videos paired with unmatched audio clips can be created. For the question "Is there a voiceover?", the answers in the imbalanced training dataset may be seriously unbalanced (for example, 79.6% of the answers are "no"). This imbalance may arise from the fact that some labels define "voice-over" as a human voice appearing over instrumental sounds, while others define it as any general "off-screen" sound. To address such inconsistencies, the term "voice-over" can be defined as any "off-screen sound". After manually examining 1,278 video-audio-question pairs in the imbalanced training set, the incorrectly labeled entries can be corrected (e.g., 169 incorrectly labeled entries). Despite this correction, a serious imbalance may still exist, with 68% of the labels still being "no". To address this issue, additional question-answer pairs can be added. For example, in the case where "voice-over" appears (answered "yes"), an additional 456 question-answer pairs can be added, resulting in a balanced distribution of 51.7% "yes" and 48.3% "no" answers.

[0036] In an unbalanced training dataset, for audio-visual counting problems, the problem can be structured using four templates that address the following aspects of counting: (i) the total number of sounding instruments, (ii) the number of instrument types that are sounding, (iii) the number of specific sounding instrument types, and (iv) the number of instruments that do not sound from the beginning to the end. The answers are limited to "0-10" and "more than 10". In the unbalanced training dataset, "0", "1", and "2" dominate the answers. For all four templates, the most common answers exceed 50% of the total, and one template even reaches 60%. In order to balance the audio-visual counting problem, music ensemble performance videos with at least one answer exceeding two in the above question templates can be collected (e.g., manually or through a computing device). For example, music ensemble performance videos tagged with terms such as "music ensemble", "string quartet", or specific instrument names can be collected. For each instrument type, videos tagged with that type can be collected and combined with videos tagged as "music ensemble", "string quartet", "quartet ensemble", or specific instrument names, which helps narrow the range of potential candidates. From this collection, videos of very low quality, with static scenes (e.g., album covers), or with ambiguous content can be filtered out (e.g., manually or by a computing device). For each selected video, the following can be annotated: (i) the total number of instruments, (ii) the number of different instrument types, (iii) the count of each instrument type, (iv) the number of sounding instruments, (v) the number of different sounding instrument types, (vi) the number of most frequently occurring instruments that also sound, and (vii) the number of instruments that sound from the beginning to the end of the video. In an embodiment, a total of 591 videos for the audio-visual count question can be collected. Using annotations from these videos, additional question-answer pairs can be generated for each template: 572 (+39%) for T1, 502 (+25.4%) for T2, 815 (+40.1%, 350 from the original unlabeled videos for the question template in the imbalanced dataset, 465 from the collected videos) for T3, and 413 (+30.3%) for T4. These new pairs have answers from less frequent answer categories. After adding these pairs, the imbalance problem of all four question templates is fully alleviated, where the percentage of the most common answer for the four templates drops by 16%, 17%, 15%, and 13%, respectively.

[0037] In the imbalanced training dataset, for audio-visual time questions, the candidate answers range from 22 instrument categories, as well as position indicators that specify the position of the instrument, such as "left", "middle", "right". Additionally, the term "simultaneous" means that the instruments are played simultaneously. Among the three question templates for this question category, the question "Which object is the first / last sound?" shows a strong imbalance: 74% of the answers are "simultaneous". In order to address this imbalance in a multi-class environment, question-answer pairs with answers of "left", "right" or "middle" can be marked to weaken the dominance of the "simultaneous" category. For example, for a video in which three violinists are performing, if the violinist on the left starts to perform, followed by the violinists in the middle and on the right, the question-answer pair can be formulated as: "Question: Which violin starts playing first? Answer: left". Additionally, the video can be enhanced by horizontal flipping. This transforms the question-answer pair into: "Question: Which violin starts playing first? Answer: right". After the above process, additional target videos (e.g., additional 203 videos) can be collected to create question-answer pairs. After augmentation, there may be 713 (+81.1%) additional question-answer pairs in the dataset whose answer is not “simultaneously”, thereby reducing the most frequent answer percentage from 74% to 41%, a reduction of 33%.

[0038] In the imbalanced training dataset, unlike audio-visual questions, visual counting questions rely entirely on visual information. The first two templates, "Is there an object in the entire video" and "Is there an object and an object instrument in the video", determine the presence of a specific musical instrument. For these templates, most of the answers are "yes", accounting for 78.4% and 62.7%, respectively. To eliminate this bias, additional (e.g., 794 and 423, respectively) question-answer pairs with the answer "no" can be generated for each template. These pairs can be created using labels in the collected videos. The third template focuses on counting the types of musical instruments present in the video. The third template is associated with a significant imbalance, with answers "1" and "2" accounting for 91% of all responses. To address this issue, labels from 591 videos in the audio-visual count collection can be used. Videos with more than 2 answers to this question can be selected. This selection can help reduce the dominance of the first two answer categories, reducing their combined percentage from 91% to 80%.

[0039] In the imbalanced training dataset, for visual location questions, this question category involves the performance location and specific positioning of the performers. It seeks the answer whether the performance is indoors or outdoors and asks the relative position of the instruments in the video. Among the four question templates in this category, the template "Where is the performance?" with the answer "indoor" or "outdoor" shows a clear imbalance. To address this issue, additional (e.g., 456) question-answer pairs with the answer "outdoor" can be collected, thereby reducing the dominant category of question-answer pairs ("indoor") from 72.6% to 54.7%, a reduction of 17.9%.

[0040] In the imbalanced training dataset, for audio counting questions, the first two question templates, “Is there an object sound?” and “Are there sounds of object 1 and object 2?” focus on the audio aspect, determining the presence of specific sounds. For these templates, most of the answers are “yes”, accounting for 76.0% and 78.8%, respectively. To address this imbalance, additional (e.g., 794 and 423, respectively) question-answer pairs with the answer “no” can be created for each template using labels in the video where the sounding instruments are identified. This reduces the dominance of the “yes” answers for the two templates to 52.5% and 61.5%, respectively. The third template queries the total number of different instrument sounds heard in the entire video. It is worth noting that the answers “1” and “2” account for more than 89% of all responses, highlighting the significant imbalance. To address this issue, question-answer pairs collected from ensemble performance videos can be labeled, especially videos where the answer to the question is neither “1” nor “2”. After balancing, the combined advantage of the “1” and “2” answers is reduced by 28% to 61%.

[0041] In the imbalanced training dataset, for audio comparison questions, this type of question compares different instrument sounds in a video in terms of loudness, duration, and tempo. Among the 3 question templates, the question template asking which instrument played longer has a strong imbalance, with the answer "no" (indicating that no instrument played significantly longer) accounting for 68% of the data. To address this imbalance, question-answer pairs from previously unlabeled videos in the original imbalanced dataset can be curated to ensure that the first object is played longer than the second object. This reduces the dominance of the "no" answer to 53.9%.

[0042] Figure 4A comparison between a balanced dataset and an unbalanced dataset 400 is shown. The balanced dataset includes a balanced distribution of answers for all three major question categories: audio-visual questions, visual questions, and audio questions. Across all three question categories, the balanced dataset includes a balanced distribution of answers for five question aspects, including presence question aspects, time question aspects, counting question aspects, location question aspects, and comparison question aspects. When trained on a balanced dataset, the machine learning model outperforms the AVQA machine learning model trained on an unbalanced dataset.

[0043] Figure 5 An overview of a training data sample 500 from a balanced training dataset is shown. Training sample 501 shows two videos. Each of the two videos has the same visuals. But each of the two videos has different audio. Training sample 502 shows two videos. Each of the two videos shows two violinists. But each of the two videos shows the two violinists in a different order. Training sample 503 shows two videos. Each of the two videos presents an acoustic guitar ensemble. But each of the two videos shows a different count of acoustic guitar ensembles. Training sample 504 shows two videos of a piano-flute duet. In the first video, the piano is played for a longer time. In order to answer accurately, the AVQA machine learning model must take into account these audio-visual nuances, not just the language priors. Therefore, the AVQA machine learning model trained on a balanced dataset performs better than the AVQA machine learning model trained on an unbalanced dataset.

[0044] Figure 6 An example process 600 performed by a system (e.g., system 100) is illustrated. System 100 may perform process 600 to improve audio-visual responses. Figure 6 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, delete, reorder, or modify the operations depicted.

[0045] At 602, a machine learning model may be configured. The machine learning model may be configured for audio-visual question answering (AVQA). The machine learning model may include a first sub-model. The first sub-model may include an audio spectrogram transformer (AST). The AST may be configured to capture semantic audio information. The AST may be configured to output an audio spatial feature map. The machine learning model may include a second sub-model. The second sub-model may be configured to extract the visual feature x vs and audio feature x as The second sub-model can also be configured to obtain the question vector x q. The machine learning model may include a third sub-model. The third sub-model may include a cross-modal pixel attention sub-model. The cross-modal pixel attention sub-model may be configured to capture audio-visual correspondences. The cross-modal pixel attention sub-model may be configured to capture audio-visual correspondences at a granular level.

[0046] At 604, a balanced AVQA dataset may be created. The balanced AVQA dataset includes a balanced distribution of answers for each question category. For example, the balanced AVQA dataset may include a balanced distribution of answers for three main question categories: audio-visual questions, visual questions, and audio questions. Across all three question categories, the balanced AVQA dataset may include a balanced distribution of answers for five question aspects, including a presence question aspect, a time question aspect, a count question aspect, a location question aspect, and a comparison question aspect.

[0047] At 606, a machine learning model can be trained. The machine learning model can be trained using at least a subset of the balanced AVAQ dataset. The machine learning model can be trained to answer questions related to visual objects, sounds, and their associations in the video. At least in part, because the machine learning model is trained on a balanced dataset, the machine learning model described herein performs better than an AVQA machine learning model trained on an unbalanced dataset.

[0048] Figure 7 An example process 700 performed by a first sub-model (e.g., first sub-model 102) is illustrated. First sub-model 102 may perform process 700 to improve an audio-visual response. Figure 7 Although depicted as a sequence of operations, one of ordinary skill in the art will understand that various embodiments may add, delete, reorder, or modify the operations depicted.

[0049] At 702, semantic audio information can be captured. The semantic audio information can be captured by a first sub-model. The first sub-model can receive as input an audio stream (e.g., a spectrogram) associated with a music performance video. The first sub-model can be configured to extract audio features from the audio stream using a pre-trained AST. Audio Spatial Feature Map Can be generated. Audio spatial feature map It can be generated by the first sub-model. Audio spatial feature map Can be generated based on the captured semantic audio information.

[0050] The first sub-model 102 may include an audio-visual fusion branch. The first sub-model 102 may include an audio spectrogram transformer (AST). The first sub-model 102 may receive an audio stream (e.g., a spectrogram) associated with a music performance video as input. The first sub-model 102 may be configured to extract audio features from the audio stream using a pre-trained AST. The first sub-model 102 may merge the extracted features with the visual feature branch of the second sub-model 104 using audio-visual spatial and temporal localization. The first sub-model 102 may enable the system 100 to capture richer semantic audio information than existing systems. Specifically, the hidden output of the first sub-model 102 may be extracted. The hidden output of the first sub-model 102 may include a spatial feature map The first sub-model 102 can be in the spatial feature map Average pooling is applied to produce an audio vector At 704, average pooling may be applied. Average pooling may be applied to the audio spatial feature map. is applied to generate the audio vector

[0051] At 706, spatial attention may be calculated. The spatial attention may be calculated relative to the visual features output from the second sub-model. The first sub-model may be calculated relative to the visual features (e.g., visual map) output from the visual branch of the second sub-model. vs , calculate the audio vector At 708, the first audio-visual fusion A vector may be generated. A first audio-visual fusion vector The visual features focused by the audio can be captured. At 710, time localization can be performed. Time localization can be performed along the time axis in the space defined by the question vector x. q The first audio-visual fusion vector of interest Execute on. It can be concatenated with the audio features and then generated by the long short-term memory (LSTM) encoder along the time axis to produce the question vector x q The output of the time positioning module can be the first problem perceptual audio and video fusion vector At 712, the first problem is to perceive the audio-visual fusion vector Can be generated.

[0052] Figure 8 An example process 800 performed by a third sub-model (e.g., third sub-model 106) is illustrated. The third sub-model 106 may perform process 800 to improve the audio-visual response. Figure 8 Although depicted as a sequence of operations, one of ordinary skill in the art will appreciate that various embodiments may add, delete, reorder, or modify the operations depicted.

[0053] The third sub-model may be a cross-modal pixel attention sub-model configured to capture and refine audio-visual correspondences at a granular level. vs With audio feature x as The pixel-wise audiovisual attention between the visual features x can be captured. vs With audio feature x as The pixel-wise audiovisual attention between can be captured by the third sub-model to capture the correspondence between the visual and audio modalities. The visual features and audio features can be extracted by the second sub-model. Specifically, given x vs ∈R H×W×C represents the frame-level visual map output from the second sub-model and x as ∈R H×W×C Representing the frame-level features of the spectrogram output from the second sub-model, the third sub-model can calculate the two-pixel audiovisual attention between the two maps. The spatial dimensions of the two feature maps can first be flattened to (HW)×C, resulting in x′ vs and x′ as Then the cross attention between the two flattened graphs can be computed, where each graph pays attention to the other graph.

[0054] The obtained x vas and x avs Denote the pixel fusion maps for the audio and visual branches respectively. The third sub-model may normalize the two maps. The third sub-model may average pool the spatial dimensions of the normalized maps to generate two vectors x av and x va The third submodel can convert two vectors x av and x va concatenate to generate the second audio-visual fusion vector At 804, the second audio-visual fusion vector The second audio-visual fusion vector The second audio-visual fusion vector Can be used with (i.e., the outputs of the spatial localization modules from the first sub-model and the second sub-model) have the same dimension. At 806, the temporal localization can be performed along the time axis in the space represented by the problem vector x q The second audio-visual fusion vector of interest At 808, the second problem perceptual audio-visual fusion vector Can be generated.

[0055] Fig. 9An example process 900 performed by a system (e.g., system 100) is illustrated. System 100 may perform process 900 to improve audio-visual responses. Fig. 9 Although depicted as a sequence of operations, one of ordinary skill in the art will appreciate that various embodiments may add, delete, reorder, or modify the operations described.

[0056] At 902, a machine learning model may be configured. The machine learning model may be configured for audio-visual question answering (AVQA). The machine learning model may include a first sub-model. The first sub-model may be an audio spectrogram transformer (AST). The AST may be configured to capture semantic audio information. The AST may be configured to output an audio spatial feature map. The machine learning model may include a second sub-model. The second sub-model may be configured to extract the visual feature x vs and audio feature x as The second sub-model can also be configured to obtain the question vector x q . The machine learning model may include a third sub-model. The third sub-model may be a cross-modal pixel attention sub-model. The cross-modal pixel attention sub-model may be configured to capture audio-visual correspondences. The cross-modal pixel attention sub-model may be configured to capture audio-visual correspondences at a granular level.

[0057] At 904, a balanced AVQA dataset can be created. The balanced AVQA dataset includes a balanced distribution of answers for each question category. For example, a balanced AVQA dataset can include a balanced distribution of answers for three main question categories: audio-visual questions, visual questions, and audio questions. In all three question categories, the balanced AVQA dataset can include a balanced distribution of answers for five question aspects, including existence question aspects, time question aspects, counting question aspects, location question aspects, and comparison question aspects. At 906, a machine learning model can be trained. The machine learning model can be trained using at least a subset of the balanced AVAQ dataset. The machine learning model can be trained to answer questions related to visual objects, sounds, and their associations in the video. At least to some extent, because the machine learning model is trained on a balanced dataset, the machine learning model described herein performs better than the AVQA machine learning model trained on an unbalanced dataset.

[0058] At 908, the problem-aware audio-visual fusion vector First question: Perceptual audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector Can be connected. Problem-aware audio-visual fusion vector Can be generated based on the output from the second sub-model. Question-aware audio-visual fusion vector First question: Perceptual audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector can be concatenated to generate the final question-aware audio-visual fusion vector. For example, the question-aware audio-visual fusion vector , the first problem is the perception of audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector The connection of can be passed to an MLP for classification. The classification can indicate the answer to the audio-visual question.

[0059] Fig.10 An example qualitative result 1000 according to the present disclosure is shown. Qualitative result 1000 shows some contrasting question-answer samples predicted by the machine learning model of system 100. Overall, the machine learning model of system 100 can correctly predict contrasting question-answer pairs by better reasoning about audiovisual context rather than relying on language priors. The performance of the existing AVQA machine learning model when trained on a balanced training dataset is compared with the performance of the same existing AVQA machine learning model when trained on an unbalanced training dataset. In particular, the performance of the existing models AVST and LAVISH when trained on biased datasets and balanced datasets is evaluated. The balanced test set is determined by sampling 20% ​​from MUSIC-AVQA v2.0, where stratified sampling is used to balance the answer distribution in each question category. Then, in this balanced test set, the biased test set is sampled by keeping the same question-answer distribution as the biased dataset (e.g., using corrected question-answer pairs in the "voiceover" category). Both the biased test set and the balanced test set are used to evaluate the existing models AVST and LAVISH. After keeping the balanced test set from MUSIC-AVQA v2.0, the remaining data is kept balanced. To ensure a fair comparison, the training set capacity for the biased and balanced datasets is equal. To achieve this, a biased subset is first sampled from the remaining balanced data. To maximize the number of training samples in this biased set, any question-answer pair whose answer belongs to the most common answer category within its question template is included in the biased subset. Once the balanced test set is taken from MUSICAVQA v2.0, the remaining is still a balanced dataset. To ensure a fair comparison, the training set capacity is kept the same for the biased and balanced sets. To achieve this, a biased subset is first sampled from the remaining balanced set. To maximize the number of training samples for the biased set, any question-answer pair whose answer is the most common answer category within its question template is included in the biased subset. For question-answer pairs in the remaining answer categories in the question template, the number is increased by To determine, where N most is the number of question-answer pairs whose answer is the most common answer category in the remaining balanced set of question templates, N′ mostis the number of question-answer pairs whose answers are the most common answer category for the question template in the imbalanced training dataset, and N′ other is the number of question-answer pairs whose answers come from another less common question template category in the imbalanced training dataset. Once the biased subset is created, the balanced other subset can be sampled by keeping the same number of samples as the biased subset. In the final step, 1 / 8 can be reserved for validation, and the remaining 7 / 8 form the biased training set and the balanced training set.

[0060] The process outlined thus leads to the creation of two datasets: a biased set and a balanced set. Each dataset contains 31,513 training samples, reflecting the capacity of the original unbalanced training set (about 31k). Additionally, the two datasets share 4,502 validation samples, 10,819 balanced test samples, and 9,119 biased test samples. The performance of two existing open source models (AVST model and the most advanced LAVISH model) is then evaluated on the biased set and the balanced set, strictly adhering to their original training instructions without any modification. In order to verify the integrity of the code and data, the two models were trained, validated, and tested using the original unbalanced dataset. The AVST model achieves an overall accuracy of 71.25% on the test set (comparable to the reported 71.51%), while the LAVISH model achieves an accuracy of 77.17% (close to the reported 77.20%). Then, both models are trained on the biased training set and the balanced training set, resulting in 4 different models. For models trained on the biased set, the total accuracy on the biased validation set was used to select the best checkpoint across epochs. Similarly, for models trained on the balanced set, the total accuracy on the balanced validation set was used for checkpoint selection. After training, all four models were evaluated on the biased and balanced test slices. The evaluation results on the balanced test set are given in Fig.11 and the evaluation results on the biased test set are presented in Fig.12 Presented in.

[0061] like Fig.11 As shown in Table 1100, both models trained using the balanced set achieve higher overall accuracy on the balanced test set, with a gain of +1.75% for AVST and +2.96% for LAVISH. Specifically, in question types with severe answer imbalance, such as audio-visual count and presence questions, the models trained on the balanced dataset consistently have higher accuracy. For example, the LAVISH model shows improvements of +3.93% and +5.23%, respectively. In contrast, Fig.12As shown in Table 1300, when evaluated on the biased test set, the models trained on the biased set outperformed the models trained on the balanced set. These results suggest that data bias exists in the original imbalanced training dataset. Models trained from the data overfit to the bias, destroying their ability to generalize well.

[0062] The machine learning model described in this article ("LASTAtt") was evaluated using the balanced entire dataset described in this article. The balanced entire dataset described in this article includes 36.7k question-answer pairs for training, 5250 question-answer pairs for validation, and 10819 question-answer pairs for testing. Variants of the machine learning model described in this article were also evaluated. Variants (e.g., "LAST") integrated the existing LAVISH model with the first sub-model, but did not include the third sub-model. Fig.13 Table 1300 and Fig.14 As shown in Table 1400, both LASTAtt and LAST outperform existing models (LAVISH and AVST) on both test sets. It is worth noting that in the balanced test set, LASTAtt's performance improvement is +2.26% higher than LAVISH and +4.42% higher than AVST.

[0063] In order to evaluate the real ability of models to understand the audio-visual context and ensure that they do not only rely on language priors, a contrastive binary question answering evaluation was designed. A subset consisting of pairs of binary QA samples was selected from the balanced test slice of MUSIC-AVQA v2.0. Each pair contains two questions with the same wording, but involves different videos with opposite answers. For example, the question "Are there sounds of ukulele and violin in the video?" is asked for videos A and B. The answer to video A is "yes", while the answer to video B is "no". From the balanced test set, 1643 such contrastive binary question answering pairs were identified. For evaluation, the model must answer both questions in the pair correctly to be considered accurate for the pair. This ensures that given the same wording of the questions, the model truly understands the audio and visual context of each video. Fig.15 As shown in Table 1500, in this challenging scenario, our LASTAtt model significantly outperforms the existing LAVISH and AVST models trained on the balanced set by +4.39% and +6.46%, respectively. Overall, the model described in this paper can correctly predict QA pairs by better reasoning about the audio-visual context rather than relying on language priors.

[0064] In summary, data biases present in the public MUSIC-AVQA dataset have hampered progress in multimodal research. The techniques described in this paper systematically address the data imbalance problem and propose an improved model and a new, larger, balanced dataset, MUSIC-AVQA v2.0. The new AST branch and additional cross-modal pixel attention sub-model described in this paper allow the improved model to outperform previous methods on almost all question categories, serving as a strong baseline for the AVQA task.

[0065] Fig.16 The diagram illustrates a computing device that can be used in various aspects, such as Figure 1 The services, networks, modules and / or devices described in. Figure 1 , any or all of the components can be represented by Fig.16 The system may be implemented by one or more instances of computing device 1600. Fig.16 The computer architecture shown illustrates a conventional server computer, workstation, desktop computer, laptop computer, tablet computer, network appliance, PDA, e-reader, digital cellular telephone, or other computing node and can be used to perform any aspects of the computer described herein, such as implementing the methods described herein.

[0066] The computing device 1600 may include a baseboard or "motherboard," which is a printed circuit board to which many components or devices may be connected via a system bus or other electrical communication paths. One or more central processing units (CPUs) 1604 may operate in conjunction with a chipset 1606. The CPU(s) 1604 may be a standard programmable processor that performs the arithmetic and logic operations necessary for the operation of the computing device 1600.

[0067] The CPU(s) 1604 can perform necessary operations by transitioning from one discrete physical state to the next by manipulating switching elements that distinguish and change these states. Switching elements can typically include electronic circuits that maintain one of two binary states, such as flip-flops, and electronic circuits that provide output states based on a logical combination of the states of one or more other switching elements, such as logic gates. These basic switching elements can be combined to create more complex logic circuits, including registers, adder-subtractors, arithmetic logic units, floating point units, etc.

[0068] The CPU(s) 1604 may be enhanced or replaced with other processing units, such as GPU(s). GPU(s) may include processing units specialized for, but not limited to, highly parallel computations, such as graphics processing and other processing related to visualization.

[0069] Chipset 1606 may provide an interface between CPU(s) 1604 and the rest of the components and devices on the baseboard. Chipset 1606 may provide an interface for random access memory (RAM) 1608, which is used as main memory in computing device 1600. Chipset 1606 may also provide an interface for computer-readable storage media, such as read-only memory (ROM) 1620 or non-volatile RAM (NVRAM) (not shown), which is used to store basic routines that may help boot computing device 1600 and transfer information between various components and devices. ROM 1620 or NVRAM may also store other software components necessary to operate computing device 1600 in accordance with various aspects described herein.

[0070] The computing device 1600 may operate in a networked environment using logical connections to remote computing nodes and computer systems through a local area network (LAN). Chipset 1606 may include functionality to provide network connectivity through a network interface controller (NIC) 1622, such as a Gigabit Ethernet adapter. NIC 1622 may be capable of connecting computing device 1600 to other computing nodes through network 1616. It should be understood that multiple network cards 1622 may be present in computing device 1600, connecting the computing device to other types of networks and remote computer systems.

[0071] The computing device 1600 may be connected to a mass storage device 1628, which provides non-volatile storage for the computer. The mass storage device 1628 may store system programs, applications, other program modules, and data described in more detail herein. The mass storage device 1628 may be connected to the computing device 1600 via a storage controller 1624 connected to the chipset 1606. The mass storage device 1628 may be composed of one or more physical storage units. The mass storage device 1628 may include a management component 1610. The storage controller 1624 may be connected to the physical storage unit via a serial attached SCSI (SAS) interface, a serial advanced technology attachment (SATA) interface, a fiber channel (FC) interface, or other types of interfaces for physically connecting and transmitting data between the computer and the physical storage unit.

[0072] The computing device 1600 may store data on the mass storage device 1628 by transforming the physical state of the physical storage unit to reflect that the information is stored. The specific transformation of the physical state may be related to various factors and different implementations of the description. Examples of these factors may include, but are not limited to, the technology used to implement the physical storage unit and whether the mass storage device 1628 is characterized as a primary storage device or an auxiliary storage device.

[0073] For example, the computing device 1600 can store information to the mass storage device 1628 by issuing instructions through the storage controller 1624 to change the magnetic properties of specific locations in the disk drive unit, the reflective or refractive properties of specific locations in the optical storage unit, or the electrical properties of specific capacitors, transistors, or other discrete components in the solid-state storage unit. Other transformations of the physical media are possible without departing from the scope and spirit of the present description, wherein the above examples are provided only to facilitate the description. The computing device 1600 can read further information from the mass storage device 1628 by detecting the physical state or properties of one or more specific locations in the physical storage unit.

[0074] In addition to the mass storage device 1628 described above, the computing device 1600 may also access other computer-readable storage media to store and retrieve information, such as program modules, data structures, or other data. Those skilled in the art will appreciate that computer-readable storage media may be any available media that provides storage of non-transitory data and can be accessed by the computing device 1600.

[0075] By way of example and not limitation, computer-readable storage media may include volatile and nonvolatile, temporary and non-temporary computer-readable storage media, and removable and non-removable media implemented in any method or technology. Computer-readable storage media include, but are not limited to, RAM, ROM, erasable programmable ROM ("EPROM"), electrically erasable programmable ROM ("EEPROM"), flash memory or other solid-state memory technology, compact disk ROM ("CD-ROM"), digital versatile disk ("DVD"), high-definition DVD ("HD-DVD"), Blu-ray or other optical storage, cassettes, magnetic tape, disk storage, other magnetic storage devices, or any other medium that can be used to store desired content in a non-temporary manner.

[0076] Mass storage devices such as Fig.16 The mass storage device 1628 depicted in FIG. 1 may store an operating system for controlling the operation of the computing device 1600. The operating system may include a version of a LINUX operating system. The operating system may include a version of a WINDOWS SERVER operating system from Microsoft Corporation. According to other aspects, the operating system may include a version of a UNIX operating system. Various mobile phone operating systems, such as IOS and Android, may also be used. It should be understood that other operating systems may also be used. The mass storage device 1628 may store other systems or applications and data utilized by the computing device 1600.

[0077] The mass storage device 1628 or other computer-readable storage medium may also be encoded with computer-executable instructions that, when loaded into the computing device 1600, transform the computing device from a general-purpose computing system into a special-purpose computer capable of implementing aspects described herein. As described above, these computer-executable instructions transform the computing device 1600 by specifying how the CPU(s) 1604 transition between states. The computing device 1600 may access a computer-readable storage medium storing computer-executable instructions that, when executed by the computing device 1600, may perform the methods described herein.

[0078] Computing devices such as Fig.16 The computing device 1600 depicted in FIG. 1 may also include an input / output controller 1632 for receiving and processing input from a number of input devices, such as a keyboard, a mouse, a touch pad, a touch screen, an electronic stylus, or other types of input devices. Similarly, the input / output controller 1632 may provide output to a display, such as a computer monitor, a flat panel display, a digital projector, a printer, a plotter, or other types of output devices. It is understood that the computing device 1600 may not include Fig.16 All components shown in the Fig.16 Other components not explicitly shown in the figure or may be used in conjunction with Fig.16 A completely different architecture as shown in .

[0079] As described herein, a computing device may be a physical computing device such as Fig.16 The computing device 1600 of the present invention may also include a virtual machine host process and one or more virtual machine instances. The computer executable instructions may be indirectly executed by the physical hardware of the computing device by interpreting and / or executing instructions stored and executed in the context of the virtual machine.

[0080] It should be understood that the methods and systems are not limited to specific methods, specific components or specific implementations. It should also be understood that the terminology used herein is for the purpose of describing specific embodiments only and is not intended to be limiting.

[0081] Unless the context clearly dictates otherwise, as used in the specification and the appended claims, the singular forms "a", "an", and "the" include plural references. Scopes herein may be expressed as from "about" one particular value and / or to "about" another particular value. When such a range is expressed, another embodiment includes from a particular value and / or to another particular value. Similarly, when values ​​are expressed as approximations, by using the antecedent "about", it may be understood that the particular value forms another embodiment. It will be further understood that the endpoints of each range are important relative to and independent of the other endpoint.

[0082] "Optional" or "optionally" means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not.

[0083] Throughout the description and claims of this specification, the word "comprise" and its variations, such as "comprising" and "comprises", means "including but not limited to" and is not intended to exclude, for example, other components, integers or steps. "Exemplary" means "example" and is not intended to convey an indication of a preferred or ideal embodiment. "Such as" is not used in a narrow sense, but for interpretation purposes.

[0084] Components that can be used to perform the methods and systems are described. When combinations, subsets, interactions, groups, etc. of these components are described, it is understood that each component is specifically contemplated and described herein for all methods and systems, although specific references to each of the various individual and collective combinations and arrangements may not be explicitly described. This applies to all aspects of the application, including but not limited to the operations in the methods. Therefore, if there are various additional operations that can be performed, it is understood that each of these additional operations can be performed using any specific embodiment or combination of the described methods.

[0085] The present method and system may be understood more readily by reference to the following detailed description of the preferred embodiments and embodiments included therein and the accompanying drawings and their descriptions.

[0086] As will be appreciated by those skilled in the art, the methods and systems may take the form of a fully hardware embodiment, a fully software embodiment, or an embodiment combining software and hardware aspects. In addition, the methods and systems may take the form of a computer program product on a computer-readable storage medium having computer-readable program instructions (e.g., computer software) embodied in the storage medium. More specifically, the methods and systems may take the form of web-implemented computer software. Any suitable computer-readable storage medium may be used, including a hard disk, a CD-ROM, an optical storage device, or a magnetic storage device.

[0087] Embodiments of the method and system are described below with reference to block diagrams and flow charts of the method, system, apparatus, and computer program product. It will be appreciated that each block of the block diagram and flow chart illustration, and the combination of blocks in the block diagram and flow chart illustration, respectively, can be implemented by computer program instructions. These computer program instructions can be loaded on a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, so that the instructions executed on the computer or other programmable data processing device create components for implementing the functions specified in one or more flow chart blocks.

[0088] These computer program instructions may also be stored in a computer-readable memory, which may instruct a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture, which includes computer-readable instructions for implementing the functions specified in one or more flowchart blocks. The computer program instructions may also be loaded onto a computer or other programmable data processing device to cause a series of operating steps to be performed on the computer or other programmable device, thereby producing a computer-implemented process, so that the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more flowchart blocks.

[0089] The various features and processes described above can be used independently of each other or combined in various ways. All possible combinations and sub-combinations are intended to fall within the scope of the present disclosure. Additionally, some method or process blocks may be omitted in some implementations. The methods and processes described herein are also not limited to any particular sequence, and the blocks or states associated therewith may be performed in other appropriate sequences. For example, the blocks or states described may be performed in an order other than the order of the particular description, or a plurality of blocks or states may be combined in a single block or state. The example blocks or states may be performed serially, in parallel or otherwise. The blocks or states may be added to or removed from the described example embodiments. The example systems and components described herein may be configured differently from the described configurations. For example, elements may be added, removed or rearranged compared to the described example embodiments.

[0090] It is also understood that various items are illustrated as being stored in memory or on storage devices when being used, and these items or portions thereof may be transferred between memory and other storage devices for the purpose of memory management and data integrity. Alternatively, in other embodiments, some or all of the software modules and / or systems may be executed in memory on another device and communicate with the illustrated computing system via inter-computer communication. In addition, in some embodiments, some or all of the systems and / or modules may be implemented or provided in other ways, such as at least partially implemented or provided in firmware and / or hardware, which includes but is not limited to one or more application specific integrated circuits (“ASICs”), standard integrated circuits, controllers (e.g., by executing appropriate instructions and including microcontrollers and / or embedded controllers), field programmable gate arrays (“FPGAs”), complex programmable logic devices (“CPLDs”), etc. Some or all of the modules, systems, and data structures may also be stored (e.g., as software instructions or structured data) on a computer-readable medium (such as a hard disk, memory, network, or portable media product) for reading by an appropriate device or through an appropriate connection. Systems, modules, and data structures may also be transmitted as a generated data signal (e.g., as part of a carrier wave or other analog or digital propagation signal) on various computer-readable transmission media (including wireless and wire / cable-based media) and may take a variety of forms (e.g., as part of a single or multiplexed analog signal, or as multiple discrete digital data packets or frames). Such a computer program product may also take other forms in other embodiments. Therefore, the present invention may be practiced using other computer system configurations.

[0091] Although these methods and systems have been described in conjunction with preferred and specific embodiments, it is not intended to limit the scope to the particular embodiments set forth, as the embodiments herein are intended in all respects to be illustrative rather than restrictive.

[0092] Unless otherwise expressly stated, any method described herein is not intended to be construed as requiring that its operations be performed in a specific order. Therefore, if a method claim does not actually recite the order in which its operations are to be performed, or if the claims or specification do not otherwise expressly state that the operations are to be limited to a specific order, no order is intended to be inferred in any respect. This applies to any possible non-express basis for interpretation, including: logical issues regarding arrangement of steps or flow of operations; plain meaning derived from grammatical organization or punctuation; and the number or type of embodiments described in the specification.

[0093] It is obvious to those skilled in the art that various modifications and variations may be made without departing from the scope or spirit of the present disclosure. Other embodiments will be apparent to those skilled in the art by considering the description and practice described herein. It is intended that the description and example figures be considered as exemplary only, with the true scope and spirit being indicated by the appended claims.

Claims

1. A method for improving audio-visual question answering, comprising: A machine learning model is configured for audio-visual question answering (AVQA), wherein the machine learning model includes a first sub-model configured to capture semantic audio information and output an audio spatial feature map. The machine learning model includes a second sub-model, which is configured to extract the visual feature x vs and audio feature x as , and is also configured to obtain the problem vector x q , wherein the machine learning model includes a third sub-model, the third sub-model being configured to capture audio-visual correspondence at a granular level; creating a balanced AVQA dataset, wherein the balanced AVQA dataset includes a balanced distribution of answers in each question category; and The machine learning model is trained using at least a subset of the balanced AVAQ dataset, wherein the machine learning model is trained to answer questions related to visual objects, sounds, and their associations in a video.

2. The method according to claim 1, further comprising: In the audio space feature map Average pooling is applied to generate the audio vector as well as Calculate the visual feature x output from the second sub-model vs Related to spatial attention.

3. The method according to claim 2, further comprising: Generate the first audio-visual fusion vector The first audio-visual fusion vector Capturing visual features that audio is interested in.

4. The method according to claim 3, further comprising: Along the time axis, in the problem vector x q The first audio-visual fusion vector of interest Execute time positioning on the top; as well as Generate the first problem perceptual audio-visual fusion vector 5. The method according to claim 1, further comprising: The visual feature x is calculated by the third sub-model vs With the audio feature x as Pixel-wise audio-visual attention is used between the two to capture the correspondence between the visual modality and the audio modality.

6. The method according to claim 5, further comprising: Generate a second audio-visual fusion vector by using the third sub-model 7. The method according to claim 6, further comprising: Along the time axis, in the problem vector x q The second audio-visual fusion vector of interest Execute time positioning on the top; as well as Generating the second problem perceptual audio-visual fusion vector 8. The method according to claim 1, further comprising: The question-aware audio-visual fusion vector generated based on the output from the second sub-model First problem: Perceptual audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector concatenate to generate the final problem-aware audio-visual fusion vector.

9. The method according to claim 1, wherein there are multiple question categories, and the multiple question categories include an audiovisual presence question category, an audiovisual count question category, an audiovisual time question category, a visual count question category, a visual position question category, an audio count question category, and an audio comparison question category.

10. A system for improving audio-visual question answering, comprising: at least one processor; as well as at least one memory, the at least one memory comprising computer readable instructions that, when executed by the at least one processor, cause the system to perform operations comprising: A machine learning model is configured for audio-visual question answering (AVQA), wherein the machine learning model includes a first sub-model configured to capture semantic audio information and output an audio spatial feature map. The machine learning model includes a second sub-model, which is configured to extract the visual feature x vs and audio feature x as , and is also configured to obtain the problem vector x q , wherein the machine learning model includes a third sub-model, the third sub-model being configured to capture audio-visual correspondence at a granular level; creating a balanced AVQA dataset, wherein the balanced AVQA dataset includes a balanced distribution of answers in each question category; and The machine learning model is trained using at least a subset of the balanced AVAQ dataset, wherein the machine learning model is trained to answer questions related to visual objects, sounds, and their associations in a video.

11. The system of claim 10, wherein the operations further comprise: In the audio space feature map Average pooling is applied to generate the audio vector Calculate the visual feature x output from the second sub-model vs related spatial attention, and Generate the first audio-visual fusion vector The first audio-visual fusion vector Capturing visual features that audio is interested in.

12. The system of claim 11, the operations further comprising: Along the time axis, in the problem vector x q The first audio-visual fusion vector of interest Execute time positioning on the top; as well as Generate the first problem perceptual audio-visual fusion vector 13. The system of claim 10, wherein the operations further comprise: The visual feature x is calculated by the third sub-model vs With the audio feature x as pixel-wise audiovisual attention between to capture the correspondence between the visual modality and the audio modality; and Generate a second audio-visual fusion vector by using the third sub-model 14. The system of claim 13, wherein the operations further comprise: Along the time axis, in the problem vector x q The second audio-visual fusion vector of interest Execute time positioning on the top; as well as Generating the second problem perceptual audio-visual fusion vector 15. The system of claim 10, the operations further comprising: The question-aware audio-visual fusion vector generated based on the output from the second sub-model First question: Perceptual audio-visual fusion vector and the second problem of perceptual audio-visual fusion vector concatenate to generate the final problem-aware audio-visual fusion vector.

16. A non-transitory computer-readable storage medium storing computer-readable instructions that, when executed by a processor, cause the processor to perform operations comprising: A machine learning model is configured for audio-visual question answering (AVQA), wherein the machine learning model includes a first sub-model configured to capture semantic audio information and output an audio spatial feature map. The machine learning model includes a second sub-model, which is configured to extract the visual feature x vs and audio feature x as , and is also configured to obtain the problem vector x q , wherein the machine learning model includes a third sub-model, the third sub-model being configured to capture audio-visual correspondence at a granular level; creating a balanced AVQA dataset, wherein the balanced AVQA dataset includes a balanced distribution of answers in each question category; and The machine learning model is trained using at least a subset of the balanced AVAQ dataset, wherein the machine learning model is trained to answer questions related to visual objects, sounds, and their associations in a video.

17. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: In the audio space feature map Average pooling is applied to generate the audio vector Calculate the visual feature x output from the second sub-model vs related spatial attention, and Generate the first audio-visual fusion vector The first audio-visual fusion vector Capturing visual features that audio is interested in.

18. The non-transitory computer-readable storage medium of claim 17, the operations further comprising: Along the time axis, in the problem vector x q The first audio-visual fusion vector of interest Execute time positioning on the top; as well as Generate the first problem perceptual audio-visual fusion vector 19. The non-transitory computer-readable storage medium of claim 16, the operations further comprising: The visual feature x is calculated by the third sub-model vs With the audio feature x as pixel-wise audiovisual attention between to capture the correspondence between the visual modality and the audio modality; and Generate a second audio-visual fusion vector by using the third sub-model 20. The non-transitory computer-readable storage medium of claim 19, the operations further comprising: Along the time axis, in the problem vector x q The second audio-visual fusion vector of interest Execute time positioning on the top; as well as Generating the second problem perceptual audio-visual fusion vector

Images